Sep 09 14:31:06.316162 ip-10-0-133-66 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Sep 09 14:31:06.316173 ip-10-0-133-66 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Sep 09 14:31:06.316181 ip-10-0-133-66 systemd[1]: kubelet.service: Failed with result 'resources'. Sep 09 14:31:06.316457 ip-10-0-133-66 systemd[1]: Failed to start Kubernetes Kubelet. Sep 09 14:31:15.916547 ip-10-0-133-66 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Sep 09 14:31:15.916558 ip-10-0-133-66 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 8e94858ad0b14b2d90f08ae6d07f3b49 -- Sep 09 14:33:04.876609 ip-10-0-133-66 systemd[1]: Starting Kubernetes Kubelet... Sep 09 14:33:05.285430 ip-10-0-133-66 kubenswrapper[2576]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 09 14:33:05.285430 ip-10-0-133-66 kubenswrapper[2576]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 09 14:33:05.285430 ip-10-0-133-66 kubenswrapper[2576]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 09 14:33:05.285430 ip-10-0-133-66 kubenswrapper[2576]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 09 14:33:05.285430 ip-10-0-133-66 kubenswrapper[2576]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 09 14:33:05.286097 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.285489 2576 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 09 14:33:05.288285 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.288267 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 09 14:33:05.288285 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.288285 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 09 14:33:05.288367 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.288289 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 09 14:33:05.288367 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.288293 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 09 14:33:05.288367 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.288312 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 09 14:33:05.288367 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.288315 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 09 14:33:05.288367 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.288319 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 09 14:33:05.288367 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.288323 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 09 14:33:05.288367 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.288326 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 09 14:33:05.288367 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.288328 2576 feature_gate.go:328] unrecognized feature gate: Example2 Sep 09 14:33:05.288367 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.288333 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 09 14:33:05.288367 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.288337 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 09 14:33:05.288367 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.288340 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 09 14:33:05.288367 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.288343 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 09 14:33:05.288367 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.288349 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 09 14:33:05.288367 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.288352 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 09 14:33:05.288367 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.288355 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 09 14:33:05.288367 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.288358 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 09 14:33:05.288367 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.288361 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 09 14:33:05.288367 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.288363 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 09 14:33:05.288367 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.288366 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 09 14:33:05.288823 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.288368 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 09 14:33:05.288823 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.288372 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 09 14:33:05.288823 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.288375 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 09 14:33:05.288823 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.288378 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 09 14:33:05.288823 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.288381 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 09 14:33:05.288823 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.288383 2576 feature_gate.go:328] unrecognized feature gate: Example Sep 09 14:33:05.288823 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.288386 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 09 14:33:05.288823 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.288388 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 09 14:33:05.288823 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.288391 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 09 14:33:05.288823 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.288393 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 09 14:33:05.288823 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.288396 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 09 14:33:05.288823 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.288398 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 09 14:33:05.288823 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.288401 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 09 14:33:05.288823 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.288403 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 09 14:33:05.288823 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.288407 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 09 14:33:05.288823 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.288409 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 09 14:33:05.288823 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.288412 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 09 14:33:05.288823 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.288415 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 09 14:33:05.288823 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.288418 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 09 14:33:05.288823 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.288420 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 09 14:33:05.289328 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.288423 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 09 14:33:05.289328 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.288425 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 09 14:33:05.289328 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.288427 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 09 14:33:05.289328 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.288430 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 09 14:33:05.289328 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.288432 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 09 14:33:05.289328 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.288435 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 09 14:33:05.289328 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.288437 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 09 14:33:05.289328 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.288439 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 09 14:33:05.289328 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.288442 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 09 14:33:05.289328 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.288444 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 09 14:33:05.289328 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.288447 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 09 14:33:05.289328 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.288449 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 09 14:33:05.289328 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.288451 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 09 14:33:05.289328 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.288455 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 09 14:33:05.289328 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.288459 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 09 14:33:05.289328 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.288463 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 09 14:33:05.289328 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.288466 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 09 14:33:05.289328 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.288469 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 09 14:33:05.289328 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.288472 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 09 14:33:05.289838 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.288475 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 09 14:33:05.289838 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.288477 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 09 14:33:05.289838 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.288480 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 09 14:33:05.289838 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.288482 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 09 14:33:05.289838 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.288485 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 09 14:33:05.289838 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.288487 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 09 14:33:05.289838 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.288490 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 09 14:33:05.289838 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.288492 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 09 14:33:05.289838 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.288495 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 09 14:33:05.289838 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.288497 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 09 14:33:05.289838 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.288500 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 09 14:33:05.289838 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.288503 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 09 14:33:05.289838 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.288505 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 09 14:33:05.289838 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.288508 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 09 14:33:05.289838 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.288510 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 09 14:33:05.289838 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.288512 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 09 14:33:05.289838 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.288515 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 09 14:33:05.289838 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.288518 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 09 14:33:05.289838 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.288521 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 09 14:33:05.289838 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.288523 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 09 14:33:05.290353 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.288526 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 09 14:33:05.290353 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.288528 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 09 14:33:05.290353 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.288530 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 09 14:33:05.290353 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.288533 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 09 14:33:05.290353 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.288535 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 09 14:33:05.290353 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.288538 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 09 14:33:05.290353 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.288541 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 09 14:33:05.290353 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.288544 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 09 14:33:05.290353 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.288547 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 09 14:33:05.290353 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.288549 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 09 14:33:05.290353 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.288552 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 09 14:33:05.290353 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.288554 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 09 14:33:05.290353 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.288557 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 09 14:33:05.290353 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.288559 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 09 14:33:05.290353 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.288562 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 09 14:33:05.290353 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.288565 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 09 14:33:05.290353 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.288567 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 09 14:33:05.290353 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.288570 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 09 14:33:05.290353 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.288572 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 09 14:33:05.290353 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.289284 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 09 14:33:05.290836 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.289290 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 09 14:33:05.290836 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.289293 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 09 14:33:05.290836 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.289311 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 09 14:33:05.290836 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.289315 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 09 14:33:05.290836 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.289325 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 09 14:33:05.290836 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.289327 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 09 14:33:05.290836 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.289331 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 09 14:33:05.290836 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.289333 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 09 14:33:05.290836 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.289336 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 09 14:33:05.290836 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.289339 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 09 14:33:05.290836 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.289341 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 09 14:33:05.290836 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.289344 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 09 14:33:05.290836 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.289346 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 09 14:33:05.290836 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.289349 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 09 14:33:05.290836 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.289352 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 09 14:33:05.290836 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.289355 2576 feature_gate.go:328] unrecognized feature gate: Example Sep 09 14:33:05.290836 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.289357 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 09 14:33:05.290836 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.289359 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 09 14:33:05.290836 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.289363 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 09 14:33:05.290836 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.289366 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 09 14:33:05.291367 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.289368 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 09 14:33:05.291367 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.289371 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 09 14:33:05.291367 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.289373 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 09 14:33:05.291367 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.289376 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 09 14:33:05.291367 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.289378 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 09 14:33:05.291367 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.289381 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 09 14:33:05.291367 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.289383 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 09 14:33:05.291367 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.289387 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 09 14:33:05.291367 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.289392 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 09 14:33:05.291367 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.289396 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 09 14:33:05.291367 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.289399 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 09 14:33:05.291367 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.289402 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 09 14:33:05.291367 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.289405 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 09 14:33:05.291367 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.289408 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 09 14:33:05.291367 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.289411 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 09 14:33:05.291367 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.289414 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 09 14:33:05.291367 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.289417 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 09 14:33:05.291367 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.289420 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 09 14:33:05.291367 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.289423 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 09 14:33:05.291830 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.289425 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 09 14:33:05.291830 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.289428 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 09 14:33:05.291830 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.289431 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 09 14:33:05.291830 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.289433 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 09 14:33:05.291830 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.289436 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 09 14:33:05.291830 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.289438 2576 feature_gate.go:328] unrecognized feature gate: Example2 Sep 09 14:33:05.291830 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.289441 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 09 14:33:05.291830 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.289443 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 09 14:33:05.291830 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.289446 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 09 14:33:05.291830 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.289449 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 09 14:33:05.291830 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.289451 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 09 14:33:05.291830 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.289454 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 09 14:33:05.291830 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.289458 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 09 14:33:05.291830 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.289460 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 09 14:33:05.291830 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.289463 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 09 14:33:05.291830 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.289465 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 09 14:33:05.291830 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.289468 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 09 14:33:05.291830 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.289471 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 09 14:33:05.291830 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.289474 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 09 14:33:05.291830 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.289476 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 09 14:33:05.292340 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.289478 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 09 14:33:05.292340 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.289481 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 09 14:33:05.292340 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.289483 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 09 14:33:05.292340 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.289486 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 09 14:33:05.292340 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.289488 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 09 14:33:05.292340 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.289490 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 09 14:33:05.292340 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.289493 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 09 14:33:05.292340 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.289495 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 09 14:33:05.292340 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.289497 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 09 14:33:05.292340 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.289500 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 09 14:33:05.292340 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.289503 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 09 14:33:05.292340 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.289507 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 09 14:33:05.292340 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.289510 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 09 14:33:05.292340 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.289512 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 09 14:33:05.292340 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.289515 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 09 14:33:05.292340 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.289517 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 09 14:33:05.292340 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.289521 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 09 14:33:05.292340 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.289524 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 09 14:33:05.292340 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.289527 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 09 14:33:05.292340 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.289529 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 09 14:33:05.292828 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.289532 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 09 14:33:05.292828 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.289534 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 09 14:33:05.292828 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.289537 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 09 14:33:05.292828 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.289540 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 09 14:33:05.292828 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.289542 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 09 14:33:05.292828 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.289545 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 09 14:33:05.292828 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.289547 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 09 14:33:05.292828 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.289550 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 09 14:33:05.292828 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.289552 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 09 14:33:05.292828 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.289555 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 09 14:33:05.292828 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.289557 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 09 14:33:05.292828 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.289560 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 09 14:33:05.292828 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.289562 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 09 14:33:05.292828 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.289564 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 09 14:33:05.292828 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.289567 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 09 14:33:05.292828 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.289569 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 09 14:33:05.292828 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.289572 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 09 14:33:05.292828 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.289575 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 09 14:33:05.292828 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.289577 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 09 14:33:05.292828 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.289958 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 09 14:33:05.293332 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.289962 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 09 14:33:05.293332 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.289965 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 09 14:33:05.293332 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.289967 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 09 14:33:05.293332 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.289976 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 09 14:33:05.293332 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.289979 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 09 14:33:05.293332 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.289982 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 09 14:33:05.293332 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.290003 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 09 14:33:05.293332 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.290006 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 09 14:33:05.293332 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.290010 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 09 14:33:05.293332 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.290013 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 09 14:33:05.293332 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.290016 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 09 14:33:05.293332 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.290019 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 09 14:33:05.293332 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.290022 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 09 14:33:05.293332 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.290026 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 09 14:33:05.293332 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.290030 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 09 14:33:05.293332 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.290034 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 09 14:33:05.293332 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.290036 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 09 14:33:05.293332 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.290039 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 09 14:33:05.293332 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.290043 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 09 14:33:05.293801 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.290047 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 09 14:33:05.293801 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.290049 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 09 14:33:05.293801 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.290052 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 09 14:33:05.293801 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.290055 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 09 14:33:05.293801 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.290058 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 09 14:33:05.293801 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.290061 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 09 14:33:05.293801 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.290063 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 09 14:33:05.293801 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.290066 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 09 14:33:05.293801 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.290068 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 09 14:33:05.293801 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.290074 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 09 14:33:05.293801 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.290078 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 09 14:33:05.293801 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.290081 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 09 14:33:05.293801 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.290085 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 09 14:33:05.293801 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.290087 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 09 14:33:05.293801 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.290089 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 09 14:33:05.293801 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.290092 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 09 14:33:05.293801 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.290095 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 09 14:33:05.293801 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.290100 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 09 14:33:05.293801 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.290103 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 09 14:33:05.294254 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.290105 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 09 14:33:05.294254 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.290108 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 09 14:33:05.294254 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.290110 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 09 14:33:05.294254 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.290112 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 09 14:33:05.294254 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.290115 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 09 14:33:05.294254 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.290118 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 09 14:33:05.294254 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.290120 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 09 14:33:05.294254 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.290122 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 09 14:33:05.294254 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.290125 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 09 14:33:05.294254 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.290128 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 09 14:33:05.294254 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.290130 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 09 14:33:05.294254 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.290132 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 09 14:33:05.294254 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.290135 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 09 14:33:05.294254 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.290138 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 09 14:33:05.294254 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.290140 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 09 14:33:05.294254 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.290143 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 09 14:33:05.294254 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.290145 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 09 14:33:05.294254 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.290148 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 09 14:33:05.294254 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.290150 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 09 14:33:05.294254 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.290152 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 09 14:33:05.294799 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.290155 2576 feature_gate.go:328] unrecognized feature gate: Example2 Sep 09 14:33:05.294799 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.290157 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 09 14:33:05.294799 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.290161 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 09 14:33:05.294799 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.290163 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 09 14:33:05.294799 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.290166 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 09 14:33:05.294799 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.290168 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 09 14:33:05.294799 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.290171 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 09 14:33:05.294799 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.290173 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 09 14:33:05.294799 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.290176 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 09 14:33:05.294799 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.290178 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 09 14:33:05.294799 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.290181 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 09 14:33:05.294799 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.290189 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 09 14:33:05.294799 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.290191 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 09 14:33:05.294799 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.290194 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 09 14:33:05.294799 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.290196 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 09 14:33:05.294799 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.290198 2576 feature_gate.go:328] unrecognized feature gate: Example Sep 09 14:33:05.294799 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.290201 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 09 14:33:05.294799 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.290204 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 09 14:33:05.294799 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.290207 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 09 14:33:05.294799 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.290209 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 09 14:33:05.295278 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.290212 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 09 14:33:05.295278 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.290214 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 09 14:33:05.295278 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.290217 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 09 14:33:05.295278 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.290220 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 09 14:33:05.295278 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.290222 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 09 14:33:05.295278 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.290224 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 09 14:33:05.295278 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.290227 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 09 14:33:05.295278 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.290229 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 09 14:33:05.295278 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.290232 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 09 14:33:05.295278 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.290234 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 09 14:33:05.295278 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.290236 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 09 14:33:05.295278 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.290239 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 09 14:33:05.295278 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.290241 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 09 14:33:05.295278 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.290244 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 09 14:33:05.295278 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.290247 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 09 14:33:05.295278 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.290250 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 09 14:33:05.295278 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.290252 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 09 14:33:05.295278 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.290255 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 09 14:33:05.295278 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.290257 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 09 14:33:05.295278 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.290260 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 09 14:33:05.295781 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290349 2576 flags.go:64] FLAG: --address="0.0.0.0" Sep 09 14:33:05.295781 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290392 2576 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 09 14:33:05.295781 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290403 2576 flags.go:64] FLAG: --anonymous-auth="true" Sep 09 14:33:05.295781 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290408 2576 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 09 14:33:05.295781 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290418 2576 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 09 14:33:05.295781 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290421 2576 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 09 14:33:05.295781 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290425 2576 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 09 14:33:05.295781 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290430 2576 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 09 14:33:05.295781 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290433 2576 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 09 14:33:05.295781 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290436 2576 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 09 14:33:05.295781 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290440 2576 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 09 14:33:05.295781 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290443 2576 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 09 14:33:05.295781 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290446 2576 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 09 14:33:05.295781 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290449 2576 flags.go:64] FLAG: --cgroup-root="" Sep 09 14:33:05.295781 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290452 2576 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 09 14:33:05.295781 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290455 2576 flags.go:64] FLAG: --client-ca-file="" Sep 09 14:33:05.295781 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290458 2576 flags.go:64] FLAG: --cloud-provider="external" Sep 09 14:33:05.295781 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290461 2576 flags.go:64] FLAG: --cluster-dns="[]" Sep 09 14:33:05.295781 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290467 2576 flags.go:64] FLAG: --cluster-domain="" Sep 09 14:33:05.295781 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290469 2576 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 09 14:33:05.295781 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290472 2576 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 09 14:33:05.295781 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290475 2576 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 09 14:33:05.295781 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290478 2576 flags.go:64] FLAG: --container-log-max-files="5" Sep 09 14:33:05.295781 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290482 2576 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 09 14:33:05.296429 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290485 2576 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 09 14:33:05.296429 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290488 2576 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 09 14:33:05.296429 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290493 2576 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 09 14:33:05.296429 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290496 2576 flags.go:64] FLAG: --contention-profiling="false" Sep 09 14:33:05.296429 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290498 2576 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 09 14:33:05.296429 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290501 2576 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 09 14:33:05.296429 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290504 2576 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 09 14:33:05.296429 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290507 2576 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 09 14:33:05.296429 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290511 2576 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 09 14:33:05.296429 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290514 2576 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 09 14:33:05.296429 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290517 2576 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 09 14:33:05.296429 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290519 2576 flags.go:64] FLAG: --enable-load-reader="false" Sep 09 14:33:05.296429 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290522 2576 flags.go:64] FLAG: --enable-server="true" Sep 09 14:33:05.296429 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290528 2576 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 09 14:33:05.296429 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290535 2576 flags.go:64] FLAG: --event-burst="100" Sep 09 14:33:05.296429 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290538 2576 flags.go:64] FLAG: --event-qps="50" Sep 09 14:33:05.296429 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290541 2576 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 09 14:33:05.296429 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290544 2576 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 09 14:33:05.296429 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290546 2576 flags.go:64] FLAG: --eviction-hard="" Sep 09 14:33:05.296429 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290550 2576 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 09 14:33:05.296429 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290554 2576 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 09 14:33:05.296429 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290557 2576 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 09 14:33:05.296429 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290560 2576 flags.go:64] FLAG: --eviction-soft="" Sep 09 14:33:05.296429 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290563 2576 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 09 14:33:05.296429 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290566 2576 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 09 14:33:05.297104 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290569 2576 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 09 14:33:05.297104 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290572 2576 flags.go:64] FLAG: --experimental-mounter-path="" Sep 09 14:33:05.297104 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290575 2576 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 09 14:33:05.297104 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290578 2576 flags.go:64] FLAG: --fail-swap-on="true" Sep 09 14:33:05.297104 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290580 2576 flags.go:64] FLAG: --feature-gates="" Sep 09 14:33:05.297104 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290584 2576 flags.go:64] FLAG: --file-check-frequency="20s" Sep 09 14:33:05.297104 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290587 2576 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 09 14:33:05.297104 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290590 2576 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 09 14:33:05.297104 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290593 2576 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 09 14:33:05.297104 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290597 2576 flags.go:64] FLAG: --healthz-port="10248" Sep 09 14:33:05.297104 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290600 2576 flags.go:64] FLAG: --help="false" Sep 09 14:33:05.297104 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290603 2576 flags.go:64] FLAG: --hostname-override="ip-10-0-133-66.ec2.internal" Sep 09 14:33:05.297104 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290606 2576 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 09 14:33:05.297104 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290608 2576 flags.go:64] FLAG: --http-check-frequency="20s" Sep 09 14:33:05.297104 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290611 2576 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 09 14:33:05.297104 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290614 2576 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 09 14:33:05.297104 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290617 2576 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 09 14:33:05.297104 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290620 2576 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 09 14:33:05.297104 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290623 2576 flags.go:64] FLAG: --image-service-endpoint="" Sep 09 14:33:05.297104 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290625 2576 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 09 14:33:05.297104 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290628 2576 flags.go:64] FLAG: --kube-api-burst="100" Sep 09 14:33:05.297104 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290639 2576 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 09 14:33:05.297104 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290642 2576 flags.go:64] FLAG: --kube-api-qps="50" Sep 09 14:33:05.297104 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290645 2576 flags.go:64] FLAG: --kube-reserved="" Sep 09 14:33:05.297719 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290648 2576 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 09 14:33:05.297719 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290650 2576 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 09 14:33:05.297719 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290653 2576 flags.go:64] FLAG: --kubelet-cgroups="" Sep 09 14:33:05.297719 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290658 2576 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 09 14:33:05.297719 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290661 2576 flags.go:64] FLAG: --lock-file="" Sep 09 14:33:05.297719 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290663 2576 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 09 14:33:05.297719 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290666 2576 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 09 14:33:05.297719 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290669 2576 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 09 14:33:05.297719 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290675 2576 flags.go:64] FLAG: --log-json-split-stream="false" Sep 09 14:33:05.297719 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290677 2576 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 09 14:33:05.297719 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290680 2576 flags.go:64] FLAG: --log-text-split-stream="false" Sep 09 14:33:05.297719 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290683 2576 flags.go:64] FLAG: --logging-format="text" Sep 09 14:33:05.297719 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290686 2576 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 09 14:33:05.297719 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290689 2576 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 09 14:33:05.297719 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290692 2576 flags.go:64] FLAG: --manifest-url="" Sep 09 14:33:05.297719 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290695 2576 flags.go:64] FLAG: --manifest-url-header="" Sep 09 14:33:05.297719 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290699 2576 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 09 14:33:05.297719 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290704 2576 flags.go:64] FLAG: --max-open-files="1000000" Sep 09 14:33:05.297719 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290708 2576 flags.go:64] FLAG: --max-pods="110" Sep 09 14:33:05.297719 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290710 2576 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 09 14:33:05.297719 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290713 2576 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 09 14:33:05.297719 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290716 2576 flags.go:64] FLAG: --memory-manager-policy="None" Sep 09 14:33:05.297719 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290719 2576 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 09 14:33:05.297719 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290721 2576 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 09 14:33:05.297719 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290724 2576 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 09 14:33:05.298378 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290727 2576 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 09 14:33:05.298378 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290740 2576 flags.go:64] FLAG: --node-status-max-images="50" Sep 09 14:33:05.298378 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290743 2576 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 09 14:33:05.298378 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290745 2576 flags.go:64] FLAG: --oom-score-adj="-999" Sep 09 14:33:05.298378 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290748 2576 flags.go:64] FLAG: --pod-cidr="" Sep 09 14:33:05.298378 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290752 2576 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1" Sep 09 14:33:05.298378 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290757 2576 flags.go:64] FLAG: --pod-manifest-path="" Sep 09 14:33:05.298378 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290760 2576 flags.go:64] FLAG: --pod-max-pids="-1" Sep 09 14:33:05.298378 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290763 2576 flags.go:64] FLAG: --pods-per-core="0" Sep 09 14:33:05.298378 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290766 2576 flags.go:64] FLAG: --port="10250" Sep 09 14:33:05.298378 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290770 2576 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 09 14:33:05.298378 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290773 2576 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0dac09ff3cc31ee0c" Sep 09 14:33:05.298378 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290776 2576 flags.go:64] FLAG: --qos-reserved="" Sep 09 14:33:05.298378 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290779 2576 flags.go:64] FLAG: --read-only-port="10255" Sep 09 14:33:05.298378 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290782 2576 flags.go:64] FLAG: --register-node="true" Sep 09 14:33:05.298378 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290784 2576 flags.go:64] FLAG: --register-with-taints="" Sep 09 14:33:05.298378 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290788 2576 flags.go:64] FLAG: --registry-burst="10" Sep 09 14:33:05.298378 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290791 2576 flags.go:64] FLAG: --registry-qps="5" Sep 09 14:33:05.298378 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290793 2576 flags.go:64] FLAG: --reserved-cpus="" Sep 09 14:33:05.298378 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290796 2576 flags.go:64] FLAG: --reserved-memory="" Sep 09 14:33:05.298378 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290800 2576 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 09 14:33:05.298378 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290803 2576 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 09 14:33:05.298378 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290805 2576 flags.go:64] FLAG: --rotate-certificates="false" Sep 09 14:33:05.298378 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290808 2576 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 09 14:33:05.298378 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290811 2576 flags.go:64] FLAG: --runonce="false" Sep 09 14:33:05.298983 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290815 2576 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 09 14:33:05.298983 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290819 2576 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 09 14:33:05.298983 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290821 2576 flags.go:64] FLAG: --seccomp-default="false" Sep 09 14:33:05.298983 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290824 2576 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 09 14:33:05.298983 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290827 2576 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 09 14:33:05.298983 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290830 2576 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 09 14:33:05.298983 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290833 2576 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 09 14:33:05.298983 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290836 2576 flags.go:64] FLAG: --storage-driver-password="root" Sep 09 14:33:05.298983 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290839 2576 flags.go:64] FLAG: --storage-driver-secure="false" Sep 09 14:33:05.298983 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290842 2576 flags.go:64] FLAG: --storage-driver-table="stats" Sep 09 14:33:05.298983 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290845 2576 flags.go:64] FLAG: --storage-driver-user="root" Sep 09 14:33:05.298983 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290848 2576 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 09 14:33:05.298983 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290851 2576 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 09 14:33:05.298983 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290854 2576 flags.go:64] FLAG: --system-cgroups="" Sep 09 14:33:05.298983 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290857 2576 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 09 14:33:05.298983 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290862 2576 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 09 14:33:05.298983 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290865 2576 flags.go:64] FLAG: --tls-cert-file="" Sep 09 14:33:05.298983 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290870 2576 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 09 14:33:05.298983 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290877 2576 flags.go:64] FLAG: --tls-min-version="" Sep 09 14:33:05.298983 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290880 2576 flags.go:64] FLAG: --tls-private-key-file="" Sep 09 14:33:05.298983 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290882 2576 flags.go:64] FLAG: --topology-manager-policy="none" Sep 09 14:33:05.298983 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290885 2576 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 09 14:33:05.298983 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290888 2576 flags.go:64] FLAG: --topology-manager-scope="container" Sep 09 14:33:05.298983 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290891 2576 flags.go:64] FLAG: --v="2" Sep 09 14:33:05.298983 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290895 2576 flags.go:64] FLAG: --version="false" Sep 09 14:33:05.299619 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290899 2576 flags.go:64] FLAG: --vmodule="" Sep 09 14:33:05.299619 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290903 2576 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 09 14:33:05.299619 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.290906 2576 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 09 14:33:05.299619 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.291037 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 09 14:33:05.299619 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.291041 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 09 14:33:05.299619 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.291044 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 09 14:33:05.299619 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.291047 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 09 14:33:05.299619 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.291050 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 09 14:33:05.299619 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.291052 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 09 14:33:05.299619 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.291055 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 09 14:33:05.299619 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.291057 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 09 14:33:05.299619 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.291060 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 09 14:33:05.299619 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.291062 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 09 14:33:05.299619 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.291065 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 09 14:33:05.299619 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.291067 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 09 14:33:05.299619 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.291070 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 09 14:33:05.299619 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.291072 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 09 14:33:05.299619 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.291075 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 09 14:33:05.299619 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.291077 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 09 14:33:05.299619 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.291080 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 09 14:33:05.300107 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.291082 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 09 14:33:05.300107 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.291090 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 09 14:33:05.300107 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.291092 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 09 14:33:05.300107 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.291095 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 09 14:33:05.300107 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.291099 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 09 14:33:05.300107 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.291101 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 09 14:33:05.300107 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.291104 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 09 14:33:05.300107 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.291106 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 09 14:33:05.300107 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.291108 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 09 14:33:05.300107 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.291111 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 09 14:33:05.300107 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.291113 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 09 14:33:05.300107 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.291116 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 09 14:33:05.300107 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.291119 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 09 14:33:05.300107 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.291121 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 09 14:33:05.300107 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.291124 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 09 14:33:05.300107 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.291126 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 09 14:33:05.300107 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.291128 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 09 14:33:05.300107 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.291131 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 09 14:33:05.300107 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.291135 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 09 14:33:05.300107 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.291139 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 09 14:33:05.300650 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.291141 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 09 14:33:05.300650 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.291144 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 09 14:33:05.300650 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.291147 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 09 14:33:05.300650 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.291149 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 09 14:33:05.300650 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.291157 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 09 14:33:05.300650 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.291160 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 09 14:33:05.300650 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.291162 2576 feature_gate.go:328] unrecognized feature gate: Example Sep 09 14:33:05.300650 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.291165 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 09 14:33:05.300650 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.291167 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 09 14:33:05.300650 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.291170 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 09 14:33:05.300650 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.291172 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 09 14:33:05.300650 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.291175 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 09 14:33:05.300650 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.291177 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 09 14:33:05.300650 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.291180 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 09 14:33:05.300650 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.291188 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 09 14:33:05.300650 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.291190 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 09 14:33:05.300650 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.291194 2576 feature_gate.go:328] unrecognized feature gate: Example2 Sep 09 14:33:05.300650 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.291197 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 09 14:33:05.300650 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.291199 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 09 14:33:05.300650 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.291202 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 09 14:33:05.301147 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.291204 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 09 14:33:05.301147 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.291206 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 09 14:33:05.301147 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.291209 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 09 14:33:05.301147 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.291211 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 09 14:33:05.301147 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.291214 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 09 14:33:05.301147 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.291216 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 09 14:33:05.301147 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.291219 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 09 14:33:05.301147 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.291222 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 09 14:33:05.301147 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.291224 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 09 14:33:05.301147 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.291226 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 09 14:33:05.301147 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.291231 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 09 14:33:05.301147 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.291234 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 09 14:33:05.301147 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.291237 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 09 14:33:05.301147 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.291240 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 09 14:33:05.301147 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.291242 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 09 14:33:05.301147 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.291245 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 09 14:33:05.301147 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.291248 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 09 14:33:05.301147 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.291250 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 09 14:33:05.301147 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.291253 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 09 14:33:05.301636 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.291256 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 09 14:33:05.301636 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.291260 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 09 14:33:05.301636 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.291263 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 09 14:33:05.301636 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.291266 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 09 14:33:05.301636 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.291269 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 09 14:33:05.301636 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.291271 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 09 14:33:05.301636 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.291292 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 09 14:33:05.301636 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.291315 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 09 14:33:05.301636 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.291321 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 09 14:33:05.301636 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.291326 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 09 14:33:05.301636 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.291329 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 09 14:33:05.301636 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.291332 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 09 14:33:05.301636 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.291335 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 09 14:33:05.301636 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.291337 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 09 14:33:05.301636 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.291340 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 09 14:33:05.301636 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.291342 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 09 14:33:05.301636 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.291345 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 09 14:33:05.301636 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.291348 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 09 14:33:05.301636 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.291350 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 09 14:33:05.302153 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.291353 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 09 14:33:05.302153 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.291355 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 09 14:33:05.302153 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.291358 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 09 14:33:05.302153 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.291360 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 09 14:33:05.302153 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.291366 2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 09 14:33:05.302153 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.298492 2576 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 09 14:33:05.302153 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.298509 2576 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 09 14:33:05.302153 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.299240 2576 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 09 14:33:05.302153 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.299247 2576 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 09 14:33:05.302153 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299337 2576 feature_gate.go:328] unrecognized feature gate: Example2 Sep 09 14:33:05.302153 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299344 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 09 14:33:05.302153 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299348 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 09 14:33:05.302153 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299351 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 09 14:33:05.302153 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299354 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 09 14:33:05.302153 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299356 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 09 14:33:05.302568 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299359 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 09 14:33:05.302568 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299362 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 09 14:33:05.302568 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299364 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 09 14:33:05.302568 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299367 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 09 14:33:05.302568 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299370 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 09 14:33:05.302568 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299372 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 09 14:33:05.302568 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299375 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 09 14:33:05.302568 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299378 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 09 14:33:05.302568 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299395 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 09 14:33:05.302568 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299398 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 09 14:33:05.302568 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299401 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 09 14:33:05.302568 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299404 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 09 14:33:05.302568 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299408 2576 feature_gate.go:328] unrecognized feature gate: Example Sep 09 14:33:05.302568 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299411 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 09 14:33:05.302568 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299413 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 09 14:33:05.302568 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299416 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 09 14:33:05.302568 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299419 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 09 14:33:05.302568 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299422 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 09 14:33:05.302568 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299425 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 09 14:33:05.302568 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299429 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 09 14:33:05.303121 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299432 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 09 14:33:05.303121 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299436 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 09 14:33:05.303121 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299439 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 09 14:33:05.303121 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299443 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 09 14:33:05.303121 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299446 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 09 14:33:05.303121 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299449 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 09 14:33:05.303121 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299451 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 09 14:33:05.303121 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299454 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 09 14:33:05.303121 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299456 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 09 14:33:05.303121 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299459 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 09 14:33:05.303121 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299461 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 09 14:33:05.303121 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299464 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 09 14:33:05.303121 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299466 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 09 14:33:05.303121 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299469 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 09 14:33:05.303121 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299471 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 09 14:33:05.303121 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299474 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 09 14:33:05.303121 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299476 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 09 14:33:05.303121 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299478 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 09 14:33:05.303121 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299481 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 09 14:33:05.303121 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299483 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 09 14:33:05.303624 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299486 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 09 14:33:05.303624 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299488 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 09 14:33:05.303624 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299491 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 09 14:33:05.303624 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299494 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 09 14:33:05.303624 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299496 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 09 14:33:05.303624 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299498 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 09 14:33:05.303624 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299501 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 09 14:33:05.303624 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299503 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 09 14:33:05.303624 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299505 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 09 14:33:05.303624 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299508 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 09 14:33:05.303624 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299510 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 09 14:33:05.303624 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299513 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 09 14:33:05.303624 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299515 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 09 14:33:05.303624 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299519 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 09 14:33:05.303624 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299523 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 09 14:33:05.303624 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299526 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 09 14:33:05.303624 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299529 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 09 14:33:05.303624 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299532 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 09 14:33:05.303624 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299534 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 09 14:33:05.303624 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299537 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 09 14:33:05.304248 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299539 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 09 14:33:05.304248 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299542 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 09 14:33:05.304248 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299544 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 09 14:33:05.304248 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299546 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 09 14:33:05.304248 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299549 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 09 14:33:05.304248 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299552 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 09 14:33:05.304248 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299554 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 09 14:33:05.304248 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299556 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 09 14:33:05.304248 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299559 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 09 14:33:05.304248 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299561 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 09 14:33:05.304248 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299563 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 09 14:33:05.304248 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299566 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 09 14:33:05.304248 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299568 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 09 14:33:05.304248 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299571 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 09 14:33:05.304248 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299573 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 09 14:33:05.304248 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299576 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 09 14:33:05.304248 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299578 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 09 14:33:05.304248 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299580 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 09 14:33:05.304248 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299583 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 09 14:33:05.304817 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299585 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 09 14:33:05.304817 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299588 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 09 14:33:05.304817 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299590 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 09 14:33:05.304817 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299592 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 09 14:33:05.304817 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299596 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 09 14:33:05.304817 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299599 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 09 14:33:05.304817 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299603 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 09 14:33:05.304817 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299606 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 09 14:33:05.304817 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299609 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 09 14:33:05.304817 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299611 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 09 14:33:05.304817 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299614 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 09 14:33:05.304817 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299617 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 09 14:33:05.304817 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299620 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 09 14:33:05.304817 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299623 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 09 14:33:05.305173 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.299628 2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 09 14:33:05.305173 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299753 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 09 14:33:05.305173 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299758 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 09 14:33:05.305173 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299761 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 09 14:33:05.305173 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299764 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 09 14:33:05.305173 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299767 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 09 14:33:05.305173 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299769 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 09 14:33:05.305173 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299772 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 09 14:33:05.305173 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299775 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 09 14:33:05.305173 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299777 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 09 14:33:05.305173 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299780 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 09 14:33:05.305173 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299782 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 09 14:33:05.305173 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299785 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 09 14:33:05.305173 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299787 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 09 14:33:05.305173 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299789 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 09 14:33:05.305590 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299792 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 09 14:33:05.305590 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299794 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 09 14:33:05.305590 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299797 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 09 14:33:05.305590 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299800 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 09 14:33:05.305590 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299802 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 09 14:33:05.305590 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299805 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 09 14:33:05.305590 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299807 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 09 14:33:05.305590 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299810 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 09 14:33:05.305590 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299812 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 09 14:33:05.305590 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299815 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 09 14:33:05.305590 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299817 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 09 14:33:05.305590 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299820 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 09 14:33:05.305590 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299822 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 09 14:33:05.305590 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299824 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 09 14:33:05.305590 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299827 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 09 14:33:05.305590 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299830 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 09 14:33:05.305590 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299832 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 09 14:33:05.305590 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299834 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 09 14:33:05.305590 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299837 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 09 14:33:05.306099 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299839 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 09 14:33:05.306099 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299842 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 09 14:33:05.306099 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299844 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 09 14:33:05.306099 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299847 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 09 14:33:05.306099 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299849 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 09 14:33:05.306099 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299851 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 09 14:33:05.306099 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299854 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 09 14:33:05.306099 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299856 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 09 14:33:05.306099 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299859 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 09 14:33:05.306099 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299861 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 09 14:33:05.306099 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299864 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 09 14:33:05.306099 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299866 2576 feature_gate.go:328] unrecognized feature gate: Example Sep 09 14:33:05.306099 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299869 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 09 14:33:05.306099 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299871 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 09 14:33:05.306099 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299873 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 09 14:33:05.306099 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299876 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 09 14:33:05.306099 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299878 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 09 14:33:05.306099 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299881 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 09 14:33:05.306099 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299883 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 09 14:33:05.306099 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299886 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 09 14:33:05.306852 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299889 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 09 14:33:05.306852 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299892 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 09 14:33:05.306852 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299894 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 09 14:33:05.306852 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299897 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 09 14:33:05.306852 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299899 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 09 14:33:05.306852 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299903 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 09 14:33:05.306852 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299906 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 09 14:33:05.306852 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299909 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 09 14:33:05.306852 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299912 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 09 14:33:05.306852 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299915 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 09 14:33:05.306852 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299918 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 09 14:33:05.306852 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299920 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 09 14:33:05.306852 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299924 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 09 14:33:05.306852 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299927 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 09 14:33:05.306852 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299929 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 09 14:33:05.306852 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299932 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 09 14:33:05.306852 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299935 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 09 14:33:05.306852 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299937 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 09 14:33:05.306852 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299940 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 09 14:33:05.306852 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299942 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 09 14:33:05.307524 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299944 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 09 14:33:05.307524 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299947 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 09 14:33:05.307524 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299949 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 09 14:33:05.307524 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299951 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 09 14:33:05.307524 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299954 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 09 14:33:05.307524 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299956 2576 feature_gate.go:328] unrecognized feature gate: Example2 Sep 09 14:33:05.307524 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299958 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 09 14:33:05.307524 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299961 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 09 14:33:05.307524 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299963 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 09 14:33:05.307524 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299965 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 09 14:33:05.307524 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299968 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 09 14:33:05.307524 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299970 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 09 14:33:05.307524 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299973 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 09 14:33:05.307524 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299975 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 09 14:33:05.307524 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299979 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 09 14:33:05.307524 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299982 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 09 14:33:05.307524 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299985 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 09 14:33:05.307524 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299988 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 09 14:33:05.307524 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299990 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 09 14:33:05.308023 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299993 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 09 14:33:05.308023 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299995 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 09 14:33:05.308023 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.299998 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 09 14:33:05.308023 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.300000 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 09 14:33:05.308023 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.300003 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 09 14:33:05.308023 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.300005 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 09 14:33:05.308023 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:05.300007 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 09 14:33:05.308023 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.300012 2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 09 14:33:05.308023 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.300125 2576 server.go:962] "Client rotation is on, will bootstrap in background" Sep 09 14:33:05.308023 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.301987 2576 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 09 14:33:05.308023 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.302888 2576 server.go:1019] "Starting client certificate rotation" Sep 09 14:33:05.308023 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.303006 2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 09 14:33:05.308023 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.303858 2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 09 14:33:05.308514 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.305863 2576 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 09 14:33:05.313773 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.313746 2576 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 09 14:33:05.323242 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.323220 2576 log.go:25] "Validated CRI v1 runtime API" Sep 09 14:33:05.325917 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.325901 2576 log.go:25] "Validated CRI v1 image API" Sep 09 14:33:05.326750 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.326736 2576 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 09 14:33:05.328883 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.328862 2576 fs.go:135] Filesystem UUIDs: map[084d15be-2aff-40b5-b616-6a166cf87e50:/dev/nvme0n1p4 75299b55-2c30-43de-ad01-5b4e4a56ad76:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Sep 09 14:33:05.328964 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.328883 2576 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 09 14:33:05.333382 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.333256 2576 manager.go:216] Machine: {Timestamp:2026-09-09 14:33:05.332777379 +0000 UTC m=+0.343693633 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101368 MemoryCapacity:32812154880 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec225333c6a78e1c276e01bf71359a5e SystemUUID:ec225333-c6a7-8e1c-276e-01bf71359a5e BootID:8e94858a-d0b1-4b2d-90f0-8ae6d07f3b49 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406077440 Type:vfs Inodes:4005390 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562430976 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406077440 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:81:b0:7d:31:2b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:81:b0:7d:31:2b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:62:6e:33:b3:98:c4 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812154880 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 09 14:33:05.333382 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.333374 2576 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 09 14:33:05.333505 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.333453 2576 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 09 14:33:05.336681 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.336658 2576 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 09 14:33:05.336829 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.336685 2576 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-66.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 09 14:33:05.337815 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.337805 2576 topology_manager.go:138] "Creating topology manager with none policy" Sep 09 14:33:05.337847 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.337818 2576 container_manager_linux.go:309] "Creating device plugin manager" Sep 09 14:33:05.337847 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.337825 2576 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 09 14:33:05.338436 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.338426 2576 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 09 14:33:05.339227 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.339218 2576 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 09 14:33:05.342202 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.342187 2576 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 09 14:33:05.343174 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.343163 2576 state_mem.go:36] "Initialized new in-memory state store" Sep 09 14:33:05.343284 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.343275 2576 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 09 14:33:05.343341 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.343319 2576 kubelet.go:486] "Attempting to sync node with API server" Sep 09 14:33:05.343341 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.343330 2576 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 09 14:33:05.343415 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.343345 2576 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 09 14:33:05.343415 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.343353 2576 kubelet.go:398] "Adding apiserver pod source" Sep 09 14:33:05.343415 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.343362 2576 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 09 14:33:05.343704 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.343693 2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 09 14:33:05.343747 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.343717 2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 09 14:33:05.345884 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.345870 2576 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 09 14:33:05.347859 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.347845 2576 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 09 14:33:05.347917 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.347877 2576 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 09 14:33:05.349341 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.349328 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 09 14:33:05.349402 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.349347 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 09 14:33:05.349402 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.349353 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 09 14:33:05.349402 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.349360 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 09 14:33:05.349402 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.349373 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 09 14:33:05.349402 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.349379 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 09 14:33:05.349402 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.349392 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 09 14:33:05.349402 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.349397 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 09 14:33:05.349402 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.349403 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 09 14:33:05.349621 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.349408 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 09 14:33:05.349621 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.349421 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 09 14:33:05.349621 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.349427 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 09 14:33:05.350918 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.350906 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 09 14:33:05.350918 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.350916 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 09 14:33:05.353670 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.353638 2576 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-66.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 09 14:33:05.353793 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:05.353752 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 09 14:33:05.354289 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:05.354263 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-66.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 09 14:33:05.355388 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.355373 2576 server.go:1268] "Started kubelet" Sep 09 14:33:05.355468 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.355423 2576 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 09 14:33:05.355578 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.355520 2576 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 09 14:33:05.355631 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.355600 2576 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 09 14:33:05.356113 ip-10-0-133-66 systemd[1]: Started Kubernetes Kubelet. Sep 09 14:33:05.357781 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.357765 2576 server.go:310] "Adding debug handlers to kubelet server" Sep 09 14:33:05.359222 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:05.359199 2576 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 09 14:33:05.359498 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.359484 2576 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 09 14:33:05.362950 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.362929 2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 09 14:33:05.363527 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.363509 2576 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 09 14:33:05.363668 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:05.363647 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-66.ec2.internal\" not found" Sep 09 14:33:05.364086 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.364072 2576 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 09 14:33:05.364142 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.364090 2576 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 09 14:33:05.364142 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.364071 2576 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 09 14:33:05.364214 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.364172 2576 reconstruct.go:97] "Volume reconstruction finished" Sep 09 14:33:05.364214 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.364179 2576 reconciler.go:29] "Reconciler: start to sync state" Sep 09 14:33:05.364582 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.364568 2576 factory.go:55] Registering systemd factory Sep 09 14:33:05.364660 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.364586 2576 factory.go:223] Registration of the systemd container factory successfully Sep 09 14:33:05.364793 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.364781 2576 factory.go:153] Registering CRI-O factory Sep 09 14:33:05.364849 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.364795 2576 factory.go:223] Registration of the crio container factory successfully Sep 09 14:33:05.364849 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.364834 2576 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 09 14:33:05.364849 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.364847 2576 factory.go:103] Registering Raw factory Sep 09 14:33:05.364998 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.364857 2576 manager.go:1196] Started watching for new ooms in manager Sep 09 14:33:05.365244 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.365233 2576 manager.go:349] Starting recovery of all containers Sep 09 14:33:05.369137 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:05.369102 2576 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-133-66.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 09 14:33:05.369428 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:05.369271 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 09 14:33:05.370336 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:05.369174 2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-66.ec2.internal.18d3ada150edfe12 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-66.ec2.internal,UID:ip-10-0-133-66.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-66.ec2.internal,},FirstTimestamp:2026-09-09 14:33:05.355333138 +0000 UTC m=+0.366249380,LastTimestamp:2026-09-09 14:33:05.355333138 +0000 UTC m=+0.366249380,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-66.ec2.internal,}" Sep 09 14:33:05.375851 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.375831 2576 manager.go:354] Recovery completed Sep 09 14:33:05.380968 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.380955 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 09 14:33:05.382609 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.382592 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-66.ec2.internal" event="NodeHasSufficientMemory" Sep 09 14:33:05.382680 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.382622 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-66.ec2.internal" event="NodeHasNoDiskPressure" Sep 09 14:33:05.382680 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.382632 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-66.ec2.internal" event="NodeHasSufficientPID" Sep 09 14:33:05.383206 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.383191 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 09 14:33:05.383282 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.383195 2576 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 09 14:33:05.383282 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.383249 2576 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 09 14:33:05.383282 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.383267 2576 state_mem.go:36] "Initialized new in-memory state store" Sep 09 14:33:05.383851 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.383838 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-66.ec2.internal" event="NodeHasSufficientMemory" Sep 09 14:33:05.383900 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.383867 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-66.ec2.internal" event="NodeHasNoDiskPressure" Sep 09 14:33:05.383900 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.383878 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-66.ec2.internal" event="NodeHasSufficientPID" Sep 09 14:33:05.384477 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:05.384414 2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-66.ec2.internal.18d3ada1528e3199 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-66.ec2.internal,UID:ip-10-0-133-66.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-133-66.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-133-66.ec2.internal,},FirstTimestamp:2026-09-09 14:33:05.382609305 +0000 UTC m=+0.393525548,LastTimestamp:2026-09-09 14:33:05.382609305 +0000 UTC m=+0.393525548,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-66.ec2.internal,}" Sep 09 14:33:05.385570 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.385559 2576 policy_none.go:49] "None policy: Start" Sep 09 14:33:05.385611 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.385575 2576 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 09 14:33:05.385611 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.385587 2576 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 09 14:33:05.387539 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.387528 2576 policy_none.go:47] "Start" Sep 09 14:33:05.394813 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:05.394747 2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-66.ec2.internal.18d3ada1528e7661 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-66.ec2.internal,UID:ip-10-0-133-66.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-133-66.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-133-66.ec2.internal,},FirstTimestamp:2026-09-09 14:33:05.382626913 +0000 UTC m=+0.393543155,LastTimestamp:2026-09-09 14:33:05.382626913 +0000 UTC m=+0.393543155,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-66.ec2.internal,}" Sep 09 14:33:05.397164 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:05.397107 2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-66.ec2.internal.18d3ada1528e98b5 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-66.ec2.internal,UID:ip-10-0-133-66.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-133-66.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-133-66.ec2.internal,},FirstTimestamp:2026-09-09 14:33:05.382635701 +0000 UTC m=+0.393551943,LastTimestamp:2026-09-09 14:33:05.382635701 +0000 UTC m=+0.393551943,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-66.ec2.internal,}" Sep 09 14:33:05.402624 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.402607 2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-x6rsw" Sep 09 14:33:05.402767 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:05.402686 2576 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-133-66.ec2.internal.18d3ada1528e3199\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-66.ec2.internal.18d3ada1528e3199 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-66.ec2.internal,UID:ip-10-0-133-66.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-133-66.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-133-66.ec2.internal,},FirstTimestamp:2026-09-09 14:33:05.382609305 +0000 UTC m=+0.393525548,LastTimestamp:2026-09-09 14:33:05.383852235 +0000 UTC m=+0.394768481,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-66.ec2.internal,}" Sep 09 14:33:05.406985 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:05.406909 2576 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-133-66.ec2.internal.18d3ada1528e7661\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-66.ec2.internal.18d3ada1528e7661 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-66.ec2.internal,UID:ip-10-0-133-66.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-133-66.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-133-66.ec2.internal,},FirstTimestamp:2026-09-09 14:33:05.382626913 +0000 UTC m=+0.393543155,LastTimestamp:2026-09-09 14:33:05.383872151 +0000 UTC m=+0.394788393,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-66.ec2.internal,}" Sep 09 14:33:05.409163 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.409145 2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-x6rsw" Sep 09 14:33:05.411570 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:05.411488 2576 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-133-66.ec2.internal.18d3ada1528e98b5\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-66.ec2.internal.18d3ada1528e98b5 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-66.ec2.internal,UID:ip-10-0-133-66.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-133-66.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-133-66.ec2.internal,},FirstTimestamp:2026-09-09 14:33:05.382635701 +0000 UTC m=+0.393551943,LastTimestamp:2026-09-09 14:33:05.383881503 +0000 UTC m=+0.394797744,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-66.ec2.internal,}" Sep 09 14:33:05.431078 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.430913 2576 manager.go:337] "Starting Device Plugin manager" Sep 09 14:33:05.445618 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:05.431103 2576 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 09 14:33:05.445618 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.431124 2576 server.go:87] "Starting device plugin registration server" Sep 09 14:33:05.445618 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.431322 2576 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 09 14:33:05.445618 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.431332 2576 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 09 14:33:05.445618 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.431439 2576 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 09 14:33:05.445618 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.431516 2576 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 09 14:33:05.445618 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.431524 2576 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 09 14:33:05.445618 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:05.432053 2576 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 09 14:33:05.445618 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:05.432102 2576 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-66.ec2.internal\" not found" Sep 09 14:33:05.507572 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.507545 2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 09 14:33:05.508856 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.508838 2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 09 14:33:05.508945 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.508883 2576 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 09 14:33:05.508945 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.508902 2576 kubelet.go:2448] "Starting kubelet main sync loop" Sep 09 14:33:05.509023 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:05.508945 2576 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 09 14:33:05.511160 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.511140 2576 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 09 14:33:05.532121 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.532103 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 09 14:33:05.532875 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.532861 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-66.ec2.internal" event="NodeHasSufficientMemory" Sep 09 14:33:05.532950 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.532888 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-66.ec2.internal" event="NodeHasNoDiskPressure" Sep 09 14:33:05.532950 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.532899 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-66.ec2.internal" event="NodeHasSufficientPID" Sep 09 14:33:05.532950 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.532926 2576 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-133-66.ec2.internal" Sep 09 14:33:05.539719 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.539684 2576 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-133-66.ec2.internal" Sep 09 14:33:05.539719 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:05.539703 2576 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-66.ec2.internal\": node \"ip-10-0-133-66.ec2.internal\" not found" Sep 09 14:33:05.573587 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:05.573569 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-66.ec2.internal\" not found" Sep 09 14:33:05.610002 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.609982 2576 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-66.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-66.ec2.internal"] Sep 09 14:33:05.610069 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.610055 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 09 14:33:05.611266 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.611251 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-66.ec2.internal" event="NodeHasSufficientMemory" Sep 09 14:33:05.611349 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.611275 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-66.ec2.internal" event="NodeHasNoDiskPressure" Sep 09 14:33:05.611349 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.611284 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-66.ec2.internal" event="NodeHasSufficientPID" Sep 09 14:33:05.613544 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.613533 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 09 14:33:05.613678 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.613657 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-66.ec2.internal" Sep 09 14:33:05.613726 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.613687 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 09 14:33:05.614286 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.614268 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-66.ec2.internal" event="NodeHasSufficientMemory" Sep 09 14:33:05.614373 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.614313 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-66.ec2.internal" event="NodeHasNoDiskPressure" Sep 09 14:33:05.614373 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.614271 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-66.ec2.internal" event="NodeHasSufficientMemory" Sep 09 14:33:05.614373 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.614340 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-66.ec2.internal" event="NodeHasNoDiskPressure" Sep 09 14:33:05.614373 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.614349 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-66.ec2.internal" event="NodeHasSufficientPID" Sep 09 14:33:05.614491 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.614324 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-66.ec2.internal" event="NodeHasSufficientPID" Sep 09 14:33:05.616609 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.616596 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-66.ec2.internal" Sep 09 14:33:05.616679 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.616618 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 09 14:33:05.617263 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.617246 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-66.ec2.internal" event="NodeHasSufficientMemory" Sep 09 14:33:05.617345 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.617275 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-66.ec2.internal" event="NodeHasNoDiskPressure" Sep 09 14:33:05.617345 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.617291 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-66.ec2.internal" event="NodeHasSufficientPID" Sep 09 14:33:05.631807 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:05.631785 2576 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-66.ec2.internal\" not found" node="ip-10-0-133-66.ec2.internal" Sep 09 14:33:05.635476 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:05.635455 2576 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-66.ec2.internal\" not found" node="ip-10-0-133-66.ec2.internal" Sep 09 14:33:05.665908 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.665885 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9af3564e64deec6f121c01d714dc0e5a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-66.ec2.internal\" (UID: \"9af3564e64deec6f121c01d714dc0e5a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-66.ec2.internal" Sep 09 14:33:05.666040 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.665912 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3aba2b4456e66055b3846a53f8393ab4-config\") pod \"kube-apiserver-proxy-ip-10-0-133-66.ec2.internal\" (UID: \"3aba2b4456e66055b3846a53f8393ab4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-66.ec2.internal" Sep 09 14:33:05.666040 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.665929 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9af3564e64deec6f121c01d714dc0e5a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-66.ec2.internal\" (UID: \"9af3564e64deec6f121c01d714dc0e5a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-66.ec2.internal" Sep 09 14:33:05.674079 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:05.674059 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-66.ec2.internal\" not found" Sep 09 14:33:05.766260 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.766215 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9af3564e64deec6f121c01d714dc0e5a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-66.ec2.internal\" (UID: \"9af3564e64deec6f121c01d714dc0e5a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-66.ec2.internal" Sep 09 14:33:05.766441 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.766272 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9af3564e64deec6f121c01d714dc0e5a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-66.ec2.internal\" (UID: \"9af3564e64deec6f121c01d714dc0e5a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-66.ec2.internal" Sep 09 14:33:05.766441 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.766286 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9af3564e64deec6f121c01d714dc0e5a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-66.ec2.internal\" (UID: \"9af3564e64deec6f121c01d714dc0e5a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-66.ec2.internal" Sep 09 14:33:05.766441 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.766319 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9af3564e64deec6f121c01d714dc0e5a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-66.ec2.internal\" (UID: \"9af3564e64deec6f121c01d714dc0e5a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-66.ec2.internal" Sep 09 14:33:05.766441 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.766352 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3aba2b4456e66055b3846a53f8393ab4-config\") pod \"kube-apiserver-proxy-ip-10-0-133-66.ec2.internal\" (UID: \"3aba2b4456e66055b3846a53f8393ab4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-66.ec2.internal" Sep 09 14:33:05.766441 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.766372 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3aba2b4456e66055b3846a53f8393ab4-config\") pod \"kube-apiserver-proxy-ip-10-0-133-66.ec2.internal\" (UID: \"3aba2b4456e66055b3846a53f8393ab4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-66.ec2.internal" Sep 09 14:33:05.774357 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:05.774333 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-66.ec2.internal\" not found" Sep 09 14:33:05.875136 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:05.875058 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-66.ec2.internal\" not found" Sep 09 14:33:05.934269 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.934239 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-66.ec2.internal" Sep 09 14:33:05.937894 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:05.937877 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-66.ec2.internal" Sep 09 14:33:05.975750 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:05.975724 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-66.ec2.internal\" not found" Sep 09 14:33:06.076354 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:06.076321 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-66.ec2.internal\" not found" Sep 09 14:33:06.176835 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:06.176770 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-66.ec2.internal\" not found" Sep 09 14:33:06.277448 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:06.277418 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-66.ec2.internal\" not found" Sep 09 14:33:06.302591 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:06.302569 2576 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 09 14:33:06.302998 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:06.302751 2576 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 09 14:33:06.378346 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:06.378321 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-66.ec2.internal\" not found" Sep 09 14:33:06.379339 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:06.379324 2576 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 09 14:33:06.414014 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:06.413980 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-08 14:28:05 +0000 UTC" deadline="2028-05-11 06:11:08.354211864 +0000 UTC" Sep 09 14:33:06.414014 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:06.414008 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14631h38m1.940207519s" Sep 09 14:33:06.464364 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:06.464314 2576 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-66.ec2.internal" Sep 09 14:33:06.486442 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:06.486418 2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 09 14:33:06.488394 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:06.488379 2576 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-66.ec2.internal" Sep 09 14:33:06.506401 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:06.506377 2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 09 14:33:06.524737 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:06.524718 2576 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 09 14:33:06.675686 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:06.675651 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9af3564e64deec6f121c01d714dc0e5a.slice/crio-82cb097fd8ea93dcee7d3f3fac587e222e771372bdb95ec3b18570c47942f27b WatchSource:0}: Error finding container 82cb097fd8ea93dcee7d3f3fac587e222e771372bdb95ec3b18570c47942f27b: Status 404 returned error can't find the container with id 82cb097fd8ea93dcee7d3f3fac587e222e771372bdb95ec3b18570c47942f27b Sep 09 14:33:06.676081 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:06.676063 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3aba2b4456e66055b3846a53f8393ab4.slice/crio-4a2e76da54ef5a9313e49f92b3913a64cf0bac9fbb4500f9faf6ac21aa58fa88 WatchSource:0}: Error finding container 4a2e76da54ef5a9313e49f92b3913a64cf0bac9fbb4500f9faf6ac21aa58fa88: Status 404 returned error can't find the container with id 4a2e76da54ef5a9313e49f92b3913a64cf0bac9fbb4500f9faf6ac21aa58fa88 Sep 09 14:33:06.681120 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:06.681100 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 09 14:33:06.841268 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:06.841184 2576 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 09 14:33:07.343906 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.343876 2576 apiserver.go:52] "Watching apiserver" Sep 09 14:33:07.352437 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.352412 2576 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 09 14:33:07.352837 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.352809 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-nqc8q","kube-system/konnectivity-agent-ppvhx","kube-system/kube-apiserver-proxy-ip-10-0-133-66.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qlxsx","openshift-cluster-node-tuning-operator/tuned-p5wbc","openshift-dns/node-resolver-5qkp6","openshift-image-registry/node-ca-cqf8f","openshift-multus/network-metrics-daemon-wcj2p","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-66.ec2.internal","openshift-multus/multus-additional-cni-plugins-p5bsf","openshift-multus/multus-w5kx5","openshift-network-diagnostics/network-check-target-5vdl5","openshift-network-operator/iptables-alerter-4kp4s","openshift-ovn-kubernetes/ovnkube-node-7xcn5"] Sep 09 14:33:07.355063 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.355043 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-cqf8f" Sep 09 14:33:07.357083 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.357061 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-4kp4s" Sep 09 14:33:07.357665 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.357645 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 09 14:33:07.357665 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.357659 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-hgl8q\"" Sep 09 14:33:07.357817 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.357651 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 09 14:33:07.357817 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.357664 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 09 14:33:07.359237 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.359219 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-7xcn5" Sep 09 14:33:07.359488 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.359468 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 09 14:33:07.359735 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.359710 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 09 14:33:07.359932 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.359913 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 09 14:33:07.359996 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.359978 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-rwmxh\"" Sep 09 14:33:07.361390 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.361371 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qlxsx" Sep 09 14:33:07.361986 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.361970 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 09 14:33:07.362173 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.362158 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 09 14:33:07.362234 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.362192 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-fbvmh\"" Sep 09 14:33:07.362391 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.362366 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 09 14:33:07.362870 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.362852 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 09 14:33:07.362935 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.362882 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 09 14:33:07.362935 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.362899 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 09 14:33:07.363386 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.363370 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 09 14:33:07.363617 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.363598 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-p5wbc" Sep 09 14:33:07.363905 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.363890 2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 09 14:33:07.364164 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.364136 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-5lfhj\"" Sep 09 14:33:07.364253 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.364212 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 09 14:33:07.364253 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.364236 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 09 14:33:07.366201 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.365673 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5qkp6" Sep 09 14:33:07.367078 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.367056 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 09 14:33:07.367285 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.367269 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 09 14:33:07.367509 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.367496 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-xj9w8\"" Sep 09 14:33:07.367958 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.367943 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 09 14:33:07.368825 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.368805 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 09 14:33:07.369078 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.369042 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-js2ts\"" Sep 09 14:33:07.369440 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.369337 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wcj2p" Sep 09 14:33:07.369508 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:07.369444 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wcj2p" podUID="ab36c1e4-a65c-4cf9-ada4-f8a87d4b463b" Sep 09 14:33:07.373123 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.373102 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nqc8q" Sep 09 14:33:07.373231 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:07.373199 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nqc8q" podUID="f286fc80-41f7-4b6f-8506-b5f0eb2629d3" Sep 09 14:33:07.373722 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.373703 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5a445f65-2ccc-44f4-97ad-1eee2597b711-device-dir\") pod \"aws-ebs-csi-driver-node-qlxsx\" (UID: \"5a445f65-2ccc-44f4-97ad-1eee2597b711\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qlxsx" Sep 09 14:33:07.373823 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.373734 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/186aab90-266b-4eae-977a-3f053b423ef3-host-kubelet\") pod \"ovnkube-node-7xcn5\" (UID: \"186aab90-266b-4eae-977a-3f053b423ef3\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xcn5" Sep 09 14:33:07.373823 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.373748 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/186aab90-266b-4eae-977a-3f053b423ef3-systemd-units\") pod \"ovnkube-node-7xcn5\" (UID: \"186aab90-266b-4eae-977a-3f053b423ef3\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xcn5" Sep 09 14:33:07.373823 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.373763 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/186aab90-266b-4eae-977a-3f053b423ef3-etc-openvswitch\") pod \"ovnkube-node-7xcn5\" (UID: \"186aab90-266b-4eae-977a-3f053b423ef3\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xcn5" Sep 09 14:33:07.373823 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.373790 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/186aab90-266b-4eae-977a-3f053b423ef3-host-run-ovn-kubernetes\") pod \"ovnkube-node-7xcn5\" (UID: \"186aab90-266b-4eae-977a-3f053b423ef3\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xcn5" Sep 09 14:33:07.373823 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.373814 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/186aab90-266b-4eae-977a-3f053b423ef3-ovn-node-metrics-cert\") pod \"ovnkube-node-7xcn5\" (UID: \"186aab90-266b-4eae-977a-3f053b423ef3\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xcn5" Sep 09 14:33:07.374064 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.373833 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/186aab90-266b-4eae-977a-3f053b423ef3-ovnkube-script-lib\") pod \"ovnkube-node-7xcn5\" (UID: \"186aab90-266b-4eae-977a-3f053b423ef3\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xcn5" Sep 09 14:33:07.374064 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.373853 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5a445f65-2ccc-44f4-97ad-1eee2597b711-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qlxsx\" (UID: \"5a445f65-2ccc-44f4-97ad-1eee2597b711\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qlxsx" Sep 09 14:33:07.374064 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.373869 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/186aab90-266b-4eae-977a-3f053b423ef3-log-socket\") pod \"ovnkube-node-7xcn5\" (UID: \"186aab90-266b-4eae-977a-3f053b423ef3\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xcn5" Sep 09 14:33:07.374064 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.373882 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5a445f65-2ccc-44f4-97ad-1eee2597b711-socket-dir\") pod \"aws-ebs-csi-driver-node-qlxsx\" (UID: \"5a445f65-2ccc-44f4-97ad-1eee2597b711\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qlxsx" Sep 09 14:33:07.374064 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.373896 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/50c734e0-544e-44f4-9c79-a398e559a276-etc-sysconfig\") pod \"tuned-p5wbc\" (UID: \"50c734e0-544e-44f4-9c79-a398e559a276\") " pod="openshift-cluster-node-tuning-operator/tuned-p5wbc" Sep 09 14:33:07.374064 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.373925 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/50c734e0-544e-44f4-9c79-a398e559a276-etc-sysctl-d\") pod \"tuned-p5wbc\" (UID: \"50c734e0-544e-44f4-9c79-a398e559a276\") " pod="openshift-cluster-node-tuning-operator/tuned-p5wbc" Sep 09 14:33:07.374064 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.373946 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/50c734e0-544e-44f4-9c79-a398e559a276-etc-sysctl-conf\") pod \"tuned-p5wbc\" (UID: \"50c734e0-544e-44f4-9c79-a398e559a276\") " pod="openshift-cluster-node-tuning-operator/tuned-p5wbc" Sep 09 14:33:07.374064 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.373962 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/50c734e0-544e-44f4-9c79-a398e559a276-etc-systemd\") pod \"tuned-p5wbc\" (UID: \"50c734e0-544e-44f4-9c79-a398e559a276\") " pod="openshift-cluster-node-tuning-operator/tuned-p5wbc" Sep 09 14:33:07.374064 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.374000 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/50c734e0-544e-44f4-9c79-a398e559a276-host\") pod \"tuned-p5wbc\" (UID: \"50c734e0-544e-44f4-9c79-a398e559a276\") " pod="openshift-cluster-node-tuning-operator/tuned-p5wbc" Sep 09 14:33:07.374064 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.374027 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/186aab90-266b-4eae-977a-3f053b423ef3-run-ovn\") pod \"ovnkube-node-7xcn5\" (UID: \"186aab90-266b-4eae-977a-3f053b423ef3\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xcn5" Sep 09 14:33:07.374064 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.374055 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/186aab90-266b-4eae-977a-3f053b423ef3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7xcn5\" (UID: \"186aab90-266b-4eae-977a-3f053b423ef3\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xcn5" Sep 09 14:33:07.374584 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.374078 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5a445f65-2ccc-44f4-97ad-1eee2597b711-etc-selinux\") pod \"aws-ebs-csi-driver-node-qlxsx\" (UID: \"5a445f65-2ccc-44f4-97ad-1eee2597b711\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qlxsx" Sep 09 14:33:07.374584 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.374123 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5a445f65-2ccc-44f4-97ad-1eee2597b711-sys-fs\") pod \"aws-ebs-csi-driver-node-qlxsx\" (UID: \"5a445f65-2ccc-44f4-97ad-1eee2597b711\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qlxsx" Sep 09 14:33:07.374584 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.374145 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxqjx\" (UniqueName: \"kubernetes.io/projected/5a445f65-2ccc-44f4-97ad-1eee2597b711-kube-api-access-sxqjx\") pod \"aws-ebs-csi-driver-node-qlxsx\" (UID: \"5a445f65-2ccc-44f4-97ad-1eee2597b711\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qlxsx" Sep 09 14:33:07.374584 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.374163 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jqcd\" (UniqueName: \"kubernetes.io/projected/50c734e0-544e-44f4-9c79-a398e559a276-kube-api-access-7jqcd\") pod \"tuned-p5wbc\" (UID: \"50c734e0-544e-44f4-9c79-a398e559a276\") " pod="openshift-cluster-node-tuning-operator/tuned-p5wbc" Sep 09 14:33:07.374584 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.374184 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b134513f-e0e7-452d-8b61-bae4c9007795-iptables-alerter-script\") pod \"iptables-alerter-4kp4s\" (UID: \"b134513f-e0e7-452d-8b61-bae4c9007795\") " pod="openshift-network-operator/iptables-alerter-4kp4s" Sep 09 14:33:07.374584 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.374216 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/186aab90-266b-4eae-977a-3f053b423ef3-node-log\") pod \"ovnkube-node-7xcn5\" (UID: \"186aab90-266b-4eae-977a-3f053b423ef3\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xcn5" Sep 09 14:33:07.374584 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.374247 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/186aab90-266b-4eae-977a-3f053b423ef3-env-overrides\") pod \"ovnkube-node-7xcn5\" (UID: \"186aab90-266b-4eae-977a-3f053b423ef3\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xcn5" Sep 09 14:33:07.374584 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.374291 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5a445f65-2ccc-44f4-97ad-1eee2597b711-registration-dir\") pod \"aws-ebs-csi-driver-node-qlxsx\" (UID: \"5a445f65-2ccc-44f4-97ad-1eee2597b711\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qlxsx" Sep 09 14:33:07.374584 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.374468 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/50c734e0-544e-44f4-9c79-a398e559a276-etc-modprobe-d\") pod \"tuned-p5wbc\" (UID: \"50c734e0-544e-44f4-9c79-a398e559a276\") " pod="openshift-cluster-node-tuning-operator/tuned-p5wbc" Sep 09 14:33:07.374584 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.374490 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/50c734e0-544e-44f4-9c79-a398e559a276-lib-modules\") pod \"tuned-p5wbc\" (UID: \"50c734e0-544e-44f4-9c79-a398e559a276\") " pod="openshift-cluster-node-tuning-operator/tuned-p5wbc" Sep 09 14:33:07.374584 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.374510 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/50c734e0-544e-44f4-9c79-a398e559a276-etc-tuned\") pod \"tuned-p5wbc\" (UID: \"50c734e0-544e-44f4-9c79-a398e559a276\") " pod="openshift-cluster-node-tuning-operator/tuned-p5wbc" Sep 09 14:33:07.374584 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.374529 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/90785a55-a3bd-41a7-8d33-85e3b82eb41f-hosts-file\") pod \"node-resolver-5qkp6\" (UID: \"90785a55-a3bd-41a7-8d33-85e3b82eb41f\") " pod="openshift-dns/node-resolver-5qkp6" Sep 09 14:33:07.374584 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.374550 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/186aab90-266b-4eae-977a-3f053b423ef3-run-openvswitch\") pod \"ovnkube-node-7xcn5\" (UID: \"186aab90-266b-4eae-977a-3f053b423ef3\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xcn5" Sep 09 14:33:07.374584 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.374576 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/186aab90-266b-4eae-977a-3f053b423ef3-ovnkube-config\") pod \"ovnkube-node-7xcn5\" (UID: \"186aab90-266b-4eae-977a-3f053b423ef3\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xcn5" Sep 09 14:33:07.375103 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.374601 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pn6xh\" (UniqueName: \"kubernetes.io/projected/90785a55-a3bd-41a7-8d33-85e3b82eb41f-kube-api-access-pn6xh\") pod \"node-resolver-5qkp6\" (UID: \"90785a55-a3bd-41a7-8d33-85e3b82eb41f\") " pod="openshift-dns/node-resolver-5qkp6" Sep 09 14:33:07.375103 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.374622 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/186aab90-266b-4eae-977a-3f053b423ef3-host-cni-bin\") pod \"ovnkube-node-7xcn5\" (UID: \"186aab90-266b-4eae-977a-3f053b423ef3\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xcn5" Sep 09 14:33:07.375103 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.374642 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/186aab90-266b-4eae-977a-3f053b423ef3-host-cni-netd\") pod \"ovnkube-node-7xcn5\" (UID: \"186aab90-266b-4eae-977a-3f053b423ef3\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xcn5" Sep 09 14:33:07.375103 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.374697 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlvzg\" (UniqueName: \"kubernetes.io/projected/186aab90-266b-4eae-977a-3f053b423ef3-kube-api-access-dlvzg\") pod \"ovnkube-node-7xcn5\" (UID: \"186aab90-266b-4eae-977a-3f053b423ef3\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xcn5" Sep 09 14:33:07.375103 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.374730 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/50c734e0-544e-44f4-9c79-a398e559a276-etc-kubernetes\") pod \"tuned-p5wbc\" (UID: \"50c734e0-544e-44f4-9c79-a398e559a276\") " pod="openshift-cluster-node-tuning-operator/tuned-p5wbc" Sep 09 14:33:07.375103 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.374756 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/50c734e0-544e-44f4-9c79-a398e559a276-run\") pod \"tuned-p5wbc\" (UID: \"50c734e0-544e-44f4-9c79-a398e559a276\") " pod="openshift-cluster-node-tuning-operator/tuned-p5wbc" Sep 09 14:33:07.375103 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.374781 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/50c734e0-544e-44f4-9c79-a398e559a276-var-lib-kubelet\") pod \"tuned-p5wbc\" (UID: \"50c734e0-544e-44f4-9c79-a398e559a276\") " pod="openshift-cluster-node-tuning-operator/tuned-p5wbc" Sep 09 14:33:07.375103 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.374806 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/edb6129a-6cbf-4d83-82b0-018fd0e73f8f-host\") pod \"node-ca-cqf8f\" (UID: \"edb6129a-6cbf-4d83-82b0-018fd0e73f8f\") " pod="openshift-image-registry/node-ca-cqf8f" Sep 09 14:33:07.375103 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.374826 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/186aab90-266b-4eae-977a-3f053b423ef3-host-slash\") pod \"ovnkube-node-7xcn5\" (UID: \"186aab90-266b-4eae-977a-3f053b423ef3\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xcn5" Sep 09 14:33:07.375103 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.374850 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/50c734e0-544e-44f4-9c79-a398e559a276-sys\") pod \"tuned-p5wbc\" (UID: \"50c734e0-544e-44f4-9c79-a398e559a276\") " pod="openshift-cluster-node-tuning-operator/tuned-p5wbc" Sep 09 14:33:07.375103 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.374874 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/50c734e0-544e-44f4-9c79-a398e559a276-tmp\") pod \"tuned-p5wbc\" (UID: \"50c734e0-544e-44f4-9c79-a398e559a276\") " pod="openshift-cluster-node-tuning-operator/tuned-p5wbc" Sep 09 14:33:07.375103 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.374903 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90785a55-a3bd-41a7-8d33-85e3b82eb41f-tmp-dir\") pod \"node-resolver-5qkp6\" (UID: \"90785a55-a3bd-41a7-8d33-85e3b82eb41f\") " pod="openshift-dns/node-resolver-5qkp6" Sep 09 14:33:07.375103 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.374999 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57mc4\" (UniqueName: \"kubernetes.io/projected/edb6129a-6cbf-4d83-82b0-018fd0e73f8f-kube-api-access-57mc4\") pod \"node-ca-cqf8f\" (UID: \"edb6129a-6cbf-4d83-82b0-018fd0e73f8f\") " pod="openshift-image-registry/node-ca-cqf8f" Sep 09 14:33:07.375103 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.375014 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/186aab90-266b-4eae-977a-3f053b423ef3-host-run-netns\") pod \"ovnkube-node-7xcn5\" (UID: \"186aab90-266b-4eae-977a-3f053b423ef3\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xcn5" Sep 09 14:33:07.375103 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.375026 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/186aab90-266b-4eae-977a-3f053b423ef3-run-systemd\") pod \"ovnkube-node-7xcn5\" (UID: \"186aab90-266b-4eae-977a-3f053b423ef3\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xcn5" Sep 09 14:33:07.375103 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.375039 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/186aab90-266b-4eae-977a-3f053b423ef3-var-lib-openvswitch\") pod \"ovnkube-node-7xcn5\" (UID: \"186aab90-266b-4eae-977a-3f053b423ef3\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xcn5" Sep 09 14:33:07.375103 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.375061 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/edb6129a-6cbf-4d83-82b0-018fd0e73f8f-serviceca\") pod \"node-ca-cqf8f\" (UID: \"edb6129a-6cbf-4d83-82b0-018fd0e73f8f\") " pod="openshift-image-registry/node-ca-cqf8f" Sep 09 14:33:07.375818 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.375090 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b134513f-e0e7-452d-8b61-bae4c9007795-host-slash\") pod \"iptables-alerter-4kp4s\" (UID: \"b134513f-e0e7-452d-8b61-bae4c9007795\") " pod="openshift-network-operator/iptables-alerter-4kp4s" Sep 09 14:33:07.375818 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.375115 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27fsb\" (UniqueName: \"kubernetes.io/projected/b134513f-e0e7-452d-8b61-bae4c9007795-kube-api-access-27fsb\") pod \"iptables-alerter-4kp4s\" (UID: \"b134513f-e0e7-452d-8b61-bae4c9007795\") " pod="openshift-network-operator/iptables-alerter-4kp4s" Sep 09 14:33:07.375818 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.375764 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-ppvhx" Sep 09 14:33:07.376000 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.375980 2576 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 09 14:33:07.377709 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.377691 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-xqq5c\"" Sep 09 14:33:07.378330 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.378157 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 09 14:33:07.378330 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.378157 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 09 14:33:07.378446 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.378427 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-p5bsf" Sep 09 14:33:07.380535 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.380515 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 09 14:33:07.380691 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.380676 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-w5kx5" Sep 09 14:33:07.380863 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.380845 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 09 14:33:07.381157 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.381016 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 09 14:33:07.381157 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.381110 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 09 14:33:07.381157 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.381142 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 09 14:33:07.381457 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.381444 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-jcmvs\"" Sep 09 14:33:07.383268 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.382801 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 09 14:33:07.383268 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.383019 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5vdl5" Sep 09 14:33:07.383268 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:07.383094 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5vdl5" podUID="ce063353-8e00-4701-8fb5-9b32aa0586e1" Sep 09 14:33:07.383268 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.383124 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-mdlvt\"" Sep 09 14:33:07.395509 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.395489 2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-8wzns" Sep 09 14:33:07.405241 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.405218 2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-8wzns" Sep 09 14:33:07.466236 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.465941 2576 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 09 14:33:07.475638 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.475605 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/186aab90-266b-4eae-977a-3f053b423ef3-systemd-units\") pod \"ovnkube-node-7xcn5\" (UID: \"186aab90-266b-4eae-977a-3f053b423ef3\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xcn5" Sep 09 14:33:07.475816 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.475648 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/186aab90-266b-4eae-977a-3f053b423ef3-etc-openvswitch\") pod \"ovnkube-node-7xcn5\" (UID: \"186aab90-266b-4eae-977a-3f053b423ef3\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xcn5" Sep 09 14:33:07.475816 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.475677 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/50c734e0-544e-44f4-9c79-a398e559a276-etc-systemd\") pod \"tuned-p5wbc\" (UID: \"50c734e0-544e-44f4-9c79-a398e559a276\") " pod="openshift-cluster-node-tuning-operator/tuned-p5wbc" Sep 09 14:33:07.475816 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.475699 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/50c734e0-544e-44f4-9c79-a398e559a276-host\") pod \"tuned-p5wbc\" (UID: \"50c734e0-544e-44f4-9c79-a398e559a276\") " pod="openshift-cluster-node-tuning-operator/tuned-p5wbc" Sep 09 14:33:07.475816 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.475719 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/50c734e0-544e-44f4-9c79-a398e559a276-etc-tuned\") pod \"tuned-p5wbc\" (UID: \"50c734e0-544e-44f4-9c79-a398e559a276\") " pod="openshift-cluster-node-tuning-operator/tuned-p5wbc" Sep 09 14:33:07.475816 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.475744 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ab41acab-1926-436b-b208-c18b0b375e9f-cnibin\") pod \"multus-w5kx5\" (UID: \"ab41acab-1926-436b-b208-c18b0b375e9f\") " pod="openshift-multus/multus-w5kx5" Sep 09 14:33:07.475816 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.475766 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/186aab90-266b-4eae-977a-3f053b423ef3-log-socket\") pod \"ovnkube-node-7xcn5\" (UID: \"186aab90-266b-4eae-977a-3f053b423ef3\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xcn5" Sep 09 14:33:07.475816 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.475787 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5a445f65-2ccc-44f4-97ad-1eee2597b711-socket-dir\") pod \"aws-ebs-csi-driver-node-qlxsx\" (UID: \"5a445f65-2ccc-44f4-97ad-1eee2597b711\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qlxsx" Sep 09 14:33:07.475816 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.475811 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/50c734e0-544e-44f4-9c79-a398e559a276-etc-sysconfig\") pod \"tuned-p5wbc\" (UID: \"50c734e0-544e-44f4-9c79-a398e559a276\") " pod="openshift-cluster-node-tuning-operator/tuned-p5wbc" Sep 09 14:33:07.476254 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.475818 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/50c734e0-544e-44f4-9c79-a398e559a276-etc-systemd\") pod \"tuned-p5wbc\" (UID: \"50c734e0-544e-44f4-9c79-a398e559a276\") " pod="openshift-cluster-node-tuning-operator/tuned-p5wbc" Sep 09 14:33:07.476254 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.475831 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/50c734e0-544e-44f4-9c79-a398e559a276-etc-sysctl-d\") pod \"tuned-p5wbc\" (UID: \"50c734e0-544e-44f4-9c79-a398e559a276\") " pod="openshift-cluster-node-tuning-operator/tuned-p5wbc" Sep 09 14:33:07.476254 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.475853 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ab41acab-1926-436b-b208-c18b0b375e9f-host-var-lib-cni-multus\") pod \"multus-w5kx5\" (UID: \"ab41acab-1926-436b-b208-c18b0b375e9f\") " pod="openshift-multus/multus-w5kx5" Sep 09 14:33:07.476254 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.475870 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/186aab90-266b-4eae-977a-3f053b423ef3-systemd-units\") pod \"ovnkube-node-7xcn5\" (UID: \"186aab90-266b-4eae-977a-3f053b423ef3\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xcn5" Sep 09 14:33:07.476254 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.475876 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ab41acab-1926-436b-b208-c18b0b375e9f-host-var-lib-kubelet\") pod \"multus-w5kx5\" (UID: \"ab41acab-1926-436b-b208-c18b0b375e9f\") " pod="openshift-multus/multus-w5kx5" Sep 09 14:33:07.476254 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.475896 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/186aab90-266b-4eae-977a-3f053b423ef3-etc-openvswitch\") pod \"ovnkube-node-7xcn5\" (UID: \"186aab90-266b-4eae-977a-3f053b423ef3\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xcn5" Sep 09 14:33:07.476254 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.475900 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcsz5\" (UniqueName: \"kubernetes.io/projected/ab36c1e4-a65c-4cf9-ada4-f8a87d4b463b-kube-api-access-tcsz5\") pod \"network-metrics-daemon-wcj2p\" (UID: \"ab36c1e4-a65c-4cf9-ada4-f8a87d4b463b\") " pod="openshift-multus/network-metrics-daemon-wcj2p" Sep 09 14:33:07.476254 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.475924 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f286fc80-41f7-4b6f-8506-b5f0eb2629d3-dbus\") pod \"global-pull-secret-syncer-nqc8q\" (UID: \"f286fc80-41f7-4b6f-8506-b5f0eb2629d3\") " pod="kube-system/global-pull-secret-syncer-nqc8q" Sep 09 14:33:07.476254 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.475952 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5a445f65-2ccc-44f4-97ad-1eee2597b711-etc-selinux\") pod \"aws-ebs-csi-driver-node-qlxsx\" (UID: \"5a445f65-2ccc-44f4-97ad-1eee2597b711\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qlxsx" Sep 09 14:33:07.476254 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.475968 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxqjx\" (UniqueName: \"kubernetes.io/projected/5a445f65-2ccc-44f4-97ad-1eee2597b711-kube-api-access-sxqjx\") pod \"aws-ebs-csi-driver-node-qlxsx\" (UID: \"5a445f65-2ccc-44f4-97ad-1eee2597b711\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qlxsx" Sep 09 14:33:07.476254 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.475982 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ab41acab-1926-436b-b208-c18b0b375e9f-host-run-k8s-cni-cncf-io\") pod \"multus-w5kx5\" (UID: \"ab41acab-1926-436b-b208-c18b0b375e9f\") " pod="openshift-multus/multus-w5kx5" Sep 09 14:33:07.476254 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.475997 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab36c1e4-a65c-4cf9-ada4-f8a87d4b463b-metrics-certs\") pod \"network-metrics-daemon-wcj2p\" (UID: \"ab36c1e4-a65c-4cf9-ada4-f8a87d4b463b\") " pod="openshift-multus/network-metrics-daemon-wcj2p" Sep 09 14:33:07.476254 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.475998 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5a445f65-2ccc-44f4-97ad-1eee2597b711-socket-dir\") pod \"aws-ebs-csi-driver-node-qlxsx\" (UID: \"5a445f65-2ccc-44f4-97ad-1eee2597b711\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qlxsx" Sep 09 14:33:07.476254 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.476024 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3c7d6121-a834-42d0-9b78-583260fd4dce-system-cni-dir\") pod \"multus-additional-cni-plugins-p5bsf\" (UID: \"3c7d6121-a834-42d0-9b78-583260fd4dce\") " pod="openshift-multus/multus-additional-cni-plugins-p5bsf" Sep 09 14:33:07.476254 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.476045 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3c7d6121-a834-42d0-9b78-583260fd4dce-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-p5bsf\" (UID: \"3c7d6121-a834-42d0-9b78-583260fd4dce\") " pod="openshift-multus/multus-additional-cni-plugins-p5bsf" Sep 09 14:33:07.476254 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.476055 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/50c734e0-544e-44f4-9c79-a398e559a276-etc-sysconfig\") pod \"tuned-p5wbc\" (UID: \"50c734e0-544e-44f4-9c79-a398e559a276\") " pod="openshift-cluster-node-tuning-operator/tuned-p5wbc" Sep 09 14:33:07.476254 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.476072 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/186aab90-266b-4eae-977a-3f053b423ef3-node-log\") pod \"ovnkube-node-7xcn5\" (UID: \"186aab90-266b-4eae-977a-3f053b423ef3\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xcn5" Sep 09 14:33:07.477054 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.476087 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/50c734e0-544e-44f4-9c79-a398e559a276-etc-modprobe-d\") pod \"tuned-p5wbc\" (UID: \"50c734e0-544e-44f4-9c79-a398e559a276\") " pod="openshift-cluster-node-tuning-operator/tuned-p5wbc" Sep 09 14:33:07.477054 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.476108 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/50c734e0-544e-44f4-9c79-a398e559a276-lib-modules\") pod \"tuned-p5wbc\" (UID: \"50c734e0-544e-44f4-9c79-a398e559a276\") " pod="openshift-cluster-node-tuning-operator/tuned-p5wbc" Sep 09 14:33:07.477054 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.476134 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ab41acab-1926-436b-b208-c18b0b375e9f-multus-socket-dir-parent\") pod \"multus-w5kx5\" (UID: \"ab41acab-1926-436b-b208-c18b0b375e9f\") " pod="openshift-multus/multus-w5kx5" Sep 09 14:33:07.477054 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.476157 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ab41acab-1926-436b-b208-c18b0b375e9f-host-run-netns\") pod \"multus-w5kx5\" (UID: \"ab41acab-1926-436b-b208-c18b0b375e9f\") " pod="openshift-multus/multus-w5kx5" Sep 09 14:33:07.477054 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.476180 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ab41acab-1926-436b-b208-c18b0b375e9f-hostroot\") pod \"multus-w5kx5\" (UID: \"ab41acab-1926-436b-b208-c18b0b375e9f\") " pod="openshift-multus/multus-w5kx5" Sep 09 14:33:07.477054 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.476137 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/50c734e0-544e-44f4-9c79-a398e559a276-etc-sysctl-d\") pod \"tuned-p5wbc\" (UID: \"50c734e0-544e-44f4-9c79-a398e559a276\") " pod="openshift-cluster-node-tuning-operator/tuned-p5wbc" Sep 09 14:33:07.477054 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.476316 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5a445f65-2ccc-44f4-97ad-1eee2597b711-etc-selinux\") pod \"aws-ebs-csi-driver-node-qlxsx\" (UID: \"5a445f65-2ccc-44f4-97ad-1eee2597b711\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qlxsx" Sep 09 14:33:07.477054 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.476422 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/50c734e0-544e-44f4-9c79-a398e559a276-etc-modprobe-d\") pod \"tuned-p5wbc\" (UID: \"50c734e0-544e-44f4-9c79-a398e559a276\") " pod="openshift-cluster-node-tuning-operator/tuned-p5wbc" Sep 09 14:33:07.477054 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.476464 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ab41acab-1926-436b-b208-c18b0b375e9f-multus-daemon-config\") pod \"multus-w5kx5\" (UID: \"ab41acab-1926-436b-b208-c18b0b375e9f\") " pod="openshift-multus/multus-w5kx5" Sep 09 14:33:07.477054 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.476487 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ab41acab-1926-436b-b208-c18b0b375e9f-etc-kubernetes\") pod \"multus-w5kx5\" (UID: \"ab41acab-1926-436b-b208-c18b0b375e9f\") " pod="openshift-multus/multus-w5kx5" Sep 09 14:33:07.477054 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.476505 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/186aab90-266b-4eae-977a-3f053b423ef3-ovnkube-config\") pod \"ovnkube-node-7xcn5\" (UID: \"186aab90-266b-4eae-977a-3f053b423ef3\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xcn5" Sep 09 14:33:07.477054 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.476521 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/141380ba-1588-47e3-99ea-1de06df135e9-agent-certs\") pod \"konnectivity-agent-ppvhx\" (UID: \"141380ba-1588-47e3-99ea-1de06df135e9\") " pod="kube-system/konnectivity-agent-ppvhx" Sep 09 14:33:07.477054 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.476542 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3c7d6121-a834-42d0-9b78-583260fd4dce-cnibin\") pod \"multus-additional-cni-plugins-p5bsf\" (UID: \"3c7d6121-a834-42d0-9b78-583260fd4dce\") " pod="openshift-multus/multus-additional-cni-plugins-p5bsf" Sep 09 14:33:07.477054 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.476562 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/186aab90-266b-4eae-977a-3f053b423ef3-host-cni-netd\") pod \"ovnkube-node-7xcn5\" (UID: \"186aab90-266b-4eae-977a-3f053b423ef3\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xcn5" Sep 09 14:33:07.477054 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.476617 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/50c734e0-544e-44f4-9c79-a398e559a276-etc-kubernetes\") pod \"tuned-p5wbc\" (UID: \"50c734e0-544e-44f4-9c79-a398e559a276\") " pod="openshift-cluster-node-tuning-operator/tuned-p5wbc" Sep 09 14:33:07.477054 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.476640 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/edb6129a-6cbf-4d83-82b0-018fd0e73f8f-host\") pod \"node-ca-cqf8f\" (UID: \"edb6129a-6cbf-4d83-82b0-018fd0e73f8f\") " pod="openshift-image-registry/node-ca-cqf8f" Sep 09 14:33:07.477054 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.476658 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/186aab90-266b-4eae-977a-3f053b423ef3-log-socket\") pod \"ovnkube-node-7xcn5\" (UID: \"186aab90-266b-4eae-977a-3f053b423ef3\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xcn5" Sep 09 14:33:07.477823 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.476662 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ab41acab-1926-436b-b208-c18b0b375e9f-host-run-multus-certs\") pod \"multus-w5kx5\" (UID: \"ab41acab-1926-436b-b208-c18b0b375e9f\") " pod="openshift-multus/multus-w5kx5" Sep 09 14:33:07.477823 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.476697 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/186aab90-266b-4eae-977a-3f053b423ef3-host-cni-netd\") pod \"ovnkube-node-7xcn5\" (UID: \"186aab90-266b-4eae-977a-3f053b423ef3\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xcn5" Sep 09 14:33:07.477823 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.476699 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f286fc80-41f7-4b6f-8506-b5f0eb2629d3-kubelet-config\") pod \"global-pull-secret-syncer-nqc8q\" (UID: \"f286fc80-41f7-4b6f-8506-b5f0eb2629d3\") " pod="kube-system/global-pull-secret-syncer-nqc8q" Sep 09 14:33:07.477823 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.476726 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/186aab90-266b-4eae-977a-3f053b423ef3-host-slash\") pod \"ovnkube-node-7xcn5\" (UID: \"186aab90-266b-4eae-977a-3f053b423ef3\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xcn5" Sep 09 14:33:07.477823 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.476729 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/186aab90-266b-4eae-977a-3f053b423ef3-node-log\") pod \"ovnkube-node-7xcn5\" (UID: \"186aab90-266b-4eae-977a-3f053b423ef3\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xcn5" Sep 09 14:33:07.477823 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.476754 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tk85r\" (UniqueName: \"kubernetes.io/projected/3c7d6121-a834-42d0-9b78-583260fd4dce-kube-api-access-tk85r\") pod \"multus-additional-cni-plugins-p5bsf\" (UID: \"3c7d6121-a834-42d0-9b78-583260fd4dce\") " pod="openshift-multus/multus-additional-cni-plugins-p5bsf" Sep 09 14:33:07.477823 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.476809 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/edb6129a-6cbf-4d83-82b0-018fd0e73f8f-serviceca\") pod \"node-ca-cqf8f\" (UID: \"edb6129a-6cbf-4d83-82b0-018fd0e73f8f\") " pod="openshift-image-registry/node-ca-cqf8f" Sep 09 14:33:07.477823 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.476821 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/edb6129a-6cbf-4d83-82b0-018fd0e73f8f-host\") pod \"node-ca-cqf8f\" (UID: \"edb6129a-6cbf-4d83-82b0-018fd0e73f8f\") " pod="openshift-image-registry/node-ca-cqf8f" Sep 09 14:33:07.477823 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.476833 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27fsb\" (UniqueName: \"kubernetes.io/projected/b134513f-e0e7-452d-8b61-bae4c9007795-kube-api-access-27fsb\") pod \"iptables-alerter-4kp4s\" (UID: \"b134513f-e0e7-452d-8b61-bae4c9007795\") " pod="openshift-network-operator/iptables-alerter-4kp4s" Sep 09 14:33:07.477823 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.476853 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/50c734e0-544e-44f4-9c79-a398e559a276-etc-kubernetes\") pod \"tuned-p5wbc\" (UID: \"50c734e0-544e-44f4-9c79-a398e559a276\") " pod="openshift-cluster-node-tuning-operator/tuned-p5wbc" Sep 09 14:33:07.477823 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.476857 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57mc4\" (UniqueName: \"kubernetes.io/projected/edb6129a-6cbf-4d83-82b0-018fd0e73f8f-kube-api-access-57mc4\") pod \"node-ca-cqf8f\" (UID: \"edb6129a-6cbf-4d83-82b0-018fd0e73f8f\") " pod="openshift-image-registry/node-ca-cqf8f" Sep 09 14:33:07.477823 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.476881 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/50c734e0-544e-44f4-9c79-a398e559a276-host\") pod \"tuned-p5wbc\" (UID: \"50c734e0-544e-44f4-9c79-a398e559a276\") " pod="openshift-cluster-node-tuning-operator/tuned-p5wbc" Sep 09 14:33:07.477823 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.476880 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5a445f65-2ccc-44f4-97ad-1eee2597b711-device-dir\") pod \"aws-ebs-csi-driver-node-qlxsx\" (UID: \"5a445f65-2ccc-44f4-97ad-1eee2597b711\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qlxsx" Sep 09 14:33:07.477823 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.476904 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/186aab90-266b-4eae-977a-3f053b423ef3-host-run-ovn-kubernetes\") pod \"ovnkube-node-7xcn5\" (UID: \"186aab90-266b-4eae-977a-3f053b423ef3\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xcn5" Sep 09 14:33:07.477823 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.476921 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/186aab90-266b-4eae-977a-3f053b423ef3-ovn-node-metrics-cert\") pod \"ovnkube-node-7xcn5\" (UID: \"186aab90-266b-4eae-977a-3f053b423ef3\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xcn5" Sep 09 14:33:07.477823 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.476925 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5a445f65-2ccc-44f4-97ad-1eee2597b711-device-dir\") pod \"aws-ebs-csi-driver-node-qlxsx\" (UID: \"5a445f65-2ccc-44f4-97ad-1eee2597b711\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qlxsx" Sep 09 14:33:07.477823 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.476935 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/186aab90-266b-4eae-977a-3f053b423ef3-ovnkube-script-lib\") pod \"ovnkube-node-7xcn5\" (UID: \"186aab90-266b-4eae-977a-3f053b423ef3\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xcn5" Sep 09 14:33:07.478385 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.476979 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/186aab90-266b-4eae-977a-3f053b423ef3-host-slash\") pod \"ovnkube-node-7xcn5\" (UID: \"186aab90-266b-4eae-977a-3f053b423ef3\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xcn5" Sep 09 14:33:07.478385 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.477003 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5a445f65-2ccc-44f4-97ad-1eee2597b711-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qlxsx\" (UID: \"5a445f65-2ccc-44f4-97ad-1eee2597b711\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qlxsx" Sep 09 14:33:07.478385 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.477030 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/50c734e0-544e-44f4-9c79-a398e559a276-etc-sysctl-conf\") pod \"tuned-p5wbc\" (UID: \"50c734e0-544e-44f4-9c79-a398e559a276\") " pod="openshift-cluster-node-tuning-operator/tuned-p5wbc" Sep 09 14:33:07.478385 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.477051 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2krn\" (UniqueName: \"kubernetes.io/projected/ce063353-8e00-4701-8fb5-9b32aa0586e1-kube-api-access-l2krn\") pod \"network-check-target-5vdl5\" (UID: \"ce063353-8e00-4701-8fb5-9b32aa0586e1\") " pod="openshift-network-diagnostics/network-check-target-5vdl5" Sep 09 14:33:07.478385 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.477076 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/186aab90-266b-4eae-977a-3f053b423ef3-run-ovn\") pod \"ovnkube-node-7xcn5\" (UID: \"186aab90-266b-4eae-977a-3f053b423ef3\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xcn5" Sep 09 14:33:07.478385 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.477101 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/186aab90-266b-4eae-977a-3f053b423ef3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7xcn5\" (UID: \"186aab90-266b-4eae-977a-3f053b423ef3\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xcn5" Sep 09 14:33:07.478385 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.477128 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5a445f65-2ccc-44f4-97ad-1eee2597b711-sys-fs\") pod \"aws-ebs-csi-driver-node-qlxsx\" (UID: \"5a445f65-2ccc-44f4-97ad-1eee2597b711\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qlxsx" Sep 09 14:33:07.478385 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.477151 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jqcd\" (UniqueName: \"kubernetes.io/projected/50c734e0-544e-44f4-9c79-a398e559a276-kube-api-access-7jqcd\") pod \"tuned-p5wbc\" (UID: \"50c734e0-544e-44f4-9c79-a398e559a276\") " pod="openshift-cluster-node-tuning-operator/tuned-p5wbc" Sep 09 14:33:07.478385 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.477175 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b134513f-e0e7-452d-8b61-bae4c9007795-iptables-alerter-script\") pod \"iptables-alerter-4kp4s\" (UID: \"b134513f-e0e7-452d-8b61-bae4c9007795\") " pod="openshift-network-operator/iptables-alerter-4kp4s" Sep 09 14:33:07.478385 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.477262 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/186aab90-266b-4eae-977a-3f053b423ef3-env-overrides\") pod \"ovnkube-node-7xcn5\" (UID: \"186aab90-266b-4eae-977a-3f053b423ef3\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xcn5" Sep 09 14:33:07.478385 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.477284 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5a445f65-2ccc-44f4-97ad-1eee2597b711-registration-dir\") pod \"aws-ebs-csi-driver-node-qlxsx\" (UID: \"5a445f65-2ccc-44f4-97ad-1eee2597b711\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qlxsx" Sep 09 14:33:07.478385 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.477331 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/90785a55-a3bd-41a7-8d33-85e3b82eb41f-hosts-file\") pod \"node-resolver-5qkp6\" (UID: \"90785a55-a3bd-41a7-8d33-85e3b82eb41f\") " pod="openshift-dns/node-resolver-5qkp6" Sep 09 14:33:07.478385 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.477353 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ab41acab-1926-436b-b208-c18b0b375e9f-host-var-lib-cni-bin\") pod \"multus-w5kx5\" (UID: \"ab41acab-1926-436b-b208-c18b0b375e9f\") " pod="openshift-multus/multus-w5kx5" Sep 09 14:33:07.478385 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.477378 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f286fc80-41f7-4b6f-8506-b5f0eb2629d3-original-pull-secret\") pod \"global-pull-secret-syncer-nqc8q\" (UID: \"f286fc80-41f7-4b6f-8506-b5f0eb2629d3\") " pod="kube-system/global-pull-secret-syncer-nqc8q" Sep 09 14:33:07.478385 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.477402 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3c7d6121-a834-42d0-9b78-583260fd4dce-tuning-conf-dir\") pod \"multus-additional-cni-plugins-p5bsf\" (UID: \"3c7d6121-a834-42d0-9b78-583260fd4dce\") " pod="openshift-multus/multus-additional-cni-plugins-p5bsf" Sep 09 14:33:07.478385 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.477425 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3c7d6121-a834-42d0-9b78-583260fd4dce-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-p5bsf\" (UID: \"3c7d6121-a834-42d0-9b78-583260fd4dce\") " pod="openshift-multus/multus-additional-cni-plugins-p5bsf" Sep 09 14:33:07.479011 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.477449 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/186aab90-266b-4eae-977a-3f053b423ef3-run-openvswitch\") pod \"ovnkube-node-7xcn5\" (UID: \"186aab90-266b-4eae-977a-3f053b423ef3\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xcn5" Sep 09 14:33:07.479011 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.477473 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pn6xh\" (UniqueName: \"kubernetes.io/projected/90785a55-a3bd-41a7-8d33-85e3b82eb41f-kube-api-access-pn6xh\") pod \"node-resolver-5qkp6\" (UID: \"90785a55-a3bd-41a7-8d33-85e3b82eb41f\") " pod="openshift-dns/node-resolver-5qkp6" Sep 09 14:33:07.479011 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.477498 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/141380ba-1588-47e3-99ea-1de06df135e9-konnectivity-ca\") pod \"konnectivity-agent-ppvhx\" (UID: \"141380ba-1588-47e3-99ea-1de06df135e9\") " pod="kube-system/konnectivity-agent-ppvhx" Sep 09 14:33:07.479011 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.477520 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3c7d6121-a834-42d0-9b78-583260fd4dce-cni-binary-copy\") pod \"multus-additional-cni-plugins-p5bsf\" (UID: \"3c7d6121-a834-42d0-9b78-583260fd4dce\") " pod="openshift-multus/multus-additional-cni-plugins-p5bsf" Sep 09 14:33:07.479011 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.477565 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/186aab90-266b-4eae-977a-3f053b423ef3-host-cni-bin\") pod \"ovnkube-node-7xcn5\" (UID: \"186aab90-266b-4eae-977a-3f053b423ef3\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xcn5" Sep 09 14:33:07.479011 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.477592 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlvzg\" (UniqueName: \"kubernetes.io/projected/186aab90-266b-4eae-977a-3f053b423ef3-kube-api-access-dlvzg\") pod \"ovnkube-node-7xcn5\" (UID: \"186aab90-266b-4eae-977a-3f053b423ef3\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xcn5" Sep 09 14:33:07.479011 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.477615 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/50c734e0-544e-44f4-9c79-a398e559a276-run\") pod \"tuned-p5wbc\" (UID: \"50c734e0-544e-44f4-9c79-a398e559a276\") " pod="openshift-cluster-node-tuning-operator/tuned-p5wbc" Sep 09 14:33:07.479011 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.477739 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/50c734e0-544e-44f4-9c79-a398e559a276-var-lib-kubelet\") pod \"tuned-p5wbc\" (UID: \"50c734e0-544e-44f4-9c79-a398e559a276\") " pod="openshift-cluster-node-tuning-operator/tuned-p5wbc" Sep 09 14:33:07.479011 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.477754 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/edb6129a-6cbf-4d83-82b0-018fd0e73f8f-serviceca\") pod \"node-ca-cqf8f\" (UID: \"edb6129a-6cbf-4d83-82b0-018fd0e73f8f\") " pod="openshift-image-registry/node-ca-cqf8f" Sep 09 14:33:07.479011 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.477764 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ab41acab-1926-436b-b208-c18b0b375e9f-cni-binary-copy\") pod \"multus-w5kx5\" (UID: \"ab41acab-1926-436b-b208-c18b0b375e9f\") " pod="openshift-multus/multus-w5kx5" Sep 09 14:33:07.479011 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.477787 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ab41acab-1926-436b-b208-c18b0b375e9f-multus-conf-dir\") pod \"multus-w5kx5\" (UID: \"ab41acab-1926-436b-b208-c18b0b375e9f\") " pod="openshift-multus/multus-w5kx5" Sep 09 14:33:07.479011 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.477853 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/50c734e0-544e-44f4-9c79-a398e559a276-sys\") pod \"tuned-p5wbc\" (UID: \"50c734e0-544e-44f4-9c79-a398e559a276\") " pod="openshift-cluster-node-tuning-operator/tuned-p5wbc" Sep 09 14:33:07.479011 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.477885 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/50c734e0-544e-44f4-9c79-a398e559a276-tmp\") pod \"tuned-p5wbc\" (UID: \"50c734e0-544e-44f4-9c79-a398e559a276\") " pod="openshift-cluster-node-tuning-operator/tuned-p5wbc" Sep 09 14:33:07.479011 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.477907 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90785a55-a3bd-41a7-8d33-85e3b82eb41f-tmp-dir\") pod \"node-resolver-5qkp6\" (UID: \"90785a55-a3bd-41a7-8d33-85e3b82eb41f\") " pod="openshift-dns/node-resolver-5qkp6" Sep 09 14:33:07.479011 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.477940 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/186aab90-266b-4eae-977a-3f053b423ef3-host-run-netns\") pod \"ovnkube-node-7xcn5\" (UID: \"186aab90-266b-4eae-977a-3f053b423ef3\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xcn5" Sep 09 14:33:07.479011 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.477963 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/186aab90-266b-4eae-977a-3f053b423ef3-run-systemd\") pod \"ovnkube-node-7xcn5\" (UID: \"186aab90-266b-4eae-977a-3f053b423ef3\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xcn5" Sep 09 14:33:07.479011 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.477987 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/186aab90-266b-4eae-977a-3f053b423ef3-var-lib-openvswitch\") pod \"ovnkube-node-7xcn5\" (UID: \"186aab90-266b-4eae-977a-3f053b423ef3\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xcn5" Sep 09 14:33:07.479676 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.478011 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b134513f-e0e7-452d-8b61-bae4c9007795-host-slash\") pod \"iptables-alerter-4kp4s\" (UID: \"b134513f-e0e7-452d-8b61-bae4c9007795\") " pod="openshift-network-operator/iptables-alerter-4kp4s" Sep 09 14:33:07.479676 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.478039 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ab41acab-1926-436b-b208-c18b0b375e9f-multus-cni-dir\") pod \"multus-w5kx5\" (UID: \"ab41acab-1926-436b-b208-c18b0b375e9f\") " pod="openshift-multus/multus-w5kx5" Sep 09 14:33:07.479676 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.478063 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qm86k\" (UniqueName: \"kubernetes.io/projected/ab41acab-1926-436b-b208-c18b0b375e9f-kube-api-access-qm86k\") pod \"multus-w5kx5\" (UID: \"ab41acab-1926-436b-b208-c18b0b375e9f\") " pod="openshift-multus/multus-w5kx5" Sep 09 14:33:07.479676 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.478085 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ab41acab-1926-436b-b208-c18b0b375e9f-system-cni-dir\") pod \"multus-w5kx5\" (UID: \"ab41acab-1926-436b-b208-c18b0b375e9f\") " pod="openshift-multus/multus-w5kx5" Sep 09 14:33:07.479676 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.478109 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/186aab90-266b-4eae-977a-3f053b423ef3-host-kubelet\") pod \"ovnkube-node-7xcn5\" (UID: \"186aab90-266b-4eae-977a-3f053b423ef3\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xcn5" Sep 09 14:33:07.479676 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.478174 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/186aab90-266b-4eae-977a-3f053b423ef3-host-kubelet\") pod \"ovnkube-node-7xcn5\" (UID: \"186aab90-266b-4eae-977a-3f053b423ef3\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xcn5" Sep 09 14:33:07.479676 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.478259 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/50c734e0-544e-44f4-9c79-a398e559a276-run\") pod \"tuned-p5wbc\" (UID: \"50c734e0-544e-44f4-9c79-a398e559a276\") " pod="openshift-cluster-node-tuning-operator/tuned-p5wbc" Sep 09 14:33:07.479676 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.478258 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/186aab90-266b-4eae-977a-3f053b423ef3-ovnkube-script-lib\") pod \"ovnkube-node-7xcn5\" (UID: \"186aab90-266b-4eae-977a-3f053b423ef3\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xcn5" Sep 09 14:33:07.479676 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.478382 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/50c734e0-544e-44f4-9c79-a398e559a276-lib-modules\") pod \"tuned-p5wbc\" (UID: \"50c734e0-544e-44f4-9c79-a398e559a276\") " pod="openshift-cluster-node-tuning-operator/tuned-p5wbc" Sep 09 14:33:07.479676 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.478770 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/186aab90-266b-4eae-977a-3f053b423ef3-ovnkube-config\") pod \"ovnkube-node-7xcn5\" (UID: \"186aab90-266b-4eae-977a-3f053b423ef3\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xcn5" Sep 09 14:33:07.479676 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.478782 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b134513f-e0e7-452d-8b61-bae4c9007795-iptables-alerter-script\") pod \"iptables-alerter-4kp4s\" (UID: \"b134513f-e0e7-452d-8b61-bae4c9007795\") " pod="openshift-network-operator/iptables-alerter-4kp4s" Sep 09 14:33:07.479676 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.478763 2576 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 09 14:33:07.479676 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.478823 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5a445f65-2ccc-44f4-97ad-1eee2597b711-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qlxsx\" (UID: \"5a445f65-2ccc-44f4-97ad-1eee2597b711\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qlxsx" Sep 09 14:33:07.479676 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.478822 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/186aab90-266b-4eae-977a-3f053b423ef3-host-run-ovn-kubernetes\") pod \"ovnkube-node-7xcn5\" (UID: \"186aab90-266b-4eae-977a-3f053b423ef3\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xcn5" Sep 09 14:33:07.479676 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.478864 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5a445f65-2ccc-44f4-97ad-1eee2597b711-registration-dir\") pod \"aws-ebs-csi-driver-node-qlxsx\" (UID: \"5a445f65-2ccc-44f4-97ad-1eee2597b711\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qlxsx" Sep 09 14:33:07.479676 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.478900 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/90785a55-a3bd-41a7-8d33-85e3b82eb41f-hosts-file\") pod \"node-resolver-5qkp6\" (UID: \"90785a55-a3bd-41a7-8d33-85e3b82eb41f\") " pod="openshift-dns/node-resolver-5qkp6" Sep 09 14:33:07.479676 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.478936 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/186aab90-266b-4eae-977a-3f053b423ef3-run-openvswitch\") pod \"ovnkube-node-7xcn5\" (UID: \"186aab90-266b-4eae-977a-3f053b423ef3\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xcn5" Sep 09 14:33:07.479676 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.478970 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/186aab90-266b-4eae-977a-3f053b423ef3-run-ovn\") pod \"ovnkube-node-7xcn5\" (UID: \"186aab90-266b-4eae-977a-3f053b423ef3\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xcn5" Sep 09 14:33:07.480579 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.479075 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/50c734e0-544e-44f4-9c79-a398e559a276-etc-sysctl-conf\") pod \"tuned-p5wbc\" (UID: \"50c734e0-544e-44f4-9c79-a398e559a276\") " pod="openshift-cluster-node-tuning-operator/tuned-p5wbc" Sep 09 14:33:07.480579 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.479106 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/186aab90-266b-4eae-977a-3f053b423ef3-env-overrides\") pod \"ovnkube-node-7xcn5\" (UID: \"186aab90-266b-4eae-977a-3f053b423ef3\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xcn5" Sep 09 14:33:07.480579 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.479121 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/186aab90-266b-4eae-977a-3f053b423ef3-host-cni-bin\") pod \"ovnkube-node-7xcn5\" (UID: \"186aab90-266b-4eae-977a-3f053b423ef3\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xcn5" Sep 09 14:33:07.480579 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.479129 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5a445f65-2ccc-44f4-97ad-1eee2597b711-sys-fs\") pod \"aws-ebs-csi-driver-node-qlxsx\" (UID: \"5a445f65-2ccc-44f4-97ad-1eee2597b711\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qlxsx" Sep 09 14:33:07.480579 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.479167 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/186aab90-266b-4eae-977a-3f053b423ef3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7xcn5\" (UID: \"186aab90-266b-4eae-977a-3f053b423ef3\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xcn5" Sep 09 14:33:07.480579 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.479676 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/186aab90-266b-4eae-977a-3f053b423ef3-host-run-netns\") pod \"ovnkube-node-7xcn5\" (UID: \"186aab90-266b-4eae-977a-3f053b423ef3\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xcn5" Sep 09 14:33:07.480579 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.479711 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b134513f-e0e7-452d-8b61-bae4c9007795-host-slash\") pod \"iptables-alerter-4kp4s\" (UID: \"b134513f-e0e7-452d-8b61-bae4c9007795\") " pod="openshift-network-operator/iptables-alerter-4kp4s" Sep 09 14:33:07.480579 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.479743 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/186aab90-266b-4eae-977a-3f053b423ef3-run-systemd\") pod \"ovnkube-node-7xcn5\" (UID: \"186aab90-266b-4eae-977a-3f053b423ef3\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xcn5" Sep 09 14:33:07.480579 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.479793 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/50c734e0-544e-44f4-9c79-a398e559a276-sys\") pod \"tuned-p5wbc\" (UID: \"50c734e0-544e-44f4-9c79-a398e559a276\") " pod="openshift-cluster-node-tuning-operator/tuned-p5wbc" Sep 09 14:33:07.480579 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.479837 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/50c734e0-544e-44f4-9c79-a398e559a276-var-lib-kubelet\") pod \"tuned-p5wbc\" (UID: \"50c734e0-544e-44f4-9c79-a398e559a276\") " pod="openshift-cluster-node-tuning-operator/tuned-p5wbc" Sep 09 14:33:07.480579 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.479867 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90785a55-a3bd-41a7-8d33-85e3b82eb41f-tmp-dir\") pod \"node-resolver-5qkp6\" (UID: \"90785a55-a3bd-41a7-8d33-85e3b82eb41f\") " pod="openshift-dns/node-resolver-5qkp6" Sep 09 14:33:07.480579 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.479875 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/186aab90-266b-4eae-977a-3f053b423ef3-var-lib-openvswitch\") pod \"ovnkube-node-7xcn5\" (UID: \"186aab90-266b-4eae-977a-3f053b423ef3\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xcn5" Sep 09 14:33:07.482680 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.482653 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/50c734e0-544e-44f4-9c79-a398e559a276-etc-tuned\") pod \"tuned-p5wbc\" (UID: \"50c734e0-544e-44f4-9c79-a398e559a276\") " pod="openshift-cluster-node-tuning-operator/tuned-p5wbc" Sep 09 14:33:07.485070 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.485043 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/50c734e0-544e-44f4-9c79-a398e559a276-tmp\") pod \"tuned-p5wbc\" (UID: \"50c734e0-544e-44f4-9c79-a398e559a276\") " pod="openshift-cluster-node-tuning-operator/tuned-p5wbc" Sep 09 14:33:07.485660 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.485602 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/186aab90-266b-4eae-977a-3f053b423ef3-ovn-node-metrics-cert\") pod \"ovnkube-node-7xcn5\" (UID: \"186aab90-266b-4eae-977a-3f053b423ef3\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xcn5" Sep 09 14:33:07.487973 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.487948 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pn6xh\" (UniqueName: \"kubernetes.io/projected/90785a55-a3bd-41a7-8d33-85e3b82eb41f-kube-api-access-pn6xh\") pod \"node-resolver-5qkp6\" (UID: \"90785a55-a3bd-41a7-8d33-85e3b82eb41f\") " pod="openshift-dns/node-resolver-5qkp6" Sep 09 14:33:07.488096 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.488073 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jqcd\" (UniqueName: \"kubernetes.io/projected/50c734e0-544e-44f4-9c79-a398e559a276-kube-api-access-7jqcd\") pod \"tuned-p5wbc\" (UID: \"50c734e0-544e-44f4-9c79-a398e559a276\") " pod="openshift-cluster-node-tuning-operator/tuned-p5wbc" Sep 09 14:33:07.488544 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.488523 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-57mc4\" (UniqueName: \"kubernetes.io/projected/edb6129a-6cbf-4d83-82b0-018fd0e73f8f-kube-api-access-57mc4\") pod \"node-ca-cqf8f\" (UID: \"edb6129a-6cbf-4d83-82b0-018fd0e73f8f\") " pod="openshift-image-registry/node-ca-cqf8f" Sep 09 14:33:07.488836 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.488813 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-27fsb\" (UniqueName: \"kubernetes.io/projected/b134513f-e0e7-452d-8b61-bae4c9007795-kube-api-access-27fsb\") pod \"iptables-alerter-4kp4s\" (UID: \"b134513f-e0e7-452d-8b61-bae4c9007795\") " pod="openshift-network-operator/iptables-alerter-4kp4s" Sep 09 14:33:07.489633 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.489604 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlvzg\" (UniqueName: \"kubernetes.io/projected/186aab90-266b-4eae-977a-3f053b423ef3-kube-api-access-dlvzg\") pod \"ovnkube-node-7xcn5\" (UID: \"186aab90-266b-4eae-977a-3f053b423ef3\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xcn5" Sep 09 14:33:07.489801 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.489782 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxqjx\" (UniqueName: \"kubernetes.io/projected/5a445f65-2ccc-44f4-97ad-1eee2597b711-kube-api-access-sxqjx\") pod \"aws-ebs-csi-driver-node-qlxsx\" (UID: \"5a445f65-2ccc-44f4-97ad-1eee2597b711\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qlxsx" Sep 09 14:33:07.514930 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.514874 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-66.ec2.internal" event={"ID":"9af3564e64deec6f121c01d714dc0e5a","Type":"ContainerStarted","Data":"82cb097fd8ea93dcee7d3f3fac587e222e771372bdb95ec3b18570c47942f27b"} Sep 09 14:33:07.516386 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.516358 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-66.ec2.internal" event={"ID":"3aba2b4456e66055b3846a53f8393ab4","Type":"ContainerStarted","Data":"4a2e76da54ef5a9313e49f92b3913a64cf0bac9fbb4500f9faf6ac21aa58fa88"} Sep 09 14:33:07.579272 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.579235 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcsz5\" (UniqueName: \"kubernetes.io/projected/ab36c1e4-a65c-4cf9-ada4-f8a87d4b463b-kube-api-access-tcsz5\") pod \"network-metrics-daemon-wcj2p\" (UID: \"ab36c1e4-a65c-4cf9-ada4-f8a87d4b463b\") " pod="openshift-multus/network-metrics-daemon-wcj2p" Sep 09 14:33:07.579467 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.579292 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f286fc80-41f7-4b6f-8506-b5f0eb2629d3-dbus\") pod \"global-pull-secret-syncer-nqc8q\" (UID: \"f286fc80-41f7-4b6f-8506-b5f0eb2629d3\") " pod="kube-system/global-pull-secret-syncer-nqc8q" Sep 09 14:33:07.579467 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.579337 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ab41acab-1926-436b-b208-c18b0b375e9f-host-run-k8s-cni-cncf-io\") pod \"multus-w5kx5\" (UID: \"ab41acab-1926-436b-b208-c18b0b375e9f\") " pod="openshift-multus/multus-w5kx5" Sep 09 14:33:07.579467 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.579361 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab36c1e4-a65c-4cf9-ada4-f8a87d4b463b-metrics-certs\") pod \"network-metrics-daemon-wcj2p\" (UID: \"ab36c1e4-a65c-4cf9-ada4-f8a87d4b463b\") " pod="openshift-multus/network-metrics-daemon-wcj2p" Sep 09 14:33:07.579467 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.579384 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3c7d6121-a834-42d0-9b78-583260fd4dce-system-cni-dir\") pod \"multus-additional-cni-plugins-p5bsf\" (UID: \"3c7d6121-a834-42d0-9b78-583260fd4dce\") " pod="openshift-multus/multus-additional-cni-plugins-p5bsf" Sep 09 14:33:07.579467 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.579407 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3c7d6121-a834-42d0-9b78-583260fd4dce-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-p5bsf\" (UID: \"3c7d6121-a834-42d0-9b78-583260fd4dce\") " pod="openshift-multus/multus-additional-cni-plugins-p5bsf" Sep 09 14:33:07.579467 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.579436 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ab41acab-1926-436b-b208-c18b0b375e9f-multus-socket-dir-parent\") pod \"multus-w5kx5\" (UID: \"ab41acab-1926-436b-b208-c18b0b375e9f\") " pod="openshift-multus/multus-w5kx5" Sep 09 14:33:07.579467 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.579457 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ab41acab-1926-436b-b208-c18b0b375e9f-host-run-netns\") pod \"multus-w5kx5\" (UID: \"ab41acab-1926-436b-b208-c18b0b375e9f\") " pod="openshift-multus/multus-w5kx5" Sep 09 14:33:07.579790 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.579478 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ab41acab-1926-436b-b208-c18b0b375e9f-hostroot\") pod \"multus-w5kx5\" (UID: \"ab41acab-1926-436b-b208-c18b0b375e9f\") " pod="openshift-multus/multus-w5kx5" Sep 09 14:33:07.579790 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.579499 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ab41acab-1926-436b-b208-c18b0b375e9f-multus-daemon-config\") pod \"multus-w5kx5\" (UID: \"ab41acab-1926-436b-b208-c18b0b375e9f\") " pod="openshift-multus/multus-w5kx5" Sep 09 14:33:07.579790 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.579520 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ab41acab-1926-436b-b208-c18b0b375e9f-etc-kubernetes\") pod \"multus-w5kx5\" (UID: \"ab41acab-1926-436b-b208-c18b0b375e9f\") " pod="openshift-multus/multus-w5kx5" Sep 09 14:33:07.579790 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.579524 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f286fc80-41f7-4b6f-8506-b5f0eb2629d3-dbus\") pod \"global-pull-secret-syncer-nqc8q\" (UID: \"f286fc80-41f7-4b6f-8506-b5f0eb2629d3\") " pod="kube-system/global-pull-secret-syncer-nqc8q" Sep 09 14:33:07.579790 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.579542 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/141380ba-1588-47e3-99ea-1de06df135e9-agent-certs\") pod \"konnectivity-agent-ppvhx\" (UID: \"141380ba-1588-47e3-99ea-1de06df135e9\") " pod="kube-system/konnectivity-agent-ppvhx" Sep 09 14:33:07.579790 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.579578 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3c7d6121-a834-42d0-9b78-583260fd4dce-cnibin\") pod \"multus-additional-cni-plugins-p5bsf\" (UID: \"3c7d6121-a834-42d0-9b78-583260fd4dce\") " pod="openshift-multus/multus-additional-cni-plugins-p5bsf" Sep 09 14:33:07.579790 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.579608 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ab41acab-1926-436b-b208-c18b0b375e9f-host-run-multus-certs\") pod \"multus-w5kx5\" (UID: \"ab41acab-1926-436b-b208-c18b0b375e9f\") " pod="openshift-multus/multus-w5kx5" Sep 09 14:33:07.579790 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.579631 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f286fc80-41f7-4b6f-8506-b5f0eb2629d3-kubelet-config\") pod \"global-pull-secret-syncer-nqc8q\" (UID: \"f286fc80-41f7-4b6f-8506-b5f0eb2629d3\") " pod="kube-system/global-pull-secret-syncer-nqc8q" Sep 09 14:33:07.579790 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.579656 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tk85r\" (UniqueName: \"kubernetes.io/projected/3c7d6121-a834-42d0-9b78-583260fd4dce-kube-api-access-tk85r\") pod \"multus-additional-cni-plugins-p5bsf\" (UID: \"3c7d6121-a834-42d0-9b78-583260fd4dce\") " pod="openshift-multus/multus-additional-cni-plugins-p5bsf" Sep 09 14:33:07.579790 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.579691 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2krn\" (UniqueName: \"kubernetes.io/projected/ce063353-8e00-4701-8fb5-9b32aa0586e1-kube-api-access-l2krn\") pod \"network-check-target-5vdl5\" (UID: \"ce063353-8e00-4701-8fb5-9b32aa0586e1\") " pod="openshift-network-diagnostics/network-check-target-5vdl5" Sep 09 14:33:07.579790 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.579724 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ab41acab-1926-436b-b208-c18b0b375e9f-host-var-lib-cni-bin\") pod \"multus-w5kx5\" (UID: \"ab41acab-1926-436b-b208-c18b0b375e9f\") " pod="openshift-multus/multus-w5kx5" Sep 09 14:33:07.579790 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.579750 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f286fc80-41f7-4b6f-8506-b5f0eb2629d3-original-pull-secret\") pod \"global-pull-secret-syncer-nqc8q\" (UID: \"f286fc80-41f7-4b6f-8506-b5f0eb2629d3\") " pod="kube-system/global-pull-secret-syncer-nqc8q" Sep 09 14:33:07.579790 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.579772 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3c7d6121-a834-42d0-9b78-583260fd4dce-tuning-conf-dir\") pod \"multus-additional-cni-plugins-p5bsf\" (UID: \"3c7d6121-a834-42d0-9b78-583260fd4dce\") " pod="openshift-multus/multus-additional-cni-plugins-p5bsf" Sep 09 14:33:07.580379 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.579800 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3c7d6121-a834-42d0-9b78-583260fd4dce-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-p5bsf\" (UID: \"3c7d6121-a834-42d0-9b78-583260fd4dce\") " pod="openshift-multus/multus-additional-cni-plugins-p5bsf" Sep 09 14:33:07.580379 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.579835 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/141380ba-1588-47e3-99ea-1de06df135e9-konnectivity-ca\") pod \"konnectivity-agent-ppvhx\" (UID: \"141380ba-1588-47e3-99ea-1de06df135e9\") " pod="kube-system/konnectivity-agent-ppvhx" Sep 09 14:33:07.580379 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.579857 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3c7d6121-a834-42d0-9b78-583260fd4dce-cni-binary-copy\") pod \"multus-additional-cni-plugins-p5bsf\" (UID: \"3c7d6121-a834-42d0-9b78-583260fd4dce\") " pod="openshift-multus/multus-additional-cni-plugins-p5bsf" Sep 09 14:33:07.580379 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.580346 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3c7d6121-a834-42d0-9b78-583260fd4dce-cni-binary-copy\") pod \"multus-additional-cni-plugins-p5bsf\" (UID: \"3c7d6121-a834-42d0-9b78-583260fd4dce\") " pod="openshift-multus/multus-additional-cni-plugins-p5bsf" Sep 09 14:33:07.580582 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.580383 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3c7d6121-a834-42d0-9b78-583260fd4dce-cnibin\") pod \"multus-additional-cni-plugins-p5bsf\" (UID: \"3c7d6121-a834-42d0-9b78-583260fd4dce\") " pod="openshift-multus/multus-additional-cni-plugins-p5bsf" Sep 09 14:33:07.580582 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.580401 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ab41acab-1926-436b-b208-c18b0b375e9f-host-var-lib-cni-bin\") pod \"multus-w5kx5\" (UID: \"ab41acab-1926-436b-b208-c18b0b375e9f\") " pod="openshift-multus/multus-w5kx5" Sep 09 14:33:07.580582 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.580436 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ab41acab-1926-436b-b208-c18b0b375e9f-host-run-multus-certs\") pod \"multus-w5kx5\" (UID: \"ab41acab-1926-436b-b208-c18b0b375e9f\") " pod="openshift-multus/multus-w5kx5" Sep 09 14:33:07.580582 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.580452 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f286fc80-41f7-4b6f-8506-b5f0eb2629d3-kubelet-config\") pod \"global-pull-secret-syncer-nqc8q\" (UID: \"f286fc80-41f7-4b6f-8506-b5f0eb2629d3\") " pod="kube-system/global-pull-secret-syncer-nqc8q" Sep 09 14:33:07.580582 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:07.580509 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 09 14:33:07.580582 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:07.580556 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f286fc80-41f7-4b6f-8506-b5f0eb2629d3-original-pull-secret podName:f286fc80-41f7-4b6f-8506-b5f0eb2629d3 nodeName:}" failed. No retries permitted until 2026-09-09 14:33:08.080542524 +0000 UTC m=+3.091458751 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f286fc80-41f7-4b6f-8506-b5f0eb2629d3-original-pull-secret") pod "global-pull-secret-syncer-nqc8q" (UID: "f286fc80-41f7-4b6f-8506-b5f0eb2629d3") : object "kube-system"/"original-pull-secret" not registered Sep 09 14:33:07.580867 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.580638 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ab41acab-1926-436b-b208-c18b0b375e9f-cni-binary-copy\") pod \"multus-w5kx5\" (UID: \"ab41acab-1926-436b-b208-c18b0b375e9f\") " pod="openshift-multus/multus-w5kx5" Sep 09 14:33:07.580867 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.580666 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ab41acab-1926-436b-b208-c18b0b375e9f-multus-conf-dir\") pod \"multus-w5kx5\" (UID: \"ab41acab-1926-436b-b208-c18b0b375e9f\") " pod="openshift-multus/multus-w5kx5" Sep 09 14:33:07.580867 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.580724 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ab41acab-1926-436b-b208-c18b0b375e9f-multus-conf-dir\") pod \"multus-w5kx5\" (UID: \"ab41acab-1926-436b-b208-c18b0b375e9f\") " pod="openshift-multus/multus-w5kx5" Sep 09 14:33:07.580867 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.580749 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ab41acab-1926-436b-b208-c18b0b375e9f-multus-cni-dir\") pod \"multus-w5kx5\" (UID: \"ab41acab-1926-436b-b208-c18b0b375e9f\") " pod="openshift-multus/multus-w5kx5" Sep 09 14:33:07.580867 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.580773 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qm86k\" (UniqueName: \"kubernetes.io/projected/ab41acab-1926-436b-b208-c18b0b375e9f-kube-api-access-qm86k\") pod \"multus-w5kx5\" (UID: \"ab41acab-1926-436b-b208-c18b0b375e9f\") " pod="openshift-multus/multus-w5kx5" Sep 09 14:33:07.580867 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.580775 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3c7d6121-a834-42d0-9b78-583260fd4dce-tuning-conf-dir\") pod \"multus-additional-cni-plugins-p5bsf\" (UID: \"3c7d6121-a834-42d0-9b78-583260fd4dce\") " pod="openshift-multus/multus-additional-cni-plugins-p5bsf" Sep 09 14:33:07.581123 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.581086 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3c7d6121-a834-42d0-9b78-583260fd4dce-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-p5bsf\" (UID: \"3c7d6121-a834-42d0-9b78-583260fd4dce\") " pod="openshift-multus/multus-additional-cni-plugins-p5bsf" Sep 09 14:33:07.581172 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.581155 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ab41acab-1926-436b-b208-c18b0b375e9f-system-cni-dir\") pod \"multus-w5kx5\" (UID: \"ab41acab-1926-436b-b208-c18b0b375e9f\") " pod="openshift-multus/multus-w5kx5" Sep 09 14:33:07.581227 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.581176 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ab41acab-1926-436b-b208-c18b0b375e9f-cnibin\") pod \"multus-w5kx5\" (UID: \"ab41acab-1926-436b-b208-c18b0b375e9f\") " pod="openshift-multus/multus-w5kx5" Sep 09 14:33:07.581227 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.581194 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ab41acab-1926-436b-b208-c18b0b375e9f-host-var-lib-cni-multus\") pod \"multus-w5kx5\" (UID: \"ab41acab-1926-436b-b208-c18b0b375e9f\") " pod="openshift-multus/multus-w5kx5" Sep 09 14:33:07.581227 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.581208 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ab41acab-1926-436b-b208-c18b0b375e9f-host-var-lib-kubelet\") pod \"multus-w5kx5\" (UID: \"ab41acab-1926-436b-b208-c18b0b375e9f\") " pod="openshift-multus/multus-w5kx5" Sep 09 14:33:07.581405 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.581253 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ab41acab-1926-436b-b208-c18b0b375e9f-cni-binary-copy\") pod \"multus-w5kx5\" (UID: \"ab41acab-1926-436b-b208-c18b0b375e9f\") " pod="openshift-multus/multus-w5kx5" Sep 09 14:33:07.581405 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.581330 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ab41acab-1926-436b-b208-c18b0b375e9f-host-var-lib-kubelet\") pod \"multus-w5kx5\" (UID: \"ab41acab-1926-436b-b208-c18b0b375e9f\") " pod="openshift-multus/multus-w5kx5" Sep 09 14:33:07.581405 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.581345 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ab41acab-1926-436b-b208-c18b0b375e9f-multus-cni-dir\") pod \"multus-w5kx5\" (UID: \"ab41acab-1926-436b-b208-c18b0b375e9f\") " pod="openshift-multus/multus-w5kx5" Sep 09 14:33:07.581405 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.581398 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ab41acab-1926-436b-b208-c18b0b375e9f-cnibin\") pod \"multus-w5kx5\" (UID: \"ab41acab-1926-436b-b208-c18b0b375e9f\") " pod="openshift-multus/multus-w5kx5" Sep 09 14:33:07.581575 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.581432 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ab41acab-1926-436b-b208-c18b0b375e9f-host-var-lib-cni-multus\") pod \"multus-w5kx5\" (UID: \"ab41acab-1926-436b-b208-c18b0b375e9f\") " pod="openshift-multus/multus-w5kx5" Sep 09 14:33:07.581575 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.581479 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ab41acab-1926-436b-b208-c18b0b375e9f-multus-socket-dir-parent\") pod \"multus-w5kx5\" (UID: \"ab41acab-1926-436b-b208-c18b0b375e9f\") " pod="openshift-multus/multus-w5kx5" Sep 09 14:33:07.581575 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.581512 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ab41acab-1926-436b-b208-c18b0b375e9f-host-run-k8s-cni-cncf-io\") pod \"multus-w5kx5\" (UID: \"ab41acab-1926-436b-b208-c18b0b375e9f\") " pod="openshift-multus/multus-w5kx5" Sep 09 14:33:07.581706 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:07.581602 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 14:33:07.581706 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:07.581645 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab36c1e4-a65c-4cf9-ada4-f8a87d4b463b-metrics-certs podName:ab36c1e4-a65c-4cf9-ada4-f8a87d4b463b nodeName:}" failed. No retries permitted until 2026-09-09 14:33:08.081630284 +0000 UTC m=+3.092546516 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ab36c1e4-a65c-4cf9-ada4-f8a87d4b463b-metrics-certs") pod "network-metrics-daemon-wcj2p" (UID: "ab36c1e4-a65c-4cf9-ada4-f8a87d4b463b") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 14:33:07.581706 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.581657 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ab41acab-1926-436b-b208-c18b0b375e9f-system-cni-dir\") pod \"multus-w5kx5\" (UID: \"ab41acab-1926-436b-b208-c18b0b375e9f\") " pod="openshift-multus/multus-w5kx5" Sep 09 14:33:07.581706 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.581679 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3c7d6121-a834-42d0-9b78-583260fd4dce-system-cni-dir\") pod \"multus-additional-cni-plugins-p5bsf\" (UID: \"3c7d6121-a834-42d0-9b78-583260fd4dce\") " pod="openshift-multus/multus-additional-cni-plugins-p5bsf" Sep 09 14:33:07.581909 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.581897 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ab41acab-1926-436b-b208-c18b0b375e9f-hostroot\") pod \"multus-w5kx5\" (UID: \"ab41acab-1926-436b-b208-c18b0b375e9f\") " pod="openshift-multus/multus-w5kx5" Sep 09 14:33:07.581949 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.581939 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ab41acab-1926-436b-b208-c18b0b375e9f-host-run-netns\") pod \"multus-w5kx5\" (UID: \"ab41acab-1926-436b-b208-c18b0b375e9f\") " pod="openshift-multus/multus-w5kx5" Sep 09 14:33:07.582091 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.582069 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3c7d6121-a834-42d0-9b78-583260fd4dce-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-p5bsf\" (UID: \"3c7d6121-a834-42d0-9b78-583260fd4dce\") " pod="openshift-multus/multus-additional-cni-plugins-p5bsf" Sep 09 14:33:07.582159 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.582129 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ab41acab-1926-436b-b208-c18b0b375e9f-etc-kubernetes\") pod \"multus-w5kx5\" (UID: \"ab41acab-1926-436b-b208-c18b0b375e9f\") " pod="openshift-multus/multus-w5kx5" Sep 09 14:33:07.582402 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.582384 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ab41acab-1926-436b-b208-c18b0b375e9f-multus-daemon-config\") pod \"multus-w5kx5\" (UID: \"ab41acab-1926-436b-b208-c18b0b375e9f\") " pod="openshift-multus/multus-w5kx5" Sep 09 14:33:07.582705 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.582688 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/141380ba-1588-47e3-99ea-1de06df135e9-konnectivity-ca\") pod \"konnectivity-agent-ppvhx\" (UID: \"141380ba-1588-47e3-99ea-1de06df135e9\") " pod="kube-system/konnectivity-agent-ppvhx" Sep 09 14:33:07.582770 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.582718 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/141380ba-1588-47e3-99ea-1de06df135e9-agent-certs\") pod \"konnectivity-agent-ppvhx\" (UID: \"141380ba-1588-47e3-99ea-1de06df135e9\") " pod="kube-system/konnectivity-agent-ppvhx" Sep 09 14:33:07.586284 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:07.586262 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 09 14:33:07.586419 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:07.586323 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 09 14:33:07.586419 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:07.586336 2576 projected.go:196] Error preparing data for projected volume kube-api-access-l2krn for pod openshift-network-diagnostics/network-check-target-5vdl5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 14:33:07.586419 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:07.586398 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ce063353-8e00-4701-8fb5-9b32aa0586e1-kube-api-access-l2krn podName:ce063353-8e00-4701-8fb5-9b32aa0586e1 nodeName:}" failed. No retries permitted until 2026-09-09 14:33:08.086383289 +0000 UTC m=+3.097299522 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-l2krn" (UniqueName: "kubernetes.io/projected/ce063353-8e00-4701-8fb5-9b32aa0586e1-kube-api-access-l2krn") pod "network-check-target-5vdl5" (UID: "ce063353-8e00-4701-8fb5-9b32aa0586e1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 14:33:07.589286 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.589234 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tk85r\" (UniqueName: \"kubernetes.io/projected/3c7d6121-a834-42d0-9b78-583260fd4dce-kube-api-access-tk85r\") pod \"multus-additional-cni-plugins-p5bsf\" (UID: \"3c7d6121-a834-42d0-9b78-583260fd4dce\") " pod="openshift-multus/multus-additional-cni-plugins-p5bsf" Sep 09 14:33:07.592167 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.592123 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcsz5\" (UniqueName: \"kubernetes.io/projected/ab36c1e4-a65c-4cf9-ada4-f8a87d4b463b-kube-api-access-tcsz5\") pod \"network-metrics-daemon-wcj2p\" (UID: \"ab36c1e4-a65c-4cf9-ada4-f8a87d4b463b\") " pod="openshift-multus/network-metrics-daemon-wcj2p" Sep 09 14:33:07.592619 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.592600 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qm86k\" (UniqueName: \"kubernetes.io/projected/ab41acab-1926-436b-b208-c18b0b375e9f-kube-api-access-qm86k\") pod \"multus-w5kx5\" (UID: \"ab41acab-1926-436b-b208-c18b0b375e9f\") " pod="openshift-multus/multus-w5kx5" Sep 09 14:33:07.670645 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.670470 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-cqf8f" Sep 09 14:33:07.674865 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.674674 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-7xcn5" Sep 09 14:33:07.684341 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:07.684312 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod186aab90_266b_4eae_977a_3f053b423ef3.slice/crio-9c78ec8ef3dd9b1d0fac2d0cbab1839db1be54b2f63aedd9720dfeceec258480 WatchSource:0}: Error finding container 9c78ec8ef3dd9b1d0fac2d0cbab1839db1be54b2f63aedd9720dfeceec258480: Status 404 returned error can't find the container with id 9c78ec8ef3dd9b1d0fac2d0cbab1839db1be54b2f63aedd9720dfeceec258480 Sep 09 14:33:07.685983 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.685960 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-4kp4s" Sep 09 14:33:07.686565 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:07.686488 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedb6129a_6cbf_4d83_82b0_018fd0e73f8f.slice/crio-22dd98bbcda3bd5340403779b7674e10d142d7c789e540f28ef0f348661552ea WatchSource:0}: Error finding container 22dd98bbcda3bd5340403779b7674e10d142d7c789e540f28ef0f348661552ea: Status 404 returned error can't find the container with id 22dd98bbcda3bd5340403779b7674e10d142d7c789e540f28ef0f348661552ea Sep 09 14:33:07.690392 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.690188 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qlxsx" Sep 09 14:33:07.695960 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:07.695936 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb134513f_e0e7_452d_8b61_bae4c9007795.slice/crio-e94018750dbdc19b45ab1ec3bd064cf740e60f04656c04a826817c9ae4e73538 WatchSource:0}: Error finding container e94018750dbdc19b45ab1ec3bd064cf740e60f04656c04a826817c9ae4e73538: Status 404 returned error can't find the container with id e94018750dbdc19b45ab1ec3bd064cf740e60f04656c04a826817c9ae4e73538 Sep 09 14:33:07.697514 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.697347 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-p5wbc" Sep 09 14:33:07.703835 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:07.703810 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a445f65_2ccc_44f4_97ad_1eee2597b711.slice/crio-a59107195f40b6e2606e2ab3c4bb2da558699f299267010d007be9a254f17f36 WatchSource:0}: Error finding container a59107195f40b6e2606e2ab3c4bb2da558699f299267010d007be9a254f17f36: Status 404 returned error can't find the container with id a59107195f40b6e2606e2ab3c4bb2da558699f299267010d007be9a254f17f36 Sep 09 14:33:07.707261 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.707244 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5qkp6" Sep 09 14:33:07.711834 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.711812 2576 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 09 14:33:07.713982 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.713968 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-ppvhx" Sep 09 14:33:07.720707 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:07.720681 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90785a55_a3bd_41a7_8d33_85e3b82eb41f.slice/crio-370e3e5a547203ef2e27bfabcc6161f7650a188da36214e53e7d293b5274cf8b WatchSource:0}: Error finding container 370e3e5a547203ef2e27bfabcc6161f7650a188da36214e53e7d293b5274cf8b: Status 404 returned error can't find the container with id 370e3e5a547203ef2e27bfabcc6161f7650a188da36214e53e7d293b5274cf8b Sep 09 14:33:07.723550 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.723384 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-p5bsf" Sep 09 14:33:07.727953 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:07.727937 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-w5kx5" Sep 09 14:33:07.730656 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:07.730633 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod141380ba_1588_47e3_99ea_1de06df135e9.slice/crio-80b08cf735cab93afe9f00f5fae50b1cf44a7742652336ec9d783e5f8ee65155 WatchSource:0}: Error finding container 80b08cf735cab93afe9f00f5fae50b1cf44a7742652336ec9d783e5f8ee65155: Status 404 returned error can't find the container with id 80b08cf735cab93afe9f00f5fae50b1cf44a7742652336ec9d783e5f8ee65155 Sep 09 14:33:07.745538 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:07.745495 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab41acab_1926_436b_b208_c18b0b375e9f.slice/crio-02314fb5f8712df191b2d5196fa16f33c8599ed911246851af042273325414fa WatchSource:0}: Error finding container 02314fb5f8712df191b2d5196fa16f33c8599ed911246851af042273325414fa: Status 404 returned error can't find the container with id 02314fb5f8712df191b2d5196fa16f33c8599ed911246851af042273325414fa Sep 09 14:33:08.084480 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:08.084395 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab36c1e4-a65c-4cf9-ada4-f8a87d4b463b-metrics-certs\") pod \"network-metrics-daemon-wcj2p\" (UID: \"ab36c1e4-a65c-4cf9-ada4-f8a87d4b463b\") " pod="openshift-multus/network-metrics-daemon-wcj2p" Sep 09 14:33:08.084480 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:08.084468 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f286fc80-41f7-4b6f-8506-b5f0eb2629d3-original-pull-secret\") pod \"global-pull-secret-syncer-nqc8q\" (UID: \"f286fc80-41f7-4b6f-8506-b5f0eb2629d3\") " pod="kube-system/global-pull-secret-syncer-nqc8q" Sep 09 14:33:08.084681 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:08.084548 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 09 14:33:08.084681 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:08.084548 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 14:33:08.084681 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:08.084615 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f286fc80-41f7-4b6f-8506-b5f0eb2629d3-original-pull-secret podName:f286fc80-41f7-4b6f-8506-b5f0eb2629d3 nodeName:}" failed. No retries permitted until 2026-09-09 14:33:09.084598446 +0000 UTC m=+4.095514683 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f286fc80-41f7-4b6f-8506-b5f0eb2629d3-original-pull-secret") pod "global-pull-secret-syncer-nqc8q" (UID: "f286fc80-41f7-4b6f-8506-b5f0eb2629d3") : object "kube-system"/"original-pull-secret" not registered Sep 09 14:33:08.084681 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:08.084630 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab36c1e4-a65c-4cf9-ada4-f8a87d4b463b-metrics-certs podName:ab36c1e4-a65c-4cf9-ada4-f8a87d4b463b nodeName:}" failed. No retries permitted until 2026-09-09 14:33:09.084623869 +0000 UTC m=+4.095540098 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ab36c1e4-a65c-4cf9-ada4-f8a87d4b463b-metrics-certs") pod "network-metrics-daemon-wcj2p" (UID: "ab36c1e4-a65c-4cf9-ada4-f8a87d4b463b") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 14:33:08.185314 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:08.185250 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2krn\" (UniqueName: \"kubernetes.io/projected/ce063353-8e00-4701-8fb5-9b32aa0586e1-kube-api-access-l2krn\") pod \"network-check-target-5vdl5\" (UID: \"ce063353-8e00-4701-8fb5-9b32aa0586e1\") " pod="openshift-network-diagnostics/network-check-target-5vdl5" Sep 09 14:33:08.185490 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:08.185431 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 09 14:33:08.185490 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:08.185452 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 09 14:33:08.185490 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:08.185461 2576 projected.go:196] Error preparing data for projected volume kube-api-access-l2krn for pod openshift-network-diagnostics/network-check-target-5vdl5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 14:33:08.185612 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:08.185520 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ce063353-8e00-4701-8fb5-9b32aa0586e1-kube-api-access-l2krn podName:ce063353-8e00-4701-8fb5-9b32aa0586e1 nodeName:}" failed. No retries permitted until 2026-09-09 14:33:09.185502593 +0000 UTC m=+4.196418824 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-l2krn" (UniqueName: "kubernetes.io/projected/ce063353-8e00-4701-8fb5-9b32aa0586e1-kube-api-access-l2krn") pod "network-check-target-5vdl5" (UID: "ce063353-8e00-4701-8fb5-9b32aa0586e1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 14:33:08.408173 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:08.408092 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 14:28:07 +0000 UTC" deadline="2028-05-01 10:12:00.596942046 +0000 UTC" Sep 09 14:33:08.408173 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:08.408168 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14395h38m52.188777122s" Sep 09 14:33:08.513163 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:08.509694 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wcj2p" Sep 09 14:33:08.513163 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:08.509841 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wcj2p" podUID="ab36c1e4-a65c-4cf9-ada4-f8a87d4b463b" Sep 09 14:33:08.519920 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:08.519541 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p5bsf" event={"ID":"3c7d6121-a834-42d0-9b78-583260fd4dce","Type":"ContainerStarted","Data":"6d0554822562f698f41584bf698161e9f5ab53809817a5be80dd7672a8f675b6"} Sep 09 14:33:08.520915 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:08.520883 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qlxsx" event={"ID":"5a445f65-2ccc-44f4-97ad-1eee2597b711","Type":"ContainerStarted","Data":"a59107195f40b6e2606e2ab3c4bb2da558699f299267010d007be9a254f17f36"} Sep 09 14:33:08.522684 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:08.522655 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-4kp4s" event={"ID":"b134513f-e0e7-452d-8b61-bae4c9007795","Type":"ContainerStarted","Data":"e94018750dbdc19b45ab1ec3bd064cf740e60f04656c04a826817c9ae4e73538"} Sep 09 14:33:08.524618 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:08.524112 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7xcn5" event={"ID":"186aab90-266b-4eae-977a-3f053b423ef3","Type":"ContainerStarted","Data":"9c78ec8ef3dd9b1d0fac2d0cbab1839db1be54b2f63aedd9720dfeceec258480"} Sep 09 14:33:08.527504 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:08.527467 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-w5kx5" event={"ID":"ab41acab-1926-436b-b208-c18b0b375e9f","Type":"ContainerStarted","Data":"02314fb5f8712df191b2d5196fa16f33c8599ed911246851af042273325414fa"} Sep 09 14:33:08.528841 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:08.528816 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-ppvhx" event={"ID":"141380ba-1588-47e3-99ea-1de06df135e9","Type":"ContainerStarted","Data":"80b08cf735cab93afe9f00f5fae50b1cf44a7742652336ec9d783e5f8ee65155"} Sep 09 14:33:08.529936 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:08.529907 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5qkp6" event={"ID":"90785a55-a3bd-41a7-8d33-85e3b82eb41f","Type":"ContainerStarted","Data":"370e3e5a547203ef2e27bfabcc6161f7650a188da36214e53e7d293b5274cf8b"} Sep 09 14:33:08.531109 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:08.531086 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-p5wbc" event={"ID":"50c734e0-544e-44f4-9c79-a398e559a276","Type":"ContainerStarted","Data":"7106e944b1a115ade2c492f204bfbc6f66955ad997e29b680f871c818b3bef8c"} Sep 09 14:33:08.532461 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:08.532440 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-cqf8f" event={"ID":"edb6129a-6cbf-4d83-82b0-018fd0e73f8f","Type":"ContainerStarted","Data":"22dd98bbcda3bd5340403779b7674e10d142d7c789e540f28ef0f348661552ea"} Sep 09 14:33:09.092531 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:09.092490 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab36c1e4-a65c-4cf9-ada4-f8a87d4b463b-metrics-certs\") pod \"network-metrics-daemon-wcj2p\" (UID: \"ab36c1e4-a65c-4cf9-ada4-f8a87d4b463b\") " pod="openshift-multus/network-metrics-daemon-wcj2p" Sep 09 14:33:09.092685 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:09.092587 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f286fc80-41f7-4b6f-8506-b5f0eb2629d3-original-pull-secret\") pod \"global-pull-secret-syncer-nqc8q\" (UID: \"f286fc80-41f7-4b6f-8506-b5f0eb2629d3\") " pod="kube-system/global-pull-secret-syncer-nqc8q" Sep 09 14:33:09.092749 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:09.092721 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 09 14:33:09.092806 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:09.092800 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f286fc80-41f7-4b6f-8506-b5f0eb2629d3-original-pull-secret podName:f286fc80-41f7-4b6f-8506-b5f0eb2629d3 nodeName:}" failed. No retries permitted until 2026-09-09 14:33:11.09278165 +0000 UTC m=+6.103697882 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f286fc80-41f7-4b6f-8506-b5f0eb2629d3-original-pull-secret") pod "global-pull-secret-syncer-nqc8q" (UID: "f286fc80-41f7-4b6f-8506-b5f0eb2629d3") : object "kube-system"/"original-pull-secret" not registered Sep 09 14:33:09.093227 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:09.093209 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 14:33:09.093345 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:09.093267 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab36c1e4-a65c-4cf9-ada4-f8a87d4b463b-metrics-certs podName:ab36c1e4-a65c-4cf9-ada4-f8a87d4b463b nodeName:}" failed. No retries permitted until 2026-09-09 14:33:11.093250086 +0000 UTC m=+6.104166320 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ab36c1e4-a65c-4cf9-ada4-f8a87d4b463b-metrics-certs") pod "network-metrics-daemon-wcj2p" (UID: "ab36c1e4-a65c-4cf9-ada4-f8a87d4b463b") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 14:33:09.194041 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:09.193703 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2krn\" (UniqueName: \"kubernetes.io/projected/ce063353-8e00-4701-8fb5-9b32aa0586e1-kube-api-access-l2krn\") pod \"network-check-target-5vdl5\" (UID: \"ce063353-8e00-4701-8fb5-9b32aa0586e1\") " pod="openshift-network-diagnostics/network-check-target-5vdl5" Sep 09 14:33:09.194041 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:09.193891 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 09 14:33:09.194279 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:09.194062 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 09 14:33:09.194279 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:09.194079 2576 projected.go:196] Error preparing data for projected volume kube-api-access-l2krn for pod openshift-network-diagnostics/network-check-target-5vdl5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 14:33:09.194279 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:09.194145 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ce063353-8e00-4701-8fb5-9b32aa0586e1-kube-api-access-l2krn podName:ce063353-8e00-4701-8fb5-9b32aa0586e1 nodeName:}" failed. No retries permitted until 2026-09-09 14:33:11.194125533 +0000 UTC m=+6.205041785 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-l2krn" (UniqueName: "kubernetes.io/projected/ce063353-8e00-4701-8fb5-9b32aa0586e1-kube-api-access-l2krn") pod "network-check-target-5vdl5" (UID: "ce063353-8e00-4701-8fb5-9b32aa0586e1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 14:33:09.409355 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:09.409215 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 14:28:07 +0000 UTC" deadline="2028-06-19 14:30:18.914794354 +0000 UTC" Sep 09 14:33:09.409355 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:09.409261 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15575h57m9.505539677s" Sep 09 14:33:09.513483 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:09.511747 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5vdl5" Sep 09 14:33:09.513483 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:09.511865 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5vdl5" podUID="ce063353-8e00-4701-8fb5-9b32aa0586e1" Sep 09 14:33:09.513483 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:09.512177 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nqc8q" Sep 09 14:33:09.513483 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:09.512247 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nqc8q" podUID="f286fc80-41f7-4b6f-8506-b5f0eb2629d3" Sep 09 14:33:09.544523 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:09.544476 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-66.ec2.internal" event={"ID":"3aba2b4456e66055b3846a53f8393ab4","Type":"ContainerStarted","Data":"237234b836a079cd506ce54a24a6343733ac17f28a33f1a94bb9e4af9e5cb681"} Sep 09 14:33:10.509494 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:10.509456 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wcj2p" Sep 09 14:33:10.509975 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:10.509603 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wcj2p" podUID="ab36c1e4-a65c-4cf9-ada4-f8a87d4b463b" Sep 09 14:33:10.558745 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:10.558273 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-66.ec2.internal" event={"ID":"9af3564e64deec6f121c01d714dc0e5a","Type":"ContainerStarted","Data":"5bd43db37a6c12d9d21fc6cb2c596cad31e160b27d0a51a705600067663e77e6"} Sep 09 14:33:10.578030 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:10.577953 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-66.ec2.internal" podStartSLOduration=4.577927564 podStartE2EDuration="4.577927564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 14:33:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:33:09.561077707 +0000 UTC m=+4.571993970" watchObservedRunningTime="2026-09-09 14:33:10.577927564 +0000 UTC m=+5.588843816" Sep 09 14:33:11.108746 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:11.108709 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab36c1e4-a65c-4cf9-ada4-f8a87d4b463b-metrics-certs\") pod \"network-metrics-daemon-wcj2p\" (UID: \"ab36c1e4-a65c-4cf9-ada4-f8a87d4b463b\") " pod="openshift-multus/network-metrics-daemon-wcj2p" Sep 09 14:33:11.108946 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:11.108803 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f286fc80-41f7-4b6f-8506-b5f0eb2629d3-original-pull-secret\") pod \"global-pull-secret-syncer-nqc8q\" (UID: \"f286fc80-41f7-4b6f-8506-b5f0eb2629d3\") " pod="kube-system/global-pull-secret-syncer-nqc8q" Sep 09 14:33:11.108946 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:11.108862 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 14:33:11.109075 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:11.108952 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab36c1e4-a65c-4cf9-ada4-f8a87d4b463b-metrics-certs podName:ab36c1e4-a65c-4cf9-ada4-f8a87d4b463b nodeName:}" failed. No retries permitted until 2026-09-09 14:33:15.108927898 +0000 UTC m=+10.119844141 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ab36c1e4-a65c-4cf9-ada4-f8a87d4b463b-metrics-certs") pod "network-metrics-daemon-wcj2p" (UID: "ab36c1e4-a65c-4cf9-ada4-f8a87d4b463b") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 14:33:11.109075 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:11.108950 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 09 14:33:11.109075 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:11.109014 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f286fc80-41f7-4b6f-8506-b5f0eb2629d3-original-pull-secret podName:f286fc80-41f7-4b6f-8506-b5f0eb2629d3 nodeName:}" failed. No retries permitted until 2026-09-09 14:33:15.108996951 +0000 UTC m=+10.119913179 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f286fc80-41f7-4b6f-8506-b5f0eb2629d3-original-pull-secret") pod "global-pull-secret-syncer-nqc8q" (UID: "f286fc80-41f7-4b6f-8506-b5f0eb2629d3") : object "kube-system"/"original-pull-secret" not registered Sep 09 14:33:11.209934 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:11.209879 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2krn\" (UniqueName: \"kubernetes.io/projected/ce063353-8e00-4701-8fb5-9b32aa0586e1-kube-api-access-l2krn\") pod \"network-check-target-5vdl5\" (UID: \"ce063353-8e00-4701-8fb5-9b32aa0586e1\") " pod="openshift-network-diagnostics/network-check-target-5vdl5" Sep 09 14:33:11.210129 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:11.210084 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 09 14:33:11.210129 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:11.210103 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 09 14:33:11.210129 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:11.210116 2576 projected.go:196] Error preparing data for projected volume kube-api-access-l2krn for pod openshift-network-diagnostics/network-check-target-5vdl5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 14:33:11.210287 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:11.210177 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ce063353-8e00-4701-8fb5-9b32aa0586e1-kube-api-access-l2krn podName:ce063353-8e00-4701-8fb5-9b32aa0586e1 nodeName:}" failed. No retries permitted until 2026-09-09 14:33:15.210159028 +0000 UTC m=+10.221075263 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-l2krn" (UniqueName: "kubernetes.io/projected/ce063353-8e00-4701-8fb5-9b32aa0586e1-kube-api-access-l2krn") pod "network-check-target-5vdl5" (UID: "ce063353-8e00-4701-8fb5-9b32aa0586e1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 14:33:11.510637 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:11.510520 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5vdl5" Sep 09 14:33:11.511011 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:11.510661 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5vdl5" podUID="ce063353-8e00-4701-8fb5-9b32aa0586e1" Sep 09 14:33:11.511055 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:11.511033 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nqc8q" Sep 09 14:33:11.511163 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:11.511143 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nqc8q" podUID="f286fc80-41f7-4b6f-8506-b5f0eb2629d3" Sep 09 14:33:11.569108 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:11.569068 2576 generic.go:356] "Generic (PLEG): container finished" podID="9af3564e64deec6f121c01d714dc0e5a" containerID="5bd43db37a6c12d9d21fc6cb2c596cad31e160b27d0a51a705600067663e77e6" exitCode=0 Sep 09 14:33:11.569295 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:11.569132 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-66.ec2.internal" event={"ID":"9af3564e64deec6f121c01d714dc0e5a","Type":"ContainerDied","Data":"5bd43db37a6c12d9d21fc6cb2c596cad31e160b27d0a51a705600067663e77e6"} Sep 09 14:33:12.509734 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:12.509698 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wcj2p" Sep 09 14:33:12.509913 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:12.509854 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wcj2p" podUID="ab36c1e4-a65c-4cf9-ada4-f8a87d4b463b" Sep 09 14:33:13.510528 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:13.510398 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nqc8q" Sep 09 14:33:13.510970 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:13.510531 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nqc8q" podUID="f286fc80-41f7-4b6f-8506-b5f0eb2629d3" Sep 09 14:33:13.510970 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:13.510886 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5vdl5" Sep 09 14:33:13.510970 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:13.510956 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5vdl5" podUID="ce063353-8e00-4701-8fb5-9b32aa0586e1" Sep 09 14:33:14.509314 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:14.509277 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wcj2p" Sep 09 14:33:14.509503 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:14.509430 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wcj2p" podUID="ab36c1e4-a65c-4cf9-ada4-f8a87d4b463b" Sep 09 14:33:15.144163 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:15.144072 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f286fc80-41f7-4b6f-8506-b5f0eb2629d3-original-pull-secret\") pod \"global-pull-secret-syncer-nqc8q\" (UID: \"f286fc80-41f7-4b6f-8506-b5f0eb2629d3\") " pod="kube-system/global-pull-secret-syncer-nqc8q" Sep 09 14:33:15.144163 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:15.144139 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab36c1e4-a65c-4cf9-ada4-f8a87d4b463b-metrics-certs\") pod \"network-metrics-daemon-wcj2p\" (UID: \"ab36c1e4-a65c-4cf9-ada4-f8a87d4b463b\") " pod="openshift-multus/network-metrics-daemon-wcj2p" Sep 09 14:33:15.144681 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:15.144279 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 14:33:15.144681 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:15.144351 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab36c1e4-a65c-4cf9-ada4-f8a87d4b463b-metrics-certs podName:ab36c1e4-a65c-4cf9-ada4-f8a87d4b463b nodeName:}" failed. No retries permitted until 2026-09-09 14:33:23.14433414 +0000 UTC m=+18.155250375 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ab36c1e4-a65c-4cf9-ada4-f8a87d4b463b-metrics-certs") pod "network-metrics-daemon-wcj2p" (UID: "ab36c1e4-a65c-4cf9-ada4-f8a87d4b463b") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 14:33:15.144801 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:15.144711 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 09 14:33:15.144801 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:15.144761 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f286fc80-41f7-4b6f-8506-b5f0eb2629d3-original-pull-secret podName:f286fc80-41f7-4b6f-8506-b5f0eb2629d3 nodeName:}" failed. No retries permitted until 2026-09-09 14:33:23.144746138 +0000 UTC m=+18.155662373 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f286fc80-41f7-4b6f-8506-b5f0eb2629d3-original-pull-secret") pod "global-pull-secret-syncer-nqc8q" (UID: "f286fc80-41f7-4b6f-8506-b5f0eb2629d3") : object "kube-system"/"original-pull-secret" not registered Sep 09 14:33:15.245316 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:15.245260 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2krn\" (UniqueName: \"kubernetes.io/projected/ce063353-8e00-4701-8fb5-9b32aa0586e1-kube-api-access-l2krn\") pod \"network-check-target-5vdl5\" (UID: \"ce063353-8e00-4701-8fb5-9b32aa0586e1\") " pod="openshift-network-diagnostics/network-check-target-5vdl5" Sep 09 14:33:15.245481 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:15.245435 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 09 14:33:15.245481 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:15.245456 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 09 14:33:15.245481 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:15.245469 2576 projected.go:196] Error preparing data for projected volume kube-api-access-l2krn for pod openshift-network-diagnostics/network-check-target-5vdl5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 14:33:15.245629 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:15.245525 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ce063353-8e00-4701-8fb5-9b32aa0586e1-kube-api-access-l2krn podName:ce063353-8e00-4701-8fb5-9b32aa0586e1 nodeName:}" failed. No retries permitted until 2026-09-09 14:33:23.245510756 +0000 UTC m=+18.256426989 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-l2krn" (UniqueName: "kubernetes.io/projected/ce063353-8e00-4701-8fb5-9b32aa0586e1-kube-api-access-l2krn") pod "network-check-target-5vdl5" (UID: "ce063353-8e00-4701-8fb5-9b32aa0586e1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 14:33:15.510124 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:15.509792 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5vdl5" Sep 09 14:33:15.510124 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:15.509887 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5vdl5" podUID="ce063353-8e00-4701-8fb5-9b32aa0586e1" Sep 09 14:33:15.510359 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:15.510215 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nqc8q" Sep 09 14:33:15.510359 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:15.510320 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nqc8q" podUID="f286fc80-41f7-4b6f-8506-b5f0eb2629d3" Sep 09 14:33:16.510028 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:16.509994 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wcj2p" Sep 09 14:33:16.510523 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:16.510136 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wcj2p" podUID="ab36c1e4-a65c-4cf9-ada4-f8a87d4b463b" Sep 09 14:33:17.509587 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:17.509544 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5vdl5" Sep 09 14:33:17.509764 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:17.509554 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nqc8q" Sep 09 14:33:17.509764 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:17.509652 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5vdl5" podUID="ce063353-8e00-4701-8fb5-9b32aa0586e1" Sep 09 14:33:17.509908 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:17.509763 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nqc8q" podUID="f286fc80-41f7-4b6f-8506-b5f0eb2629d3" Sep 09 14:33:18.509210 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:18.509191 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wcj2p" Sep 09 14:33:18.509509 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:18.509291 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wcj2p" podUID="ab36c1e4-a65c-4cf9-ada4-f8a87d4b463b" Sep 09 14:33:19.509367 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:19.509336 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5vdl5" Sep 09 14:33:19.509772 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:19.509437 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5vdl5" podUID="ce063353-8e00-4701-8fb5-9b32aa0586e1" Sep 09 14:33:19.509869 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:19.509848 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nqc8q" Sep 09 14:33:19.509990 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:19.509968 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nqc8q" podUID="f286fc80-41f7-4b6f-8506-b5f0eb2629d3" Sep 09 14:33:20.509416 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:20.509379 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wcj2p" Sep 09 14:33:20.509862 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:20.509528 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wcj2p" podUID="ab36c1e4-a65c-4cf9-ada4-f8a87d4b463b" Sep 09 14:33:21.509794 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:21.509761 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5vdl5" Sep 09 14:33:21.510259 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:21.509892 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nqc8q" Sep 09 14:33:21.510259 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:21.509970 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nqc8q" podUID="f286fc80-41f7-4b6f-8506-b5f0eb2629d3" Sep 09 14:33:21.510259 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:21.509888 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5vdl5" podUID="ce063353-8e00-4701-8fb5-9b32aa0586e1" Sep 09 14:33:22.510116 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:22.510069 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wcj2p" Sep 09 14:33:22.510643 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:22.510195 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wcj2p" podUID="ab36c1e4-a65c-4cf9-ada4-f8a87d4b463b" Sep 09 14:33:23.205147 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:23.205099 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f286fc80-41f7-4b6f-8506-b5f0eb2629d3-original-pull-secret\") pod \"global-pull-secret-syncer-nqc8q\" (UID: \"f286fc80-41f7-4b6f-8506-b5f0eb2629d3\") " pod="kube-system/global-pull-secret-syncer-nqc8q" Sep 09 14:33:23.205478 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:23.205162 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab36c1e4-a65c-4cf9-ada4-f8a87d4b463b-metrics-certs\") pod \"network-metrics-daemon-wcj2p\" (UID: \"ab36c1e4-a65c-4cf9-ada4-f8a87d4b463b\") " pod="openshift-multus/network-metrics-daemon-wcj2p" Sep 09 14:33:23.205478 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:23.205262 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 09 14:33:23.205478 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:23.205353 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f286fc80-41f7-4b6f-8506-b5f0eb2629d3-original-pull-secret podName:f286fc80-41f7-4b6f-8506-b5f0eb2629d3 nodeName:}" failed. No retries permitted until 2026-09-09 14:33:39.205336505 +0000 UTC m=+34.216252735 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f286fc80-41f7-4b6f-8506-b5f0eb2629d3-original-pull-secret") pod "global-pull-secret-syncer-nqc8q" (UID: "f286fc80-41f7-4b6f-8506-b5f0eb2629d3") : object "kube-system"/"original-pull-secret" not registered Sep 09 14:33:23.205679 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:23.205267 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 14:33:23.205739 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:23.205719 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab36c1e4-a65c-4cf9-ada4-f8a87d4b463b-metrics-certs podName:ab36c1e4-a65c-4cf9-ada4-f8a87d4b463b nodeName:}" failed. No retries permitted until 2026-09-09 14:33:39.205702247 +0000 UTC m=+34.216618532 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ab36c1e4-a65c-4cf9-ada4-f8a87d4b463b-metrics-certs") pod "network-metrics-daemon-wcj2p" (UID: "ab36c1e4-a65c-4cf9-ada4-f8a87d4b463b") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 14:33:23.305897 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:23.305858 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2krn\" (UniqueName: \"kubernetes.io/projected/ce063353-8e00-4701-8fb5-9b32aa0586e1-kube-api-access-l2krn\") pod \"network-check-target-5vdl5\" (UID: \"ce063353-8e00-4701-8fb5-9b32aa0586e1\") " pod="openshift-network-diagnostics/network-check-target-5vdl5" Sep 09 14:33:23.306087 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:23.306066 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 09 14:33:23.306154 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:23.306093 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 09 14:33:23.306154 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:23.306108 2576 projected.go:196] Error preparing data for projected volume kube-api-access-l2krn for pod openshift-network-diagnostics/network-check-target-5vdl5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 14:33:23.306239 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:23.306170 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ce063353-8e00-4701-8fb5-9b32aa0586e1-kube-api-access-l2krn podName:ce063353-8e00-4701-8fb5-9b32aa0586e1 nodeName:}" failed. No retries permitted until 2026-09-09 14:33:39.30615232 +0000 UTC m=+34.317068554 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-l2krn" (UniqueName: "kubernetes.io/projected/ce063353-8e00-4701-8fb5-9b32aa0586e1-kube-api-access-l2krn") pod "network-check-target-5vdl5" (UID: "ce063353-8e00-4701-8fb5-9b32aa0586e1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 14:33:23.509413 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:23.509336 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5vdl5" Sep 09 14:33:23.509580 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:23.509455 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5vdl5" podUID="ce063353-8e00-4701-8fb5-9b32aa0586e1" Sep 09 14:33:23.509580 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:23.509521 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nqc8q" Sep 09 14:33:23.509697 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:23.509635 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nqc8q" podUID="f286fc80-41f7-4b6f-8506-b5f0eb2629d3" Sep 09 14:33:24.509230 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:24.509189 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wcj2p" Sep 09 14:33:24.509664 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:24.509337 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wcj2p" podUID="ab36c1e4-a65c-4cf9-ada4-f8a87d4b463b" Sep 09 14:33:25.510594 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:25.509901 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5vdl5" Sep 09 14:33:25.510594 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:25.509931 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nqc8q" Sep 09 14:33:25.510594 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:25.510009 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5vdl5" podUID="ce063353-8e00-4701-8fb5-9b32aa0586e1" Sep 09 14:33:25.510594 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:25.510160 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nqc8q" podUID="f286fc80-41f7-4b6f-8506-b5f0eb2629d3" Sep 09 14:33:26.509864 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:26.509647 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wcj2p" Sep 09 14:33:26.510013 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:26.509985 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wcj2p" podUID="ab36c1e4-a65c-4cf9-ada4-f8a87d4b463b" Sep 09 14:33:26.600491 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:26.600461 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-w5kx5" event={"ID":"ab41acab-1926-436b-b208-c18b0b375e9f","Type":"ContainerStarted","Data":"8f899f7ee4d1a0c3a16fa2af23f860a095761168d242ca80196a283d7eaf2269"} Sep 09 14:33:26.602034 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:26.602011 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-ppvhx" event={"ID":"141380ba-1588-47e3-99ea-1de06df135e9","Type":"ContainerStarted","Data":"13ef40dbef65d4e8f5650c5766022957f31453dcb7fac92e11a336417b17f1ac"} Sep 09 14:33:26.604046 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:26.604012 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5qkp6" event={"ID":"90785a55-a3bd-41a7-8d33-85e3b82eb41f","Type":"ContainerStarted","Data":"01c8844aa00a6c72e272b40f393deec02ea62037810ab22f1bec452942ef47ea"} Sep 09 14:33:26.605685 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:26.605662 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-p5wbc" event={"ID":"50c734e0-544e-44f4-9c79-a398e559a276","Type":"ContainerStarted","Data":"642e493154c52e06fa2f9b8ed9b31c0d762b2cd6f45a0727e5cfe425349524a0"} Sep 09 14:33:26.607351 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:26.607330 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-cqf8f" event={"ID":"edb6129a-6cbf-4d83-82b0-018fd0e73f8f","Type":"ContainerStarted","Data":"493ef21e30b0ba302b5d4c824663cea39e01b5d191cf231b5936f8cea2fb6ef3"} Sep 09 14:33:26.611108 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:26.611087 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-66.ec2.internal" event={"ID":"9af3564e64deec6f121c01d714dc0e5a","Type":"ContainerStarted","Data":"27caa4c79c9ba24358b2bbbdfe6e1e90ad3dd8f6cccda33b75978aef1fe7586a"} Sep 09 14:33:26.612996 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:26.612970 2576 generic.go:356] "Generic (PLEG): container finished" podID="3c7d6121-a834-42d0-9b78-583260fd4dce" containerID="06adfbb40f1383b6736d774a69d5b396285489827af0bea18f53a4e9adceb5d4" exitCode=0 Sep 09 14:33:26.613095 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:26.613054 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p5bsf" event={"ID":"3c7d6121-a834-42d0-9b78-583260fd4dce","Type":"ContainerDied","Data":"06adfbb40f1383b6736d774a69d5b396285489827af0bea18f53a4e9adceb5d4"} Sep 09 14:33:26.614683 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:26.614661 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qlxsx" event={"ID":"5a445f65-2ccc-44f4-97ad-1eee2597b711","Type":"ContainerStarted","Data":"8e2230ca1da47bd6bb8825101383b1316f19a0708eaabd581a2aa24fc2524f14"} Sep 09 14:33:26.616962 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:26.616940 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7xcn5_186aab90-266b-4eae-977a-3f053b423ef3/ovn-acl-logging/0.log" Sep 09 14:33:26.617283 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:26.617264 2576 generic.go:356] "Generic (PLEG): container finished" podID="186aab90-266b-4eae-977a-3f053b423ef3" containerID="03d53257103548056028e3c9fa25a047a1559c4bf854a2a70f2c21d62d1fd0fa" exitCode=1 Sep 09 14:33:26.617373 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:26.617292 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7xcn5" event={"ID":"186aab90-266b-4eae-977a-3f053b423ef3","Type":"ContainerStarted","Data":"67aab9ed4e9be2c16d18ad728651b6336c5410d9fad7c33d9eb82643ac577d85"} Sep 09 14:33:26.617373 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:26.617359 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7xcn5" event={"ID":"186aab90-266b-4eae-977a-3f053b423ef3","Type":"ContainerStarted","Data":"003ceb3a81d22ad1c1707b6c7fe1bbf840b1aa43c9e77df909925c693878dff0"} Sep 09 14:33:26.617493 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:26.617390 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7xcn5" event={"ID":"186aab90-266b-4eae-977a-3f053b423ef3","Type":"ContainerDied","Data":"03d53257103548056028e3c9fa25a047a1559c4bf854a2a70f2c21d62d1fd0fa"} Sep 09 14:33:26.617493 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:26.617419 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7xcn5" event={"ID":"186aab90-266b-4eae-977a-3f053b423ef3","Type":"ContainerStarted","Data":"1947c0cba2c4aef201447679aafff68809d7caf119b3b903276ed5f089caef54"} Sep 09 14:33:26.620093 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:26.620049 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-w5kx5" podStartSLOduration=2.902418971 podStartE2EDuration="21.620034307s" totalImagesPullingTime="18.717615336s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 14:33:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:33:26.619857076 +0000 UTC m=+21.630773347" watchObservedRunningTime="2026-09-09 14:33:26.620034307 +0000 UTC m=+21.630950568" Sep 09 14:33:26.656939 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:26.656898 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-cqf8f" podStartSLOduration=14.550071785 podStartE2EDuration="21.656883174s" totalImagesPullingTime="7.106811389s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 14:33:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:33:26.656404066 +0000 UTC m=+21.667320317" watchObservedRunningTime="2026-09-09 14:33:26.656883174 +0000 UTC m=+21.667799436" Sep 09 14:33:26.672875 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:26.672835 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-p5wbc" podStartSLOduration=3.334954788 podStartE2EDuration="21.672819378s" totalImagesPullingTime="18.33786459s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 14:33:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:33:26.672496972 +0000 UTC m=+21.683413222" watchObservedRunningTime="2026-09-09 14:33:26.672819378 +0000 UTC m=+21.683735629" Sep 09 14:33:26.689382 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:26.689331 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-5qkp6" podStartSLOduration=3.365568835 podStartE2EDuration="21.689293105s" totalImagesPullingTime="18.32372427s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 14:33:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:33:26.689287642 +0000 UTC m=+21.700203960" watchObservedRunningTime="2026-09-09 14:33:26.689293105 +0000 UTC m=+21.700209357" Sep 09 14:33:26.727313 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:26.727251 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-66.ec2.internal" podStartSLOduration=20.727235362000002 podStartE2EDuration="20.727235362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 14:33:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:33:26.726699939 +0000 UTC m=+21.737616190" watchObservedRunningTime="2026-09-09 14:33:26.727235362 +0000 UTC m=+21.738151613" Sep 09 14:33:26.753380 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:26.753327 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-ppvhx" podStartSLOduration=3.438861697 podStartE2EDuration="21.75331201s" totalImagesPullingTime="18.314450313s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 14:33:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:33:26.753184478 +0000 UTC m=+21.764100730" watchObservedRunningTime="2026-09-09 14:33:26.75331201 +0000 UTC m=+21.764228251" Sep 09 14:33:27.509103 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:27.509071 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5vdl5" Sep 09 14:33:27.509103 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:27.509110 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nqc8q" Sep 09 14:33:27.509407 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:27.509190 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5vdl5" podUID="ce063353-8e00-4701-8fb5-9b32aa0586e1" Sep 09 14:33:27.509407 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:27.509357 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nqc8q" podUID="f286fc80-41f7-4b6f-8506-b5f0eb2629d3" Sep 09 14:33:27.621055 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:27.620975 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-4kp4s" event={"ID":"b134513f-e0e7-452d-8b61-bae4c9007795","Type":"ContainerStarted","Data":"9be73330d9b75df35225c7c5c603b9e1f18c1cdd7f3f7cced243f38467ef2146"} Sep 09 14:33:27.624244 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:27.624226 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7xcn5_186aab90-266b-4eae-977a-3f053b423ef3/ovn-acl-logging/0.log" Sep 09 14:33:27.624691 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:27.624646 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7xcn5" event={"ID":"186aab90-266b-4eae-977a-3f053b423ef3","Type":"ContainerStarted","Data":"3b45915b6c67a7d1289dd7f599a18ebaef56d8acaa27246ae8621479d75794f9"} Sep 09 14:33:27.624796 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:27.624779 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7xcn5" event={"ID":"186aab90-266b-4eae-977a-3f053b423ef3","Type":"ContainerStarted","Data":"88d3519ac35e8014bd69543849fed9d87de46774d46d20e077af9066ed7f47c9"} Sep 09 14:33:27.636161 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:27.636120 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-4kp4s" podStartSLOduration=4.2849751959999995 podStartE2EDuration="22.636108029s" totalImagesPullingTime="18.351132833s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 14:33:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:33:27.636023593 +0000 UTC m=+22.646939874" watchObservedRunningTime="2026-09-09 14:33:27.636108029 +0000 UTC m=+22.647024280" Sep 09 14:33:28.089935 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:28.089842 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-ppvhx" Sep 09 14:33:28.090581 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:28.090558 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-ppvhx" Sep 09 14:33:28.509143 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:28.509073 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wcj2p" Sep 09 14:33:28.509285 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:28.509181 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wcj2p" podUID="ab36c1e4-a65c-4cf9-ada4-f8a87d4b463b" Sep 09 14:33:28.626624 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:28.626594 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-ppvhx" Sep 09 14:33:28.627478 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:28.627458 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-ppvhx" Sep 09 14:33:29.511195 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:29.509320 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5vdl5" Sep 09 14:33:29.511195 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:29.509433 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5vdl5" podUID="ce063353-8e00-4701-8fb5-9b32aa0586e1" Sep 09 14:33:29.511195 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:29.509327 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nqc8q" Sep 09 14:33:29.511195 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:29.509775 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nqc8q" podUID="f286fc80-41f7-4b6f-8506-b5f0eb2629d3" Sep 09 14:33:29.599747 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:29.599727 2576 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 09 14:33:29.629707 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:29.629682 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qlxsx" event={"ID":"5a445f65-2ccc-44f4-97ad-1eee2597b711","Type":"ContainerStarted","Data":"1ad9ce584578777cdf5949e05f8e44b3442a2ebdc1c2356db6cb773200be552d"} Sep 09 14:33:29.632667 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:29.632651 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7xcn5_186aab90-266b-4eae-977a-3f053b423ef3/ovn-acl-logging/0.log" Sep 09 14:33:29.633000 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:29.632980 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7xcn5" event={"ID":"186aab90-266b-4eae-977a-3f053b423ef3","Type":"ContainerStarted","Data":"33acd5e70720d10388d73d577c474485098c08cbce2c907b84f6f7981a391863"} Sep 09 14:33:30.450480 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:30.450238 2576 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-09T14:33:29.599743614Z","UUID":"59f4125f-7e58-4829-a3ce-d9fd4888667e","Handler":null,"Name":"","Endpoint":""} Sep 09 14:33:30.451841 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:30.451820 2576 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 09 14:33:30.451841 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:30.451850 2576 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 09 14:33:30.509671 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:30.509640 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wcj2p" Sep 09 14:33:30.509808 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:30.509787 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wcj2p" podUID="ab36c1e4-a65c-4cf9-ada4-f8a87d4b463b" Sep 09 14:33:31.509738 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:31.509702 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nqc8q" Sep 09 14:33:31.510247 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:31.509746 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5vdl5" Sep 09 14:33:31.510247 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:31.509852 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5vdl5" podUID="ce063353-8e00-4701-8fb5-9b32aa0586e1" Sep 09 14:33:31.510247 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:31.509850 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nqc8q" podUID="f286fc80-41f7-4b6f-8506-b5f0eb2629d3" Sep 09 14:33:32.509859 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:32.509700 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wcj2p" Sep 09 14:33:32.510531 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:32.509955 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wcj2p" podUID="ab36c1e4-a65c-4cf9-ada4-f8a87d4b463b" Sep 09 14:33:32.641295 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:32.641221 2576 generic.go:356] "Generic (PLEG): container finished" podID="3c7d6121-a834-42d0-9b78-583260fd4dce" containerID="3c635f39bed2b1f067c470a58b4503dde2004385210fefca674b48dc0fabd231" exitCode=0 Sep 09 14:33:32.641458 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:32.641323 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p5bsf" event={"ID":"3c7d6121-a834-42d0-9b78-583260fd4dce","Type":"ContainerDied","Data":"3c635f39bed2b1f067c470a58b4503dde2004385210fefca674b48dc0fabd231"} Sep 09 14:33:32.643562 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:32.643535 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qlxsx" event={"ID":"5a445f65-2ccc-44f4-97ad-1eee2597b711","Type":"ContainerStarted","Data":"bb1b6e40c07c54edd9630efa8604e62696758878454f9d908143321e8760821e"} Sep 09 14:33:32.646837 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:32.646812 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7xcn5_186aab90-266b-4eae-977a-3f053b423ef3/ovn-acl-logging/0.log" Sep 09 14:33:32.647235 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:32.647213 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7xcn5" event={"ID":"186aab90-266b-4eae-977a-3f053b423ef3","Type":"ContainerStarted","Data":"8619b31c2575844cc2b7107a3982e3a1d843c5658f99661610f4e2485f29c4de"} Sep 09 14:33:32.647496 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:32.647479 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7xcn5" Sep 09 14:33:32.647572 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:32.647551 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7xcn5" Sep 09 14:33:32.647615 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:32.647601 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7xcn5" Sep 09 14:33:32.647685 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:32.647661 2576 scope.go:117] "RemoveContainer" containerID="03d53257103548056028e3c9fa25a047a1559c4bf854a2a70f2c21d62d1fd0fa" Sep 09 14:33:32.663461 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:32.663439 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7xcn5" Sep 09 14:33:32.663584 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:32.663572 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7xcn5" Sep 09 14:33:33.512983 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:33.512956 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5vdl5" Sep 09 14:33:33.513469 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:33.513048 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5vdl5" podUID="ce063353-8e00-4701-8fb5-9b32aa0586e1" Sep 09 14:33:33.513469 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:33.513098 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nqc8q" Sep 09 14:33:33.513469 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:33.513179 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nqc8q" podUID="f286fc80-41f7-4b6f-8506-b5f0eb2629d3" Sep 09 14:33:33.575214 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:33.575161 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qlxsx" podStartSLOduration=4.402785617 podStartE2EDuration="28.575141401s" totalImagesPullingTime="24.172355784s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 14:33:05 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:33:32.709445447 +0000 UTC m=+27.720361816" watchObservedRunningTime="2026-09-09 14:33:33.575141401 +0000 UTC m=+28.586057654" Sep 09 14:33:33.575910 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:33.575890 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-nqc8q"] Sep 09 14:33:33.583058 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:33.583036 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-5vdl5"] Sep 09 14:33:33.583605 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:33.583586 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-wcj2p"] Sep 09 14:33:33.583687 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:33.583676 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wcj2p" Sep 09 14:33:33.583765 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:33.583748 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wcj2p" podUID="ab36c1e4-a65c-4cf9-ada4-f8a87d4b463b" Sep 09 14:33:33.653670 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:33.653648 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7xcn5_186aab90-266b-4eae-977a-3f053b423ef3/ovn-acl-logging/0.log" Sep 09 14:33:33.654103 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:33.654084 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5vdl5" Sep 09 14:33:33.654262 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:33.654078 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7xcn5" event={"ID":"186aab90-266b-4eae-977a-3f053b423ef3","Type":"ContainerStarted","Data":"2a8eb2b0679af409f1186a3a60aa66187e228c042d695c4f6241f3ee7d8b965d"} Sep 09 14:33:33.654397 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:33.654110 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nqc8q" Sep 09 14:33:33.654450 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:33.654405 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5vdl5" podUID="ce063353-8e00-4701-8fb5-9b32aa0586e1" Sep 09 14:33:33.654533 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:33.654506 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nqc8q" podUID="f286fc80-41f7-4b6f-8506-b5f0eb2629d3" Sep 09 14:33:33.682258 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:33.682215 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-7xcn5" podStartSLOduration=10.270414741 podStartE2EDuration="28.682201159s" totalImagesPullingTime="18.411786418s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 14:33:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:33:33.68026962 +0000 UTC m=+28.691185888" watchObservedRunningTime="2026-09-09 14:33:33.682201159 +0000 UTC m=+28.693117412" Sep 09 14:33:34.657696 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:34.657659 2576 generic.go:356] "Generic (PLEG): container finished" podID="3c7d6121-a834-42d0-9b78-583260fd4dce" containerID="06beeaba67f21a1971529cd0c4aa87cda8c7e4637bbaa0e1d7c6050b508d8c29" exitCode=0 Sep 09 14:33:34.658060 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:34.657745 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p5bsf" event={"ID":"3c7d6121-a834-42d0-9b78-583260fd4dce","Type":"ContainerDied","Data":"06beeaba67f21a1971529cd0c4aa87cda8c7e4637bbaa0e1d7c6050b508d8c29"} Sep 09 14:33:35.510040 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:35.510013 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wcj2p" Sep 09 14:33:35.510166 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:35.510139 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nqc8q" Sep 09 14:33:35.510166 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:35.510153 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5vdl5" Sep 09 14:33:35.510270 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:35.510135 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wcj2p" podUID="ab36c1e4-a65c-4cf9-ada4-f8a87d4b463b" Sep 09 14:33:35.510270 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:35.510240 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nqc8q" podUID="f286fc80-41f7-4b6f-8506-b5f0eb2629d3" Sep 09 14:33:35.510478 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:35.510343 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5vdl5" podUID="ce063353-8e00-4701-8fb5-9b32aa0586e1" Sep 09 14:33:35.662225 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:35.662198 2576 generic.go:356] "Generic (PLEG): container finished" podID="3c7d6121-a834-42d0-9b78-583260fd4dce" containerID="fdfbaa83e586b36b56f5e94242cb2bd3eea52c642862fda15dae8196fda52047" exitCode=0 Sep 09 14:33:35.662606 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:35.662248 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p5bsf" event={"ID":"3c7d6121-a834-42d0-9b78-583260fd4dce","Type":"ContainerDied","Data":"fdfbaa83e586b36b56f5e94242cb2bd3eea52c642862fda15dae8196fda52047"} Sep 09 14:33:37.512998 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:37.512762 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5vdl5" Sep 09 14:33:37.513493 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:37.512817 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nqc8q" Sep 09 14:33:37.513493 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:37.513036 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5vdl5" podUID="ce063353-8e00-4701-8fb5-9b32aa0586e1" Sep 09 14:33:37.513493 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:37.512817 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wcj2p" Sep 09 14:33:37.513493 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:37.513166 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nqc8q" podUID="f286fc80-41f7-4b6f-8506-b5f0eb2629d3" Sep 09 14:33:37.513493 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:37.513243 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wcj2p" podUID="ab36c1e4-a65c-4cf9-ada4-f8a87d4b463b" Sep 09 14:33:39.227840 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:39.227762 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab36c1e4-a65c-4cf9-ada4-f8a87d4b463b-metrics-certs\") pod \"network-metrics-daemon-wcj2p\" (UID: \"ab36c1e4-a65c-4cf9-ada4-f8a87d4b463b\") " pod="openshift-multus/network-metrics-daemon-wcj2p" Sep 09 14:33:39.228401 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:39.227845 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f286fc80-41f7-4b6f-8506-b5f0eb2629d3-original-pull-secret\") pod \"global-pull-secret-syncer-nqc8q\" (UID: \"f286fc80-41f7-4b6f-8506-b5f0eb2629d3\") " pod="kube-system/global-pull-secret-syncer-nqc8q" Sep 09 14:33:39.228401 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:39.227926 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 14:33:39.228401 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:39.227952 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 09 14:33:39.228401 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:39.227996 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab36c1e4-a65c-4cf9-ada4-f8a87d4b463b-metrics-certs podName:ab36c1e4-a65c-4cf9-ada4-f8a87d4b463b nodeName:}" failed. No retries permitted until 2026-09-09 14:34:11.227977427 +0000 UTC m=+66.238893677 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ab36c1e4-a65c-4cf9-ada4-f8a87d4b463b-metrics-certs") pod "network-metrics-daemon-wcj2p" (UID: "ab36c1e4-a65c-4cf9-ada4-f8a87d4b463b") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 14:33:39.228401 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:39.228016 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f286fc80-41f7-4b6f-8506-b5f0eb2629d3-original-pull-secret podName:f286fc80-41f7-4b6f-8506-b5f0eb2629d3 nodeName:}" failed. No retries permitted until 2026-09-09 14:34:11.228007664 +0000 UTC m=+66.238924079 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f286fc80-41f7-4b6f-8506-b5f0eb2629d3-original-pull-secret") pod "global-pull-secret-syncer-nqc8q" (UID: "f286fc80-41f7-4b6f-8506-b5f0eb2629d3") : object "kube-system"/"original-pull-secret" not registered Sep 09 14:33:39.310801 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:39.310774 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-66.ec2.internal" event="NodeReady" Sep 09 14:33:39.310961 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:39.310927 2576 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 09 14:33:39.328497 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:39.328466 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2krn\" (UniqueName: \"kubernetes.io/projected/ce063353-8e00-4701-8fb5-9b32aa0586e1-kube-api-access-l2krn\") pod \"network-check-target-5vdl5\" (UID: \"ce063353-8e00-4701-8fb5-9b32aa0586e1\") " pod="openshift-network-diagnostics/network-check-target-5vdl5" Sep 09 14:33:39.328634 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:39.328617 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 09 14:33:39.328680 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:39.328646 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 09 14:33:39.328680 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:39.328656 2576 projected.go:196] Error preparing data for projected volume kube-api-access-l2krn for pod openshift-network-diagnostics/network-check-target-5vdl5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 14:33:39.328756 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:39.328701 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ce063353-8e00-4701-8fb5-9b32aa0586e1-kube-api-access-l2krn podName:ce063353-8e00-4701-8fb5-9b32aa0586e1 nodeName:}" failed. No retries permitted until 2026-09-09 14:34:11.32868743 +0000 UTC m=+66.339603658 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-l2krn" (UniqueName: "kubernetes.io/projected/ce063353-8e00-4701-8fb5-9b32aa0586e1-kube-api-access-l2krn") pod "network-check-target-5vdl5" (UID: "ce063353-8e00-4701-8fb5-9b32aa0586e1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 14:33:39.347237 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:39.347211 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-gpnd5"] Sep 09 14:33:39.398370 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:39.398345 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-86f77b57bb-8dbpd"] Sep 09 14:33:39.398507 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:39.398454 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-gpnd5" Sep 09 14:33:39.401756 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:39.401736 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Sep 09 14:33:39.402083 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:39.402064 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-2lx2l\"" Sep 09 14:33:39.402539 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:39.402512 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Sep 09 14:33:39.417942 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:39.417922 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-pkvlm"] Sep 09 14:33:39.418038 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:39.417925 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-86f77b57bb-8dbpd" Sep 09 14:33:39.422473 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:39.422452 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 09 14:33:39.422579 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:39.422520 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 09 14:33:39.422820 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:39.422803 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 09 14:33:39.422907 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:39.422867 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-m98bc\"" Sep 09 14:33:39.432241 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:39.432216 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 09 14:33:39.442093 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:39.442076 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-gpnd5"] Sep 09 14:33:39.442203 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:39.442140 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-4cj2m"] Sep 09 14:33:39.442263 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:39.442194 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pkvlm" Sep 09 14:33:39.444618 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:39.444602 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 09 14:33:39.444725 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:39.444677 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 09 14:33:39.444816 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:39.444792 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-btvxq\"" Sep 09 14:33:39.460497 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:39.460472 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-86f77b57bb-8dbpd"] Sep 09 14:33:39.460599 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:39.460514 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-pkvlm"] Sep 09 14:33:39.460599 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:39.460541 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-4cj2m"] Sep 09 14:33:39.460599 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:39.460574 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-4cj2m" Sep 09 14:33:39.463492 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:39.463219 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 09 14:33:39.463492 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:39.463292 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-dtd7b\"" Sep 09 14:33:39.463492 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:39.463328 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 09 14:33:39.463635 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:39.463610 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 09 14:33:39.509204 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:39.509133 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nqc8q" Sep 09 14:33:39.509204 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:39.509153 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wcj2p" Sep 09 14:33:39.509404 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:39.509140 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5vdl5" Sep 09 14:33:39.512905 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:39.512886 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-l56jk\"" Sep 09 14:33:39.513022 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:39.512989 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 09 14:33:39.513399 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:39.513217 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 09 14:33:39.513399 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:39.513243 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 09 14:33:39.513399 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:39.513284 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 09 14:33:39.513574 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:39.513409 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-bjh9b\"" Sep 09 14:33:39.529586 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:39.529564 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1fd6e104-62ef-4fb0-9bf7-66da53264398-image-registry-private-configuration\") pod \"image-registry-86f77b57bb-8dbpd\" (UID: \"1fd6e104-62ef-4fb0-9bf7-66da53264398\") " pod="openshift-image-registry/image-registry-86f77b57bb-8dbpd" Sep 09 14:33:39.529702 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:39.529597 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb6068ac-f5ef-42a9-9d6e-a3ab55b63949-tmp-dir\") pod \"dns-default-pkvlm\" (UID: \"bb6068ac-f5ef-42a9-9d6e-a3ab55b63949\") " pod="openshift-dns/dns-default-pkvlm" Sep 09 14:33:39.529702 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:39.529649 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2177b959-0d75-41f6-b309-06f66757e45b-nginx-conf\") pod \"networking-console-plugin-79977f9844-gpnd5\" (UID: \"2177b959-0d75-41f6-b309-06f66757e45b\") " pod="openshift-network-console/networking-console-plugin-79977f9844-gpnd5" Sep 09 14:33:39.529702 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:39.529690 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8zzf\" (UniqueName: \"kubernetes.io/projected/1fd6e104-62ef-4fb0-9bf7-66da53264398-kube-api-access-j8zzf\") pod \"image-registry-86f77b57bb-8dbpd\" (UID: \"1fd6e104-62ef-4fb0-9bf7-66da53264398\") " pod="openshift-image-registry/image-registry-86f77b57bb-8dbpd" Sep 09 14:33:39.529822 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:39.529719 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1fd6e104-62ef-4fb0-9bf7-66da53264398-bound-sa-token\") pod \"image-registry-86f77b57bb-8dbpd\" (UID: \"1fd6e104-62ef-4fb0-9bf7-66da53264398\") " pod="openshift-image-registry/image-registry-86f77b57bb-8dbpd" Sep 09 14:33:39.529822 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:39.529742 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lw5t\" (UniqueName: \"kubernetes.io/projected/bb6068ac-f5ef-42a9-9d6e-a3ab55b63949-kube-api-access-8lw5t\") pod \"dns-default-pkvlm\" (UID: \"bb6068ac-f5ef-42a9-9d6e-a3ab55b63949\") " pod="openshift-dns/dns-default-pkvlm" Sep 09 14:33:39.529822 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:39.529770 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bb6068ac-f5ef-42a9-9d6e-a3ab55b63949-config-volume\") pod \"dns-default-pkvlm\" (UID: \"bb6068ac-f5ef-42a9-9d6e-a3ab55b63949\") " pod="openshift-dns/dns-default-pkvlm" Sep 09 14:33:39.529925 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:39.529837 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1fd6e104-62ef-4fb0-9bf7-66da53264398-registry-tls\") pod \"image-registry-86f77b57bb-8dbpd\" (UID: \"1fd6e104-62ef-4fb0-9bf7-66da53264398\") " pod="openshift-image-registry/image-registry-86f77b57bb-8dbpd" Sep 09 14:33:39.529958 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:39.529923 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1fd6e104-62ef-4fb0-9bf7-66da53264398-installation-pull-secrets\") pod \"image-registry-86f77b57bb-8dbpd\" (UID: \"1fd6e104-62ef-4fb0-9bf7-66da53264398\") " pod="openshift-image-registry/image-registry-86f77b57bb-8dbpd" Sep 09 14:33:39.529996 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:39.529974 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bb6068ac-f5ef-42a9-9d6e-a3ab55b63949-metrics-tls\") pod \"dns-default-pkvlm\" (UID: \"bb6068ac-f5ef-42a9-9d6e-a3ab55b63949\") " pod="openshift-dns/dns-default-pkvlm" Sep 09 14:33:39.530056 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:39.529998 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2177b959-0d75-41f6-b309-06f66757e45b-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-gpnd5\" (UID: \"2177b959-0d75-41f6-b309-06f66757e45b\") " pod="openshift-network-console/networking-console-plugin-79977f9844-gpnd5" Sep 09 14:33:39.530101 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:39.530063 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1fd6e104-62ef-4fb0-9bf7-66da53264398-registry-certificates\") pod \"image-registry-86f77b57bb-8dbpd\" (UID: \"1fd6e104-62ef-4fb0-9bf7-66da53264398\") " pod="openshift-image-registry/image-registry-86f77b57bb-8dbpd" Sep 09 14:33:39.530137 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:39.530112 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1fd6e104-62ef-4fb0-9bf7-66da53264398-ca-trust-extracted\") pod \"image-registry-86f77b57bb-8dbpd\" (UID: \"1fd6e104-62ef-4fb0-9bf7-66da53264398\") " pod="openshift-image-registry/image-registry-86f77b57bb-8dbpd" Sep 09 14:33:39.530171 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:39.530135 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1fd6e104-62ef-4fb0-9bf7-66da53264398-trusted-ca\") pod \"image-registry-86f77b57bb-8dbpd\" (UID: \"1fd6e104-62ef-4fb0-9bf7-66da53264398\") " pod="openshift-image-registry/image-registry-86f77b57bb-8dbpd" Sep 09 14:33:39.580580 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:39.580551 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppvhx_141380ba-1588-47e3-99ea-1de06df135e9/konnectivity-agent/0.log" limit=1024 Sep 09 14:33:39.630879 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:39.630837 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1fd6e104-62ef-4fb0-9bf7-66da53264398-ca-trust-extracted\") pod \"image-registry-86f77b57bb-8dbpd\" (UID: \"1fd6e104-62ef-4fb0-9bf7-66da53264398\") " pod="openshift-image-registry/image-registry-86f77b57bb-8dbpd" Sep 09 14:33:39.630879 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:39.630869 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1fd6e104-62ef-4fb0-9bf7-66da53264398-trusted-ca\") pod \"image-registry-86f77b57bb-8dbpd\" (UID: \"1fd6e104-62ef-4fb0-9bf7-66da53264398\") " pod="openshift-image-registry/image-registry-86f77b57bb-8dbpd" Sep 09 14:33:39.631092 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:39.630941 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1fd6e104-62ef-4fb0-9bf7-66da53264398-image-registry-private-configuration\") pod \"image-registry-86f77b57bb-8dbpd\" (UID: \"1fd6e104-62ef-4fb0-9bf7-66da53264398\") " pod="openshift-image-registry/image-registry-86f77b57bb-8dbpd" Sep 09 14:33:39.631092 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:39.630962 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb6068ac-f5ef-42a9-9d6e-a3ab55b63949-tmp-dir\") pod \"dns-default-pkvlm\" (UID: \"bb6068ac-f5ef-42a9-9d6e-a3ab55b63949\") " pod="openshift-dns/dns-default-pkvlm" Sep 09 14:33:39.631092 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:39.630985 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2177b959-0d75-41f6-b309-06f66757e45b-nginx-conf\") pod \"networking-console-plugin-79977f9844-gpnd5\" (UID: \"2177b959-0d75-41f6-b309-06f66757e45b\") " pod="openshift-network-console/networking-console-plugin-79977f9844-gpnd5" Sep 09 14:33:39.631290 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:39.631269 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1fd6e104-62ef-4fb0-9bf7-66da53264398-ca-trust-extracted\") pod \"image-registry-86f77b57bb-8dbpd\" (UID: \"1fd6e104-62ef-4fb0-9bf7-66da53264398\") " pod="openshift-image-registry/image-registry-86f77b57bb-8dbpd" Sep 09 14:33:39.631413 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:39.631318 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb6068ac-f5ef-42a9-9d6e-a3ab55b63949-tmp-dir\") pod \"dns-default-pkvlm\" (UID: \"bb6068ac-f5ef-42a9-9d6e-a3ab55b63949\") " pod="openshift-dns/dns-default-pkvlm" Sep 09 14:33:39.631474 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:39.631437 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8zzf\" (UniqueName: \"kubernetes.io/projected/1fd6e104-62ef-4fb0-9bf7-66da53264398-kube-api-access-j8zzf\") pod \"image-registry-86f77b57bb-8dbpd\" (UID: \"1fd6e104-62ef-4fb0-9bf7-66da53264398\") " pod="openshift-image-registry/image-registry-86f77b57bb-8dbpd" Sep 09 14:33:39.631533 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:39.631477 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1fd6e104-62ef-4fb0-9bf7-66da53264398-bound-sa-token\") pod \"image-registry-86f77b57bb-8dbpd\" (UID: \"1fd6e104-62ef-4fb0-9bf7-66da53264398\") " pod="openshift-image-registry/image-registry-86f77b57bb-8dbpd" Sep 09 14:33:39.631533 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:39.631507 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lw5t\" (UniqueName: \"kubernetes.io/projected/bb6068ac-f5ef-42a9-9d6e-a3ab55b63949-kube-api-access-8lw5t\") pod \"dns-default-pkvlm\" (UID: \"bb6068ac-f5ef-42a9-9d6e-a3ab55b63949\") " pod="openshift-dns/dns-default-pkvlm" Sep 09 14:33:39.631635 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:39.631538 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bb6068ac-f5ef-42a9-9d6e-a3ab55b63949-config-volume\") pod \"dns-default-pkvlm\" (UID: \"bb6068ac-f5ef-42a9-9d6e-a3ab55b63949\") " pod="openshift-dns/dns-default-pkvlm" Sep 09 14:33:39.631635 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:39.631568 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1fd6e104-62ef-4fb0-9bf7-66da53264398-registry-tls\") pod \"image-registry-86f77b57bb-8dbpd\" (UID: \"1fd6e104-62ef-4fb0-9bf7-66da53264398\") " pod="openshift-image-registry/image-registry-86f77b57bb-8dbpd" Sep 09 14:33:39.631635 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:39.631605 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfdnw\" (UniqueName: \"kubernetes.io/projected/bb7ae7e5-fd88-4894-b4c8-4b8ea88776d4-kube-api-access-dfdnw\") pod \"ingress-canary-4cj2m\" (UID: \"bb7ae7e5-fd88-4894-b4c8-4b8ea88776d4\") " pod="openshift-ingress-canary/ingress-canary-4cj2m" Sep 09 14:33:39.631783 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:39.631653 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1fd6e104-62ef-4fb0-9bf7-66da53264398-installation-pull-secrets\") pod \"image-registry-86f77b57bb-8dbpd\" (UID: \"1fd6e104-62ef-4fb0-9bf7-66da53264398\") " pod="openshift-image-registry/image-registry-86f77b57bb-8dbpd" Sep 09 14:33:39.631783 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:39.631657 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2177b959-0d75-41f6-b309-06f66757e45b-nginx-conf\") pod \"networking-console-plugin-79977f9844-gpnd5\" (UID: \"2177b959-0d75-41f6-b309-06f66757e45b\") " pod="openshift-network-console/networking-console-plugin-79977f9844-gpnd5" Sep 09 14:33:39.631783 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:39.631675 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bb6068ac-f5ef-42a9-9d6e-a3ab55b63949-metrics-tls\") pod \"dns-default-pkvlm\" (UID: \"bb6068ac-f5ef-42a9-9d6e-a3ab55b63949\") " pod="openshift-dns/dns-default-pkvlm" Sep 09 14:33:39.631783 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:39.631699 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2177b959-0d75-41f6-b309-06f66757e45b-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-gpnd5\" (UID: \"2177b959-0d75-41f6-b309-06f66757e45b\") " pod="openshift-network-console/networking-console-plugin-79977f9844-gpnd5" Sep 09 14:33:39.631783 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:39.631725 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1fd6e104-62ef-4fb0-9bf7-66da53264398-registry-certificates\") pod \"image-registry-86f77b57bb-8dbpd\" (UID: \"1fd6e104-62ef-4fb0-9bf7-66da53264398\") " pod="openshift-image-registry/image-registry-86f77b57bb-8dbpd" Sep 09 14:33:39.631783 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:39.631746 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bb7ae7e5-fd88-4894-b4c8-4b8ea88776d4-cert\") pod \"ingress-canary-4cj2m\" (UID: \"bb7ae7e5-fd88-4894-b4c8-4b8ea88776d4\") " pod="openshift-ingress-canary/ingress-canary-4cj2m" Sep 09 14:33:39.632079 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:39.631871 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 09 14:33:39.632079 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:39.631883 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-86f77b57bb-8dbpd: secret "image-registry-tls" not found Sep 09 14:33:39.632079 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:39.631927 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1fd6e104-62ef-4fb0-9bf7-66da53264398-registry-tls podName:1fd6e104-62ef-4fb0-9bf7-66da53264398 nodeName:}" failed. No retries permitted until 2026-09-09 14:33:40.131912697 +0000 UTC m=+35.142828926 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1fd6e104-62ef-4fb0-9bf7-66da53264398-registry-tls") pod "image-registry-86f77b57bb-8dbpd" (UID: "1fd6e104-62ef-4fb0-9bf7-66da53264398") : secret "image-registry-tls" not found Sep 09 14:33:39.632079 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:39.631997 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bb6068ac-f5ef-42a9-9d6e-a3ab55b63949-config-volume\") pod \"dns-default-pkvlm\" (UID: \"bb6068ac-f5ef-42a9-9d6e-a3ab55b63949\") " pod="openshift-dns/dns-default-pkvlm" Sep 09 14:33:39.632079 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:39.632007 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 09 14:33:39.632079 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:39.632046 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb6068ac-f5ef-42a9-9d6e-a3ab55b63949-metrics-tls podName:bb6068ac-f5ef-42a9-9d6e-a3ab55b63949 nodeName:}" failed. No retries permitted until 2026-09-09 14:33:40.132034156 +0000 UTC m=+35.142950385 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bb6068ac-f5ef-42a9-9d6e-a3ab55b63949-metrics-tls") pod "dns-default-pkvlm" (UID: "bb6068ac-f5ef-42a9-9d6e-a3ab55b63949") : secret "dns-default-metrics-tls" not found Sep 09 14:33:39.632079 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:39.632052 2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 09 14:33:39.632079 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:39.632083 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2177b959-0d75-41f6-b309-06f66757e45b-networking-console-plugin-cert podName:2177b959-0d75-41f6-b309-06f66757e45b nodeName:}" failed. No retries permitted until 2026-09-09 14:33:40.132072751 +0000 UTC m=+35.142988982 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/2177b959-0d75-41f6-b309-06f66757e45b-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-gpnd5" (UID: "2177b959-0d75-41f6-b309-06f66757e45b") : secret "networking-console-plugin-cert" not found Sep 09 14:33:39.632413 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:39.632172 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1fd6e104-62ef-4fb0-9bf7-66da53264398-trusted-ca\") pod \"image-registry-86f77b57bb-8dbpd\" (UID: \"1fd6e104-62ef-4fb0-9bf7-66da53264398\") " pod="openshift-image-registry/image-registry-86f77b57bb-8dbpd" Sep 09 14:33:39.632548 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:39.632530 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1fd6e104-62ef-4fb0-9bf7-66da53264398-registry-certificates\") pod \"image-registry-86f77b57bb-8dbpd\" (UID: \"1fd6e104-62ef-4fb0-9bf7-66da53264398\") " pod="openshift-image-registry/image-registry-86f77b57bb-8dbpd" Sep 09 14:33:39.635639 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:39.635620 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1fd6e104-62ef-4fb0-9bf7-66da53264398-installation-pull-secrets\") pod \"image-registry-86f77b57bb-8dbpd\" (UID: \"1fd6e104-62ef-4fb0-9bf7-66da53264398\") " pod="openshift-image-registry/image-registry-86f77b57bb-8dbpd" Sep 09 14:33:39.635731 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:39.635709 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1fd6e104-62ef-4fb0-9bf7-66da53264398-image-registry-private-configuration\") pod \"image-registry-86f77b57bb-8dbpd\" (UID: \"1fd6e104-62ef-4fb0-9bf7-66da53264398\") " pod="openshift-image-registry/image-registry-86f77b57bb-8dbpd" Sep 09 14:33:39.642403 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:39.642384 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lw5t\" (UniqueName: \"kubernetes.io/projected/bb6068ac-f5ef-42a9-9d6e-a3ab55b63949-kube-api-access-8lw5t\") pod \"dns-default-pkvlm\" (UID: \"bb6068ac-f5ef-42a9-9d6e-a3ab55b63949\") " pod="openshift-dns/dns-default-pkvlm" Sep 09 14:33:39.642708 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:39.642687 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8zzf\" (UniqueName: \"kubernetes.io/projected/1fd6e104-62ef-4fb0-9bf7-66da53264398-kube-api-access-j8zzf\") pod \"image-registry-86f77b57bb-8dbpd\" (UID: \"1fd6e104-62ef-4fb0-9bf7-66da53264398\") " pod="openshift-image-registry/image-registry-86f77b57bb-8dbpd" Sep 09 14:33:39.642906 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:39.642889 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1fd6e104-62ef-4fb0-9bf7-66da53264398-bound-sa-token\") pod \"image-registry-86f77b57bb-8dbpd\" (UID: \"1fd6e104-62ef-4fb0-9bf7-66da53264398\") " pod="openshift-image-registry/image-registry-86f77b57bb-8dbpd" Sep 09 14:33:39.680628 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:39.680594 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppvhx_141380ba-1588-47e3-99ea-1de06df135e9/konnectivity-agent/0.log" limit=1024 Sep 09 14:33:39.732379 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:39.732340 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfdnw\" (UniqueName: \"kubernetes.io/projected/bb7ae7e5-fd88-4894-b4c8-4b8ea88776d4-kube-api-access-dfdnw\") pod \"ingress-canary-4cj2m\" (UID: \"bb7ae7e5-fd88-4894-b4c8-4b8ea88776d4\") " pod="openshift-ingress-canary/ingress-canary-4cj2m" Sep 09 14:33:39.732543 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:39.732418 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bb7ae7e5-fd88-4894-b4c8-4b8ea88776d4-cert\") pod \"ingress-canary-4cj2m\" (UID: \"bb7ae7e5-fd88-4894-b4c8-4b8ea88776d4\") " pod="openshift-ingress-canary/ingress-canary-4cj2m" Sep 09 14:33:39.735039 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:39.735016 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bb7ae7e5-fd88-4894-b4c8-4b8ea88776d4-cert\") pod \"ingress-canary-4cj2m\" (UID: \"bb7ae7e5-fd88-4894-b4c8-4b8ea88776d4\") " pod="openshift-ingress-canary/ingress-canary-4cj2m" Sep 09 14:33:39.742837 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:39.742812 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfdnw\" (UniqueName: \"kubernetes.io/projected/bb7ae7e5-fd88-4894-b4c8-4b8ea88776d4-kube-api-access-dfdnw\") pod \"ingress-canary-4cj2m\" (UID: \"bb7ae7e5-fd88-4894-b4c8-4b8ea88776d4\") " pod="openshift-ingress-canary/ingress-canary-4cj2m" Sep 09 14:33:39.769800 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:39.769715 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-4cj2m" Sep 09 14:33:40.097895 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:40.097801 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-4cj2m"] Sep 09 14:33:40.136753 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:40.136718 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1fd6e104-62ef-4fb0-9bf7-66da53264398-registry-tls\") pod \"image-registry-86f77b57bb-8dbpd\" (UID: \"1fd6e104-62ef-4fb0-9bf7-66da53264398\") " pod="openshift-image-registry/image-registry-86f77b57bb-8dbpd" Sep 09 14:33:40.136937 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:40.136767 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bb6068ac-f5ef-42a9-9d6e-a3ab55b63949-metrics-tls\") pod \"dns-default-pkvlm\" (UID: \"bb6068ac-f5ef-42a9-9d6e-a3ab55b63949\") " pod="openshift-dns/dns-default-pkvlm" Sep 09 14:33:40.136937 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:40.136786 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2177b959-0d75-41f6-b309-06f66757e45b-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-gpnd5\" (UID: \"2177b959-0d75-41f6-b309-06f66757e45b\") " pod="openshift-network-console/networking-console-plugin-79977f9844-gpnd5" Sep 09 14:33:40.136937 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:40.136889 2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 09 14:33:40.136937 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:40.136891 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 09 14:33:40.136937 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:40.136898 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 09 14:33:40.136937 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:40.136928 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-86f77b57bb-8dbpd: secret "image-registry-tls" not found Sep 09 14:33:40.137165 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:40.136958 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2177b959-0d75-41f6-b309-06f66757e45b-networking-console-plugin-cert podName:2177b959-0d75-41f6-b309-06f66757e45b nodeName:}" failed. No retries permitted until 2026-09-09 14:33:41.136940151 +0000 UTC m=+36.147856380 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/2177b959-0d75-41f6-b309-06f66757e45b-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-gpnd5" (UID: "2177b959-0d75-41f6-b309-06f66757e45b") : secret "networking-console-plugin-cert" not found Sep 09 14:33:40.137165 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:40.136983 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1fd6e104-62ef-4fb0-9bf7-66da53264398-registry-tls podName:1fd6e104-62ef-4fb0-9bf7-66da53264398 nodeName:}" failed. No retries permitted until 2026-09-09 14:33:41.136964247 +0000 UTC m=+36.147880487 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1fd6e104-62ef-4fb0-9bf7-66da53264398-registry-tls") pod "image-registry-86f77b57bb-8dbpd" (UID: "1fd6e104-62ef-4fb0-9bf7-66da53264398") : secret "image-registry-tls" not found Sep 09 14:33:40.137165 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:40.137007 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb6068ac-f5ef-42a9-9d6e-a3ab55b63949-metrics-tls podName:bb6068ac-f5ef-42a9-9d6e-a3ab55b63949 nodeName:}" failed. No retries permitted until 2026-09-09 14:33:41.136996905 +0000 UTC m=+36.147913137 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bb6068ac-f5ef-42a9-9d6e-a3ab55b63949-metrics-tls") pod "dns-default-pkvlm" (UID: "bb6068ac-f5ef-42a9-9d6e-a3ab55b63949") : secret "dns-default-metrics-tls" not found Sep 09 14:33:41.145767 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:41.145722 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1fd6e104-62ef-4fb0-9bf7-66da53264398-registry-tls\") pod \"image-registry-86f77b57bb-8dbpd\" (UID: \"1fd6e104-62ef-4fb0-9bf7-66da53264398\") " pod="openshift-image-registry/image-registry-86f77b57bb-8dbpd" Sep 09 14:33:41.146506 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:41.145786 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bb6068ac-f5ef-42a9-9d6e-a3ab55b63949-metrics-tls\") pod \"dns-default-pkvlm\" (UID: \"bb6068ac-f5ef-42a9-9d6e-a3ab55b63949\") " pod="openshift-dns/dns-default-pkvlm" Sep 09 14:33:41.146506 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:41.145813 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2177b959-0d75-41f6-b309-06f66757e45b-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-gpnd5\" (UID: \"2177b959-0d75-41f6-b309-06f66757e45b\") " pod="openshift-network-console/networking-console-plugin-79977f9844-gpnd5" Sep 09 14:33:41.146506 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:41.145889 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 09 14:33:41.146506 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:41.145908 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-86f77b57bb-8dbpd: secret "image-registry-tls" not found Sep 09 14:33:41.146506 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:41.145944 2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 09 14:33:41.146506 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:41.145966 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1fd6e104-62ef-4fb0-9bf7-66da53264398-registry-tls podName:1fd6e104-62ef-4fb0-9bf7-66da53264398 nodeName:}" failed. No retries permitted until 2026-09-09 14:33:43.145946508 +0000 UTC m=+38.156862756 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1fd6e104-62ef-4fb0-9bf7-66da53264398-registry-tls") pod "image-registry-86f77b57bb-8dbpd" (UID: "1fd6e104-62ef-4fb0-9bf7-66da53264398") : secret "image-registry-tls" not found Sep 09 14:33:41.146506 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:41.145984 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2177b959-0d75-41f6-b309-06f66757e45b-networking-console-plugin-cert podName:2177b959-0d75-41f6-b309-06f66757e45b nodeName:}" failed. No retries permitted until 2026-09-09 14:33:43.145975192 +0000 UTC m=+38.156891428 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/2177b959-0d75-41f6-b309-06f66757e45b-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-gpnd5" (UID: "2177b959-0d75-41f6-b309-06f66757e45b") : secret "networking-console-plugin-cert" not found Sep 09 14:33:41.146506 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:41.146013 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 09 14:33:41.146506 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:41.146039 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb6068ac-f5ef-42a9-9d6e-a3ab55b63949-metrics-tls podName:bb6068ac-f5ef-42a9-9d6e-a3ab55b63949 nodeName:}" failed. No retries permitted until 2026-09-09 14:33:43.146030085 +0000 UTC m=+38.156946321 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bb6068ac-f5ef-42a9-9d6e-a3ab55b63949-metrics-tls") pod "dns-default-pkvlm" (UID: "bb6068ac-f5ef-42a9-9d6e-a3ab55b63949") : secret "dns-default-metrics-tls" not found Sep 09 14:33:41.420273 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:33:41.420210 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb7ae7e5_fd88_4894_b4c8_4b8ea88776d4.slice/crio-dd37068098edbce2761a54daba8402e72c723fb0dd9f0a186c51caccb0940109 WatchSource:0}: Error finding container dd37068098edbce2761a54daba8402e72c723fb0dd9f0a186c51caccb0940109: Status 404 returned error can't find the container with id dd37068098edbce2761a54daba8402e72c723fb0dd9f0a186c51caccb0940109 Sep 09 14:33:41.678443 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:41.678375 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-4cj2m" event={"ID":"bb7ae7e5-fd88-4894-b4c8-4b8ea88776d4","Type":"ContainerStarted","Data":"dd37068098edbce2761a54daba8402e72c723fb0dd9f0a186c51caccb0940109"} Sep 09 14:33:42.685947 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:42.684941 2576 generic.go:356] "Generic (PLEG): container finished" podID="3c7d6121-a834-42d0-9b78-583260fd4dce" containerID="76ef55c199c405f76b8590e7409a6efa53b792ebd94c5ef67c6c0c02bc02920f" exitCode=0 Sep 09 14:33:42.685947 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:42.685035 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p5bsf" event={"ID":"3c7d6121-a834-42d0-9b78-583260fd4dce","Type":"ContainerDied","Data":"76ef55c199c405f76b8590e7409a6efa53b792ebd94c5ef67c6c0c02bc02920f"} Sep 09 14:33:43.163013 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:43.162965 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1fd6e104-62ef-4fb0-9bf7-66da53264398-registry-tls\") pod \"image-registry-86f77b57bb-8dbpd\" (UID: \"1fd6e104-62ef-4fb0-9bf7-66da53264398\") " pod="openshift-image-registry/image-registry-86f77b57bb-8dbpd" Sep 09 14:33:43.163205 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:43.163044 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bb6068ac-f5ef-42a9-9d6e-a3ab55b63949-metrics-tls\") pod \"dns-default-pkvlm\" (UID: \"bb6068ac-f5ef-42a9-9d6e-a3ab55b63949\") " pod="openshift-dns/dns-default-pkvlm" Sep 09 14:33:43.163205 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:43.163074 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2177b959-0d75-41f6-b309-06f66757e45b-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-gpnd5\" (UID: \"2177b959-0d75-41f6-b309-06f66757e45b\") " pod="openshift-network-console/networking-console-plugin-79977f9844-gpnd5" Sep 09 14:33:43.163205 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:43.163147 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 09 14:33:43.163205 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:43.163170 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-86f77b57bb-8dbpd: secret "image-registry-tls" not found Sep 09 14:33:43.163205 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:43.163196 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 09 14:33:43.163446 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:43.163216 2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 09 14:33:43.163446 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:43.163238 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1fd6e104-62ef-4fb0-9bf7-66da53264398-registry-tls podName:1fd6e104-62ef-4fb0-9bf7-66da53264398 nodeName:}" failed. No retries permitted until 2026-09-09 14:33:47.163217866 +0000 UTC m=+42.174134103 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1fd6e104-62ef-4fb0-9bf7-66da53264398-registry-tls") pod "image-registry-86f77b57bb-8dbpd" (UID: "1fd6e104-62ef-4fb0-9bf7-66da53264398") : secret "image-registry-tls" not found Sep 09 14:33:43.163446 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:43.163262 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2177b959-0d75-41f6-b309-06f66757e45b-networking-console-plugin-cert podName:2177b959-0d75-41f6-b309-06f66757e45b nodeName:}" failed. No retries permitted until 2026-09-09 14:33:47.163246048 +0000 UTC m=+42.174162284 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/2177b959-0d75-41f6-b309-06f66757e45b-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-gpnd5" (UID: "2177b959-0d75-41f6-b309-06f66757e45b") : secret "networking-console-plugin-cert" not found Sep 09 14:33:43.163446 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:43.163278 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb6068ac-f5ef-42a9-9d6e-a3ab55b63949-metrics-tls podName:bb6068ac-f5ef-42a9-9d6e-a3ab55b63949 nodeName:}" failed. No retries permitted until 2026-09-09 14:33:47.163269252 +0000 UTC m=+42.174185488 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bb6068ac-f5ef-42a9-9d6e-a3ab55b63949-metrics-tls") pod "dns-default-pkvlm" (UID: "bb6068ac-f5ef-42a9-9d6e-a3ab55b63949") : secret "dns-default-metrics-tls" not found Sep 09 14:33:43.691601 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:43.691569 2576 generic.go:356] "Generic (PLEG): container finished" podID="3c7d6121-a834-42d0-9b78-583260fd4dce" containerID="640f94020d68bb30d484ee5ef996d245f5e3316dcf51d11bf7ed38b667450f41" exitCode=0 Sep 09 14:33:43.692104 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:43.691631 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p5bsf" event={"ID":"3c7d6121-a834-42d0-9b78-583260fd4dce","Type":"ContainerDied","Data":"640f94020d68bb30d484ee5ef996d245f5e3316dcf51d11bf7ed38b667450f41"} Sep 09 14:33:44.696010 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:44.695964 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-4cj2m" event={"ID":"bb7ae7e5-fd88-4894-b4c8-4b8ea88776d4","Type":"ContainerStarted","Data":"a99382bafe81404a7c8619a60dd0dd4551699efbb12b1a3deb9bec92a3e33b20"} Sep 09 14:33:44.699662 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:44.699627 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p5bsf" event={"ID":"3c7d6121-a834-42d0-9b78-583260fd4dce","Type":"ContainerStarted","Data":"c736037f56fec413349c6fb7533a8c3a007fc9d2ec191fe120a6ea1e30edfd84"} Sep 09 14:33:44.713657 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:44.713607 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-4cj2m" podStartSLOduration=2.627274094 podStartE2EDuration="5.713589665s" totalImagesPullingTime="3.086315571s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 14:33:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:33:44.711963737 +0000 UTC m=+39.722880006" watchObservedRunningTime="2026-09-09 14:33:44.713589665 +0000 UTC m=+39.724505919" Sep 09 14:33:44.734339 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:44.734264 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-p5bsf" podStartSLOduration=8.355859124 podStartE2EDuration="39.734241676s" totalImagesPullingTime="31.378382552s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 14:33:05 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:33:44.733096708 +0000 UTC m=+39.744013005" watchObservedRunningTime="2026-09-09 14:33:44.734241676 +0000 UTC m=+39.745157928" Sep 09 14:33:47.194573 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:47.194535 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1fd6e104-62ef-4fb0-9bf7-66da53264398-registry-tls\") pod \"image-registry-86f77b57bb-8dbpd\" (UID: \"1fd6e104-62ef-4fb0-9bf7-66da53264398\") " pod="openshift-image-registry/image-registry-86f77b57bb-8dbpd" Sep 09 14:33:47.195013 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:47.194594 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bb6068ac-f5ef-42a9-9d6e-a3ab55b63949-metrics-tls\") pod \"dns-default-pkvlm\" (UID: \"bb6068ac-f5ef-42a9-9d6e-a3ab55b63949\") " pod="openshift-dns/dns-default-pkvlm" Sep 09 14:33:47.195013 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:47.194617 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2177b959-0d75-41f6-b309-06f66757e45b-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-gpnd5\" (UID: \"2177b959-0d75-41f6-b309-06f66757e45b\") " pod="openshift-network-console/networking-console-plugin-79977f9844-gpnd5" Sep 09 14:33:47.195013 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:47.194688 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 09 14:33:47.195013 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:47.194707 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-86f77b57bb-8dbpd: secret "image-registry-tls" not found Sep 09 14:33:47.195013 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:47.194732 2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 09 14:33:47.195013 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:47.194739 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 09 14:33:47.195013 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:47.194760 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1fd6e104-62ef-4fb0-9bf7-66da53264398-registry-tls podName:1fd6e104-62ef-4fb0-9bf7-66da53264398 nodeName:}" failed. No retries permitted until 2026-09-09 14:33:55.194744354 +0000 UTC m=+50.205660583 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1fd6e104-62ef-4fb0-9bf7-66da53264398-registry-tls") pod "image-registry-86f77b57bb-8dbpd" (UID: "1fd6e104-62ef-4fb0-9bf7-66da53264398") : secret "image-registry-tls" not found Sep 09 14:33:47.195013 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:47.194785 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2177b959-0d75-41f6-b309-06f66757e45b-networking-console-plugin-cert podName:2177b959-0d75-41f6-b309-06f66757e45b nodeName:}" failed. No retries permitted until 2026-09-09 14:33:55.194769751 +0000 UTC m=+50.205686001 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/2177b959-0d75-41f6-b309-06f66757e45b-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-gpnd5" (UID: "2177b959-0d75-41f6-b309-06f66757e45b") : secret "networking-console-plugin-cert" not found Sep 09 14:33:47.195013 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:47.194803 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb6068ac-f5ef-42a9-9d6e-a3ab55b63949-metrics-tls podName:bb6068ac-f5ef-42a9-9d6e-a3ab55b63949 nodeName:}" failed. No retries permitted until 2026-09-09 14:33:55.19479342 +0000 UTC m=+50.205709656 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bb6068ac-f5ef-42a9-9d6e-a3ab55b63949-metrics-tls") pod "dns-default-pkvlm" (UID: "bb6068ac-f5ef-42a9-9d6e-a3ab55b63949") : secret "dns-default-metrics-tls" not found Sep 09 14:33:55.257043 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:55.256995 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1fd6e104-62ef-4fb0-9bf7-66da53264398-registry-tls\") pod \"image-registry-86f77b57bb-8dbpd\" (UID: \"1fd6e104-62ef-4fb0-9bf7-66da53264398\") " pod="openshift-image-registry/image-registry-86f77b57bb-8dbpd" Sep 09 14:33:55.257544 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:55.257052 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bb6068ac-f5ef-42a9-9d6e-a3ab55b63949-metrics-tls\") pod \"dns-default-pkvlm\" (UID: \"bb6068ac-f5ef-42a9-9d6e-a3ab55b63949\") " pod="openshift-dns/dns-default-pkvlm" Sep 09 14:33:55.257544 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:33:55.257080 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2177b959-0d75-41f6-b309-06f66757e45b-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-gpnd5\" (UID: \"2177b959-0d75-41f6-b309-06f66757e45b\") " pod="openshift-network-console/networking-console-plugin-79977f9844-gpnd5" Sep 09 14:33:55.257544 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:55.257141 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 09 14:33:55.257544 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:55.257160 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-86f77b57bb-8dbpd: secret "image-registry-tls" not found Sep 09 14:33:55.257544 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:55.257211 2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 09 14:33:55.257544 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:55.257212 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 09 14:33:55.257544 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:55.257221 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1fd6e104-62ef-4fb0-9bf7-66da53264398-registry-tls podName:1fd6e104-62ef-4fb0-9bf7-66da53264398 nodeName:}" failed. No retries permitted until 2026-09-09 14:34:11.257204386 +0000 UTC m=+66.268120615 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1fd6e104-62ef-4fb0-9bf7-66da53264398-registry-tls") pod "image-registry-86f77b57bb-8dbpd" (UID: "1fd6e104-62ef-4fb0-9bf7-66da53264398") : secret "image-registry-tls" not found Sep 09 14:33:55.257544 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:55.257275 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb6068ac-f5ef-42a9-9d6e-a3ab55b63949-metrics-tls podName:bb6068ac-f5ef-42a9-9d6e-a3ab55b63949 nodeName:}" failed. No retries permitted until 2026-09-09 14:34:11.257258401 +0000 UTC m=+66.268174630 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bb6068ac-f5ef-42a9-9d6e-a3ab55b63949-metrics-tls") pod "dns-default-pkvlm" (UID: "bb6068ac-f5ef-42a9-9d6e-a3ab55b63949") : secret "dns-default-metrics-tls" not found Sep 09 14:33:55.257544 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:33:55.257315 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2177b959-0d75-41f6-b309-06f66757e45b-networking-console-plugin-cert podName:2177b959-0d75-41f6-b309-06f66757e45b nodeName:}" failed. No retries permitted until 2026-09-09 14:34:11.257289324 +0000 UTC m=+66.268205555 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/2177b959-0d75-41f6-b309-06f66757e45b-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-gpnd5" (UID: "2177b959-0d75-41f6-b309-06f66757e45b") : secret "networking-console-plugin-cert" not found Sep 09 14:34:04.668609 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:34:04.668581 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7xcn5" Sep 09 14:34:11.271616 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:34:11.271561 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1fd6e104-62ef-4fb0-9bf7-66da53264398-registry-tls\") pod \"image-registry-86f77b57bb-8dbpd\" (UID: \"1fd6e104-62ef-4fb0-9bf7-66da53264398\") " pod="openshift-image-registry/image-registry-86f77b57bb-8dbpd" Sep 09 14:34:11.271616 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:34:11.271620 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f286fc80-41f7-4b6f-8506-b5f0eb2629d3-original-pull-secret\") pod \"global-pull-secret-syncer-nqc8q\" (UID: \"f286fc80-41f7-4b6f-8506-b5f0eb2629d3\") " pod="kube-system/global-pull-secret-syncer-nqc8q" Sep 09 14:34:11.272172 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:34:11.271724 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 09 14:34:11.272172 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:34:11.271737 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bb6068ac-f5ef-42a9-9d6e-a3ab55b63949-metrics-tls\") pod \"dns-default-pkvlm\" (UID: \"bb6068ac-f5ef-42a9-9d6e-a3ab55b63949\") " pod="openshift-dns/dns-default-pkvlm" Sep 09 14:34:11.272172 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:34:11.271764 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2177b959-0d75-41f6-b309-06f66757e45b-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-gpnd5\" (UID: \"2177b959-0d75-41f6-b309-06f66757e45b\") " pod="openshift-network-console/networking-console-plugin-79977f9844-gpnd5" Sep 09 14:34:11.272172 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:34:11.271744 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-86f77b57bb-8dbpd: secret "image-registry-tls" not found Sep 09 14:34:11.272172 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:34:11.271814 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab36c1e4-a65c-4cf9-ada4-f8a87d4b463b-metrics-certs\") pod \"network-metrics-daemon-wcj2p\" (UID: \"ab36c1e4-a65c-4cf9-ada4-f8a87d4b463b\") " pod="openshift-multus/network-metrics-daemon-wcj2p" Sep 09 14:34:11.272172 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:34:11.271846 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 09 14:34:11.272172 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:34:11.271873 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1fd6e104-62ef-4fb0-9bf7-66da53264398-registry-tls podName:1fd6e104-62ef-4fb0-9bf7-66da53264398 nodeName:}" failed. No retries permitted until 2026-09-09 14:34:43.271849509 +0000 UTC m=+98.282765738 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1fd6e104-62ef-4fb0-9bf7-66da53264398-registry-tls") pod "image-registry-86f77b57bb-8dbpd" (UID: "1fd6e104-62ef-4fb0-9bf7-66da53264398") : secret "image-registry-tls" not found Sep 09 14:34:11.272172 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:34:11.271875 2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 09 14:34:11.272172 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:34:11.271907 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb6068ac-f5ef-42a9-9d6e-a3ab55b63949-metrics-tls podName:bb6068ac-f5ef-42a9-9d6e-a3ab55b63949 nodeName:}" failed. No retries permitted until 2026-09-09 14:34:43.271890531 +0000 UTC m=+98.282806763 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bb6068ac-f5ef-42a9-9d6e-a3ab55b63949-metrics-tls") pod "dns-default-pkvlm" (UID: "bb6068ac-f5ef-42a9-9d6e-a3ab55b63949") : secret "dns-default-metrics-tls" not found Sep 09 14:34:11.272172 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:34:11.271935 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2177b959-0d75-41f6-b309-06f66757e45b-networking-console-plugin-cert podName:2177b959-0d75-41f6-b309-06f66757e45b nodeName:}" failed. No retries permitted until 2026-09-09 14:34:43.271920654 +0000 UTC m=+98.282836887 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/2177b959-0d75-41f6-b309-06f66757e45b-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-gpnd5" (UID: "2177b959-0d75-41f6-b309-06f66757e45b") : secret "networking-console-plugin-cert" not found Sep 09 14:34:11.275206 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:34:11.275190 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 09 14:34:11.275450 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:34:11.275436 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 09 14:34:11.282767 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:34:11.282750 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 09 14:34:11.282848 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:34:11.282797 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab36c1e4-a65c-4cf9-ada4-f8a87d4b463b-metrics-certs podName:ab36c1e4-a65c-4cf9-ada4-f8a87d4b463b nodeName:}" failed. No retries permitted until 2026-09-09 14:35:15.282783321 +0000 UTC m=+130.293699550 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ab36c1e4-a65c-4cf9-ada4-f8a87d4b463b-metrics-certs") pod "network-metrics-daemon-wcj2p" (UID: "ab36c1e4-a65c-4cf9-ada4-f8a87d4b463b") : secret "metrics-daemon-secret" not found Sep 09 14:34:11.285872 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:34:11.285838 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f286fc80-41f7-4b6f-8506-b5f0eb2629d3-original-pull-secret\") pod \"global-pull-secret-syncer-nqc8q\" (UID: \"f286fc80-41f7-4b6f-8506-b5f0eb2629d3\") " pod="kube-system/global-pull-secret-syncer-nqc8q" Sep 09 14:34:11.320417 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:34:11.320376 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nqc8q" Sep 09 14:34:11.372959 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:34:11.372925 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2krn\" (UniqueName: \"kubernetes.io/projected/ce063353-8e00-4701-8fb5-9b32aa0586e1-kube-api-access-l2krn\") pod \"network-check-target-5vdl5\" (UID: \"ce063353-8e00-4701-8fb5-9b32aa0586e1\") " pod="openshift-network-diagnostics/network-check-target-5vdl5" Sep 09 14:34:11.383968 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:34:11.383944 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 09 14:34:11.398294 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:34:11.398156 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 09 14:34:11.406373 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:34:11.406344 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2krn\" (UniqueName: \"kubernetes.io/projected/ce063353-8e00-4701-8fb5-9b32aa0586e1-kube-api-access-l2krn\") pod \"network-check-target-5vdl5\" (UID: \"ce063353-8e00-4701-8fb5-9b32aa0586e1\") " pod="openshift-network-diagnostics/network-check-target-5vdl5" Sep 09 14:34:11.590076 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:34:11.589975 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-nqc8q"] Sep 09 14:34:11.593997 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:34:11.593959 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf286fc80_41f7_4b6f_8506_b5f0eb2629d3.slice/crio-08543b9120f187c3ce6e262ac74b15332bf3c3ba180b8504d35feffe5b6a16a2 WatchSource:0}: Error finding container 08543b9120f187c3ce6e262ac74b15332bf3c3ba180b8504d35feffe5b6a16a2: Status 404 returned error can't find the container with id 08543b9120f187c3ce6e262ac74b15332bf3c3ba180b8504d35feffe5b6a16a2 Sep 09 14:34:11.628551 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:34:11.628517 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-l56jk\"" Sep 09 14:34:11.636889 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:34:11.636866 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5vdl5" Sep 09 14:34:11.762690 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:34:11.762656 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-nqc8q" event={"ID":"f286fc80-41f7-4b6f-8506-b5f0eb2629d3","Type":"ContainerStarted","Data":"08543b9120f187c3ce6e262ac74b15332bf3c3ba180b8504d35feffe5b6a16a2"} Sep 09 14:34:11.902939 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:34:11.902861 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-5vdl5"] Sep 09 14:34:11.905885 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:34:11.905858 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce063353_8e00_4701_8fb5_9b32aa0586e1.slice/crio-0d3daff708ba92481cf283d7ae3be579d6642776ae41f7f774cde6eff3107b20 WatchSource:0}: Error finding container 0d3daff708ba92481cf283d7ae3be579d6642776ae41f7f774cde6eff3107b20: Status 404 returned error can't find the container with id 0d3daff708ba92481cf283d7ae3be579d6642776ae41f7f774cde6eff3107b20 Sep 09 14:34:12.766457 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:34:12.766415 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-5vdl5" event={"ID":"ce063353-8e00-4701-8fb5-9b32aa0586e1","Type":"ContainerStarted","Data":"0d3daff708ba92481cf283d7ae3be579d6642776ae41f7f774cde6eff3107b20"} Sep 09 14:34:17.781896 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:34:17.781855 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-nqc8q" event={"ID":"f286fc80-41f7-4b6f-8506-b5f0eb2629d3","Type":"ContainerStarted","Data":"ae07f2e121053503dffe1a8316637a524c5c8fa63c3dcd38a1569e8e9d10aafd"} Sep 09 14:34:17.783360 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:34:17.783338 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-5vdl5" event={"ID":"ce063353-8e00-4701-8fb5-9b32aa0586e1","Type":"ContainerStarted","Data":"5cd27a069d14a9b439cf3885969ef24d0111d45d845a5d7551043a2fb1a6c257"} Sep 09 14:34:17.783476 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:34:17.783461 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-5vdl5" Sep 09 14:34:17.798147 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:34:17.798082 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-nqc8q" podStartSLOduration=67.252860905 podStartE2EDuration="1m12.798065127s" totalImagesPullingTime="5.545204222s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 14:33:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:34:17.797825997 +0000 UTC m=+72.808742260" watchObservedRunningTime="2026-09-09 14:34:17.798065127 +0000 UTC m=+72.808981379" Sep 09 14:34:17.814065 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:34:17.814008 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-5vdl5" podStartSLOduration=67.584161 podStartE2EDuration="1m12.813991656s" totalImagesPullingTime="5.229830656s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 14:33:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:34:17.813396176 +0000 UTC m=+72.824312426" watchObservedRunningTime="2026-09-09 14:34:17.813991656 +0000 UTC m=+72.824907906" Sep 09 14:34:36.222233 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:34:36.222104 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-56c6dd68f4-xtcgl"] Sep 09 14:34:36.226337 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:34:36.226289 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56c6dd68f4-xtcgl" Sep 09 14:34:36.229112 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:34:36.229090 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Sep 09 14:34:36.230089 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:34:36.230074 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Sep 09 14:34:36.230157 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:34:36.230142 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Sep 09 14:34:36.230219 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:34:36.230202 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Sep 09 14:34:36.233072 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:34:36.233049 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-56c6dd68f4-xtcgl"] Sep 09 14:34:36.356837 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:34:36.356779 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6992c0dc-4204-4b32-ba7a-09620684f665-tmp\") pod \"klusterlet-addon-workmgr-56c6dd68f4-xtcgl\" (UID: \"6992c0dc-4204-4b32-ba7a-09620684f665\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56c6dd68f4-xtcgl" Sep 09 14:34:36.356837 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:34:36.356832 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/6992c0dc-4204-4b32-ba7a-09620684f665-klusterlet-config\") pod \"klusterlet-addon-workmgr-56c6dd68f4-xtcgl\" (UID: \"6992c0dc-4204-4b32-ba7a-09620684f665\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56c6dd68f4-xtcgl" Sep 09 14:34:36.357056 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:34:36.356966 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swrzh\" (UniqueName: \"kubernetes.io/projected/6992c0dc-4204-4b32-ba7a-09620684f665-kube-api-access-swrzh\") pod \"klusterlet-addon-workmgr-56c6dd68f4-xtcgl\" (UID: \"6992c0dc-4204-4b32-ba7a-09620684f665\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56c6dd68f4-xtcgl" Sep 09 14:34:36.458200 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:34:36.458148 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swrzh\" (UniqueName: \"kubernetes.io/projected/6992c0dc-4204-4b32-ba7a-09620684f665-kube-api-access-swrzh\") pod \"klusterlet-addon-workmgr-56c6dd68f4-xtcgl\" (UID: \"6992c0dc-4204-4b32-ba7a-09620684f665\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56c6dd68f4-xtcgl" Sep 09 14:34:36.458200 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:34:36.458204 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6992c0dc-4204-4b32-ba7a-09620684f665-tmp\") pod \"klusterlet-addon-workmgr-56c6dd68f4-xtcgl\" (UID: \"6992c0dc-4204-4b32-ba7a-09620684f665\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56c6dd68f4-xtcgl" Sep 09 14:34:36.458372 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:34:36.458228 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/6992c0dc-4204-4b32-ba7a-09620684f665-klusterlet-config\") pod \"klusterlet-addon-workmgr-56c6dd68f4-xtcgl\" (UID: \"6992c0dc-4204-4b32-ba7a-09620684f665\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56c6dd68f4-xtcgl" Sep 09 14:34:36.458611 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:34:36.458593 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6992c0dc-4204-4b32-ba7a-09620684f665-tmp\") pod \"klusterlet-addon-workmgr-56c6dd68f4-xtcgl\" (UID: \"6992c0dc-4204-4b32-ba7a-09620684f665\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56c6dd68f4-xtcgl" Sep 09 14:34:36.460957 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:34:36.460921 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/6992c0dc-4204-4b32-ba7a-09620684f665-klusterlet-config\") pod \"klusterlet-addon-workmgr-56c6dd68f4-xtcgl\" (UID: \"6992c0dc-4204-4b32-ba7a-09620684f665\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56c6dd68f4-xtcgl" Sep 09 14:34:36.468373 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:34:36.468347 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-swrzh\" (UniqueName: \"kubernetes.io/projected/6992c0dc-4204-4b32-ba7a-09620684f665-kube-api-access-swrzh\") pod \"klusterlet-addon-workmgr-56c6dd68f4-xtcgl\" (UID: \"6992c0dc-4204-4b32-ba7a-09620684f665\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56c6dd68f4-xtcgl" Sep 09 14:34:36.535743 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:34:36.535706 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56c6dd68f4-xtcgl" Sep 09 14:34:36.807166 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:34:36.807088 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-56c6dd68f4-xtcgl"] Sep 09 14:34:36.810322 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:34:36.810273 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6992c0dc_4204_4b32_ba7a_09620684f665.slice/crio-3114e7c79f7194df01d6c146d05a970dbedcda93ff2f35384b6e8255ebc74932 WatchSource:0}: Error finding container 3114e7c79f7194df01d6c146d05a970dbedcda93ff2f35384b6e8255ebc74932: Status 404 returned error can't find the container with id 3114e7c79f7194df01d6c146d05a970dbedcda93ff2f35384b6e8255ebc74932 Sep 09 14:34:36.834470 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:34:36.834438 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56c6dd68f4-xtcgl" event={"ID":"6992c0dc-4204-4b32-ba7a-09620684f665","Type":"ContainerStarted","Data":"3114e7c79f7194df01d6c146d05a970dbedcda93ff2f35384b6e8255ebc74932"} Sep 09 14:34:42.852942 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:34:42.852909 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56c6dd68f4-xtcgl" event={"ID":"6992c0dc-4204-4b32-ba7a-09620684f665","Type":"ContainerStarted","Data":"8a2e1c0342bfe3c89d693980216d2f6abb0185b1fcb55a7fa1a1f27bba8a63c0"} Sep 09 14:34:42.853366 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:34:42.853091 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56c6dd68f4-xtcgl" Sep 09 14:34:42.854839 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:34:42.854816 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56c6dd68f4-xtcgl" Sep 09 14:34:42.871839 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:34:42.871794 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56c6dd68f4-xtcgl" podStartSLOduration=1.7055643489999999 podStartE2EDuration="6.871778324s" totalImagesPullingTime="5.166213975s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 14:34:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:34:42.87089126 +0000 UTC m=+97.881807525" watchObservedRunningTime="2026-09-09 14:34:42.871778324 +0000 UTC m=+97.882694575" Sep 09 14:34:43.315667 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:34:43.315618 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1fd6e104-62ef-4fb0-9bf7-66da53264398-registry-tls\") pod \"image-registry-86f77b57bb-8dbpd\" (UID: \"1fd6e104-62ef-4fb0-9bf7-66da53264398\") " pod="openshift-image-registry/image-registry-86f77b57bb-8dbpd" Sep 09 14:34:43.315667 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:34:43.315675 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bb6068ac-f5ef-42a9-9d6e-a3ab55b63949-metrics-tls\") pod \"dns-default-pkvlm\" (UID: \"bb6068ac-f5ef-42a9-9d6e-a3ab55b63949\") " pod="openshift-dns/dns-default-pkvlm" Sep 09 14:34:43.315667 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:34:43.315695 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2177b959-0d75-41f6-b309-06f66757e45b-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-gpnd5\" (UID: \"2177b959-0d75-41f6-b309-06f66757e45b\") " pod="openshift-network-console/networking-console-plugin-79977f9844-gpnd5" Sep 09 14:34:43.315971 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:34:43.315772 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 09 14:34:43.315971 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:34:43.315788 2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 09 14:34:43.315971 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:34:43.315792 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-86f77b57bb-8dbpd: secret "image-registry-tls" not found Sep 09 14:34:43.315971 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:34:43.315794 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 09 14:34:43.315971 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:34:43.315840 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1fd6e104-62ef-4fb0-9bf7-66da53264398-registry-tls podName:1fd6e104-62ef-4fb0-9bf7-66da53264398 nodeName:}" failed. No retries permitted until 2026-09-09 14:35:47.315825842 +0000 UTC m=+162.326742071 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1fd6e104-62ef-4fb0-9bf7-66da53264398-registry-tls") pod "image-registry-86f77b57bb-8dbpd" (UID: "1fd6e104-62ef-4fb0-9bf7-66da53264398") : secret "image-registry-tls" not found Sep 09 14:34:43.315971 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:34:43.315853 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2177b959-0d75-41f6-b309-06f66757e45b-networking-console-plugin-cert podName:2177b959-0d75-41f6-b309-06f66757e45b nodeName:}" failed. No retries permitted until 2026-09-09 14:35:47.315847313 +0000 UTC m=+162.326763542 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/2177b959-0d75-41f6-b309-06f66757e45b-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-gpnd5" (UID: "2177b959-0d75-41f6-b309-06f66757e45b") : secret "networking-console-plugin-cert" not found Sep 09 14:34:43.315971 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:34:43.315874 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb6068ac-f5ef-42a9-9d6e-a3ab55b63949-metrics-tls podName:bb6068ac-f5ef-42a9-9d6e-a3ab55b63949 nodeName:}" failed. No retries permitted until 2026-09-09 14:35:47.315869359 +0000 UTC m=+162.326785588 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bb6068ac-f5ef-42a9-9d6e-a3ab55b63949-metrics-tls") pod "dns-default-pkvlm" (UID: "bb6068ac-f5ef-42a9-9d6e-a3ab55b63949") : secret "dns-default-metrics-tls" not found Sep 09 14:34:48.788951 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:34:48.788923 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-5vdl5" Sep 09 14:35:15.345647 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:35:15.345607 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab36c1e4-a65c-4cf9-ada4-f8a87d4b463b-metrics-certs\") pod \"network-metrics-daemon-wcj2p\" (UID: \"ab36c1e4-a65c-4cf9-ada4-f8a87d4b463b\") " pod="openshift-multus/network-metrics-daemon-wcj2p" Sep 09 14:35:15.346117 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:35:15.345747 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 09 14:35:15.346117 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:35:15.345809 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab36c1e4-a65c-4cf9-ada4-f8a87d4b463b-metrics-certs podName:ab36c1e4-a65c-4cf9-ada4-f8a87d4b463b nodeName:}" failed. No retries permitted until 2026-09-09 14:37:17.345794207 +0000 UTC m=+252.356710436 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ab36c1e4-a65c-4cf9-ada4-f8a87d4b463b-metrics-certs") pod "network-metrics-daemon-wcj2p" (UID: "ab36c1e4-a65c-4cf9-ada4-f8a87d4b463b") : secret "metrics-daemon-secret" not found Sep 09 14:35:42.408628 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:35:42.408571 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-79977f9844-gpnd5" podUID="2177b959-0d75-41f6-b309-06f66757e45b" Sep 09 14:35:42.426415 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:35:42.426375 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-86f77b57bb-8dbpd" podUID="1fd6e104-62ef-4fb0-9bf7-66da53264398" Sep 09 14:35:42.451105 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:35:42.451061 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-pkvlm" podUID="bb6068ac-f5ef-42a9-9d6e-a3ab55b63949" Sep 09 14:35:42.531797 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:35:42.531748 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-wcj2p" podUID="ab36c1e4-a65c-4cf9-ada4-f8a87d4b463b" Sep 09 14:35:42.992342 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:35:42.992315 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-gpnd5" Sep 09 14:35:42.992342 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:35:42.992328 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pkvlm" Sep 09 14:35:42.992585 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:35:42.992557 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-86f77b57bb-8dbpd" Sep 09 14:35:47.374844 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:35:47.374811 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bb6068ac-f5ef-42a9-9d6e-a3ab55b63949-metrics-tls\") pod \"dns-default-pkvlm\" (UID: \"bb6068ac-f5ef-42a9-9d6e-a3ab55b63949\") " pod="openshift-dns/dns-default-pkvlm" Sep 09 14:35:47.374844 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:35:47.374844 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2177b959-0d75-41f6-b309-06f66757e45b-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-gpnd5\" (UID: \"2177b959-0d75-41f6-b309-06f66757e45b\") " pod="openshift-network-console/networking-console-plugin-79977f9844-gpnd5" Sep 09 14:35:47.375295 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:35:47.374910 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1fd6e104-62ef-4fb0-9bf7-66da53264398-registry-tls\") pod \"image-registry-86f77b57bb-8dbpd\" (UID: \"1fd6e104-62ef-4fb0-9bf7-66da53264398\") " pod="openshift-image-registry/image-registry-86f77b57bb-8dbpd" Sep 09 14:35:47.375295 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:35:47.374964 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 09 14:35:47.375295 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:35:47.375008 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 09 14:35:47.375295 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:35:47.375011 2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 09 14:35:47.375295 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:35:47.375020 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-86f77b57bb-8dbpd: secret "image-registry-tls" not found Sep 09 14:35:47.375295 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:35:47.375033 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb6068ac-f5ef-42a9-9d6e-a3ab55b63949-metrics-tls podName:bb6068ac-f5ef-42a9-9d6e-a3ab55b63949 nodeName:}" failed. No retries permitted until 2026-09-09 14:37:49.375014271 +0000 UTC m=+284.385930520 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bb6068ac-f5ef-42a9-9d6e-a3ab55b63949-metrics-tls") pod "dns-default-pkvlm" (UID: "bb6068ac-f5ef-42a9-9d6e-a3ab55b63949") : secret "dns-default-metrics-tls" not found Sep 09 14:35:47.375295 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:35:47.375057 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1fd6e104-62ef-4fb0-9bf7-66da53264398-registry-tls podName:1fd6e104-62ef-4fb0-9bf7-66da53264398 nodeName:}" failed. No retries permitted until 2026-09-09 14:37:49.37504531 +0000 UTC m=+284.385961538 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1fd6e104-62ef-4fb0-9bf7-66da53264398-registry-tls") pod "image-registry-86f77b57bb-8dbpd" (UID: "1fd6e104-62ef-4fb0-9bf7-66da53264398") : secret "image-registry-tls" not found Sep 09 14:35:47.375295 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:35:47.375071 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2177b959-0d75-41f6-b309-06f66757e45b-networking-console-plugin-cert podName:2177b959-0d75-41f6-b309-06f66757e45b nodeName:}" failed. No retries permitted until 2026-09-09 14:37:49.375063306 +0000 UTC m=+284.385979537 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/2177b959-0d75-41f6-b309-06f66757e45b-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-gpnd5" (UID: "2177b959-0d75-41f6-b309-06f66757e45b") : secret "networking-console-plugin-cert" not found Sep 09 14:35:56.509793 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:35:56.509700 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wcj2p" Sep 09 14:36:42.136935 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:36:42.136904 2576 generic.go:356] "Generic (PLEG): container finished" podID="6992c0dc-4204-4b32-ba7a-09620684f665" containerID="8a2e1c0342bfe3c89d693980216d2f6abb0185b1fcb55a7fa1a1f27bba8a63c0" exitCode=1 Sep 09 14:36:42.137358 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:36:42.136965 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56c6dd68f4-xtcgl" event={"ID":"6992c0dc-4204-4b32-ba7a-09620684f665","Type":"ContainerDied","Data":"8a2e1c0342bfe3c89d693980216d2f6abb0185b1fcb55a7fa1a1f27bba8a63c0"} Sep 09 14:36:42.137358 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:36:42.137237 2576 scope.go:117] "RemoveContainer" containerID="8a2e1c0342bfe3c89d693980216d2f6abb0185b1fcb55a7fa1a1f27bba8a63c0" Sep 09 14:36:42.853801 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:36:42.853770 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56c6dd68f4-xtcgl" Sep 09 14:36:43.141490 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:36:43.141399 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56c6dd68f4-xtcgl" event={"ID":"6992c0dc-4204-4b32-ba7a-09620684f665","Type":"ContainerStarted","Data":"e19637b3149c13ef1010014d2510f3930bc8a59221f9ede2a1f55a7e610552a4"} Sep 09 14:36:43.141858 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:36:43.141610 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56c6dd68f4-xtcgl" Sep 09 14:36:43.142223 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:36:43.142203 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56c6dd68f4-xtcgl" Sep 09 14:37:17.386248 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:37:17.386198 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab36c1e4-a65c-4cf9-ada4-f8a87d4b463b-metrics-certs\") pod \"network-metrics-daemon-wcj2p\" (UID: \"ab36c1e4-a65c-4cf9-ada4-f8a87d4b463b\") " pod="openshift-multus/network-metrics-daemon-wcj2p" Sep 09 14:37:17.386770 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:37:17.386361 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 09 14:37:17.386770 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:37:17.386431 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab36c1e4-a65c-4cf9-ada4-f8a87d4b463b-metrics-certs podName:ab36c1e4-a65c-4cf9-ada4-f8a87d4b463b nodeName:}" failed. No retries permitted until 2026-09-09 14:39:19.386414157 +0000 UTC m=+374.397330386 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ab36c1e4-a65c-4cf9-ada4-f8a87d4b463b-metrics-certs") pod "network-metrics-daemon-wcj2p" (UID: "ab36c1e4-a65c-4cf9-ada4-f8a87d4b463b") : secret "metrics-daemon-secret" not found Sep 09 14:37:31.663527 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:37:31.663447 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-xtkrm"] Sep 09 14:37:31.666249 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:37:31.666233 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-xtkrm" Sep 09 14:37:31.669863 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:37:31.669840 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Sep 09 14:37:31.670006 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:37:31.669980 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Sep 09 14:37:31.670075 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:37:31.670012 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Sep 09 14:37:31.670984 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:37:31.670967 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Sep 09 14:37:31.671125 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:37:31.671108 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-kn8nw\"" Sep 09 14:37:31.682017 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:37:31.681993 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-xtkrm"] Sep 09 14:37:31.759257 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:37:31.759223 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-b8gsv"] Sep 09 14:37:31.762004 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:37:31.761983 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-b8gsv" Sep 09 14:37:31.765241 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:37:31.765224 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-tpfh7\"" Sep 09 14:37:31.782261 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:37:31.782234 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/342ca2d8-8fa0-45b6-9aaf-4bcf7117b021-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-xtkrm\" (UID: \"342ca2d8-8fa0-45b6-9aaf-4bcf7117b021\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-xtkrm" Sep 09 14:37:31.782375 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:37:31.782278 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/342ca2d8-8fa0-45b6-9aaf-4bcf7117b021-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-xtkrm\" (UID: \"342ca2d8-8fa0-45b6-9aaf-4bcf7117b021\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-xtkrm" Sep 09 14:37:31.782375 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:37:31.782365 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8wzp\" (UniqueName: \"kubernetes.io/projected/342ca2d8-8fa0-45b6-9aaf-4bcf7117b021-kube-api-access-t8wzp\") pod \"kube-storage-version-migrator-operator-8d7dd4675-xtkrm\" (UID: \"342ca2d8-8fa0-45b6-9aaf-4bcf7117b021\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-xtkrm" Sep 09 14:37:31.788706 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:37:31.788685 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-b8gsv"] Sep 09 14:37:31.883365 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:37:31.883323 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8wzp\" (UniqueName: \"kubernetes.io/projected/342ca2d8-8fa0-45b6-9aaf-4bcf7117b021-kube-api-access-t8wzp\") pod \"kube-storage-version-migrator-operator-8d7dd4675-xtkrm\" (UID: \"342ca2d8-8fa0-45b6-9aaf-4bcf7117b021\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-xtkrm" Sep 09 14:37:31.883568 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:37:31.883405 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pbxv\" (UniqueName: \"kubernetes.io/projected/176011b6-b6b3-4b9d-a930-848977cb0006-kube-api-access-7pbxv\") pod \"network-check-source-685456655f-b8gsv\" (UID: \"176011b6-b6b3-4b9d-a930-848977cb0006\") " pod="openshift-network-diagnostics/network-check-source-685456655f-b8gsv" Sep 09 14:37:31.883568 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:37:31.883445 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/342ca2d8-8fa0-45b6-9aaf-4bcf7117b021-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-xtkrm\" (UID: \"342ca2d8-8fa0-45b6-9aaf-4bcf7117b021\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-xtkrm" Sep 09 14:37:31.883568 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:37:31.883500 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/342ca2d8-8fa0-45b6-9aaf-4bcf7117b021-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-xtkrm\" (UID: \"342ca2d8-8fa0-45b6-9aaf-4bcf7117b021\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-xtkrm" Sep 09 14:37:31.884038 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:37:31.884012 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/342ca2d8-8fa0-45b6-9aaf-4bcf7117b021-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-xtkrm\" (UID: \"342ca2d8-8fa0-45b6-9aaf-4bcf7117b021\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-xtkrm" Sep 09 14:37:31.885792 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:37:31.885775 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/342ca2d8-8fa0-45b6-9aaf-4bcf7117b021-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-xtkrm\" (UID: \"342ca2d8-8fa0-45b6-9aaf-4bcf7117b021\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-xtkrm" Sep 09 14:37:31.893413 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:37:31.893390 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8wzp\" (UniqueName: \"kubernetes.io/projected/342ca2d8-8fa0-45b6-9aaf-4bcf7117b021-kube-api-access-t8wzp\") pod \"kube-storage-version-migrator-operator-8d7dd4675-xtkrm\" (UID: \"342ca2d8-8fa0-45b6-9aaf-4bcf7117b021\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-xtkrm" Sep 09 14:37:31.974726 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:37:31.974636 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-xtkrm" Sep 09 14:37:31.983956 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:37:31.983932 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pbxv\" (UniqueName: \"kubernetes.io/projected/176011b6-b6b3-4b9d-a930-848977cb0006-kube-api-access-7pbxv\") pod \"network-check-source-685456655f-b8gsv\" (UID: \"176011b6-b6b3-4b9d-a930-848977cb0006\") " pod="openshift-network-diagnostics/network-check-source-685456655f-b8gsv" Sep 09 14:37:31.995950 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:37:31.995913 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pbxv\" (UniqueName: \"kubernetes.io/projected/176011b6-b6b3-4b9d-a930-848977cb0006-kube-api-access-7pbxv\") pod \"network-check-source-685456655f-b8gsv\" (UID: \"176011b6-b6b3-4b9d-a930-848977cb0006\") " pod="openshift-network-diagnostics/network-check-source-685456655f-b8gsv" Sep 09 14:37:32.070391 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:37:32.070352 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-b8gsv" Sep 09 14:37:32.252011 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:37:32.251921 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-xtkrm"] Sep 09 14:37:32.255010 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:37:32.254984 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod342ca2d8_8fa0_45b6_9aaf_4bcf7117b021.slice/crio-397f602eda48a80f2f7b29cb7e65fde49e121b2fdf982204ff1473cc0a846bd8 WatchSource:0}: Error finding container 397f602eda48a80f2f7b29cb7e65fde49e121b2fdf982204ff1473cc0a846bd8: Status 404 returned error can't find the container with id 397f602eda48a80f2f7b29cb7e65fde49e121b2fdf982204ff1473cc0a846bd8 Sep 09 14:37:32.280004 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:37:32.279972 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-xtkrm" event={"ID":"342ca2d8-8fa0-45b6-9aaf-4bcf7117b021","Type":"ContainerStarted","Data":"397f602eda48a80f2f7b29cb7e65fde49e121b2fdf982204ff1473cc0a846bd8"} Sep 09 14:37:32.337063 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:37:32.337043 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-b8gsv"] Sep 09 14:37:32.339358 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:37:32.339334 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod176011b6_b6b3_4b9d_a930_848977cb0006.slice/crio-19961b18b6e955be20b2fac13034d654de6640ad07bf77d38099280addaa7b99 WatchSource:0}: Error finding container 19961b18b6e955be20b2fac13034d654de6640ad07bf77d38099280addaa7b99: Status 404 returned error can't find the container with id 19961b18b6e955be20b2fac13034d654de6640ad07bf77d38099280addaa7b99 Sep 09 14:37:33.284374 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:37:33.284336 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-b8gsv" event={"ID":"176011b6-b6b3-4b9d-a930-848977cb0006","Type":"ContainerStarted","Data":"864de0273aec749625531bee6c92487d74f4cb947b0c23efa635110c4ff1b22f"} Sep 09 14:37:33.284832 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:37:33.284417 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-b8gsv" event={"ID":"176011b6-b6b3-4b9d-a930-848977cb0006","Type":"ContainerStarted","Data":"19961b18b6e955be20b2fac13034d654de6640ad07bf77d38099280addaa7b99"} Sep 09 14:37:33.301257 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:37:33.301183 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-685456655f-b8gsv" podStartSLOduration=2.30116406 podStartE2EDuration="2.30116406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 14:37:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:37:33.3011539 +0000 UTC m=+268.312070173" watchObservedRunningTime="2026-09-09 14:37:33.30116406 +0000 UTC m=+268.312080312" Sep 09 14:37:34.289161 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:37:34.289090 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-xtkrm" event={"ID":"342ca2d8-8fa0-45b6-9aaf-4bcf7117b021","Type":"ContainerStarted","Data":"a6f0fd786d30092b32a6efef89eb8509bcc02759037876d91f7ec93deb8aea92"} Sep 09 14:37:34.309969 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:37:34.309919 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-xtkrm" podStartSLOduration=1.47516022 podStartE2EDuration="3.309897213s" totalImagesPullingTime="1.834736993s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 14:37:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:37:34.308714174 +0000 UTC m=+269.319630442" watchObservedRunningTime="2026-09-09 14:37:34.309897213 +0000 UTC m=+269.320813464" Sep 09 14:37:35.652046 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:37:35.652008 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-c5lwg"] Sep 09 14:37:35.655338 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:37:35.655294 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-c5lwg" Sep 09 14:37:35.658067 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:37:35.658032 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 09 14:37:35.658230 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:37:35.658210 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 09 14:37:35.658334 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:37:35.658291 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 09 14:37:35.659201 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:37:35.659179 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 09 14:37:35.659201 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:37:35.659195 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-jw79l\"" Sep 09 14:37:35.669660 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:37:35.669642 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-c5lwg"] Sep 09 14:37:35.711638 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:37:35.711605 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/23259762-461b-4ee1-911b-d4a595f0ecf0-crio-socket\") pod \"insights-runtime-extractor-c5lwg\" (UID: \"23259762-461b-4ee1-911b-d4a595f0ecf0\") " pod="openshift-insights/insights-runtime-extractor-c5lwg" Sep 09 14:37:35.711809 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:37:35.711646 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/23259762-461b-4ee1-911b-d4a595f0ecf0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c5lwg\" (UID: \"23259762-461b-4ee1-911b-d4a595f0ecf0\") " pod="openshift-insights/insights-runtime-extractor-c5lwg" Sep 09 14:37:35.711809 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:37:35.711696 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/23259762-461b-4ee1-911b-d4a595f0ecf0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c5lwg\" (UID: \"23259762-461b-4ee1-911b-d4a595f0ecf0\") " pod="openshift-insights/insights-runtime-extractor-c5lwg" Sep 09 14:37:35.711809 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:37:35.711753 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4wjh\" (UniqueName: \"kubernetes.io/projected/23259762-461b-4ee1-911b-d4a595f0ecf0-kube-api-access-x4wjh\") pod \"insights-runtime-extractor-c5lwg\" (UID: \"23259762-461b-4ee1-911b-d4a595f0ecf0\") " pod="openshift-insights/insights-runtime-extractor-c5lwg" Sep 09 14:37:35.711965 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:37:35.711822 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/23259762-461b-4ee1-911b-d4a595f0ecf0-data-volume\") pod \"insights-runtime-extractor-c5lwg\" (UID: \"23259762-461b-4ee1-911b-d4a595f0ecf0\") " pod="openshift-insights/insights-runtime-extractor-c5lwg" Sep 09 14:37:35.741915 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:37:35.741888 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-mxjvn"] Sep 09 14:37:35.744920 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:37:35.744902 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-mxjvn" Sep 09 14:37:35.748023 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:37:35.748006 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Sep 09 14:37:35.748186 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:37:35.748171 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-jx64b\"" Sep 09 14:37:35.748239 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:37:35.748227 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Sep 09 14:37:35.756120 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:37:35.756100 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-mxjvn"] Sep 09 14:37:35.812766 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:37:35.812741 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4wjh\" (UniqueName: \"kubernetes.io/projected/23259762-461b-4ee1-911b-d4a595f0ecf0-kube-api-access-x4wjh\") pod \"insights-runtime-extractor-c5lwg\" (UID: \"23259762-461b-4ee1-911b-d4a595f0ecf0\") " pod="openshift-insights/insights-runtime-extractor-c5lwg" Sep 09 14:37:35.812894 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:37:35.812786 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76gcv\" (UniqueName: \"kubernetes.io/projected/afa612b5-3120-49fb-a817-e85de6937bd3-kube-api-access-76gcv\") pod \"migrator-fdbb8fdff-mxjvn\" (UID: \"afa612b5-3120-49fb-a817-e85de6937bd3\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-mxjvn" Sep 09 14:37:35.812894 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:37:35.812807 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/23259762-461b-4ee1-911b-d4a595f0ecf0-data-volume\") pod \"insights-runtime-extractor-c5lwg\" (UID: \"23259762-461b-4ee1-911b-d4a595f0ecf0\") " pod="openshift-insights/insights-runtime-extractor-c5lwg" Sep 09 14:37:35.812894 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:37:35.812835 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/23259762-461b-4ee1-911b-d4a595f0ecf0-crio-socket\") pod \"insights-runtime-extractor-c5lwg\" (UID: \"23259762-461b-4ee1-911b-d4a595f0ecf0\") " pod="openshift-insights/insights-runtime-extractor-c5lwg" Sep 09 14:37:35.813015 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:37:35.812889 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/23259762-461b-4ee1-911b-d4a595f0ecf0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c5lwg\" (UID: \"23259762-461b-4ee1-911b-d4a595f0ecf0\") " pod="openshift-insights/insights-runtime-extractor-c5lwg" Sep 09 14:37:35.813015 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:37:35.812963 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/23259762-461b-4ee1-911b-d4a595f0ecf0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c5lwg\" (UID: \"23259762-461b-4ee1-911b-d4a595f0ecf0\") " pod="openshift-insights/insights-runtime-extractor-c5lwg" Sep 09 14:37:35.813015 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:37:35.812978 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 09 14:37:35.813015 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:37:35.812897 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/23259762-461b-4ee1-911b-d4a595f0ecf0-crio-socket\") pod \"insights-runtime-extractor-c5lwg\" (UID: \"23259762-461b-4ee1-911b-d4a595f0ecf0\") " pod="openshift-insights/insights-runtime-extractor-c5lwg" Sep 09 14:37:35.813210 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:37:35.813030 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23259762-461b-4ee1-911b-d4a595f0ecf0-insights-runtime-extractor-tls podName:23259762-461b-4ee1-911b-d4a595f0ecf0 nodeName:}" failed. No retries permitted until 2026-09-09 14:37:36.313013731 +0000 UTC m=+271.323929960 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/23259762-461b-4ee1-911b-d4a595f0ecf0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-c5lwg" (UID: "23259762-461b-4ee1-911b-d4a595f0ecf0") : secret "insights-runtime-extractor-tls" not found Sep 09 14:37:35.813210 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:37:35.813052 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 09 14:37:35.813210 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:37:35.813109 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/23259762-461b-4ee1-911b-d4a595f0ecf0-kube-rbac-proxy-cm podName:23259762-461b-4ee1-911b-d4a595f0ecf0 nodeName:}" failed. No retries permitted until 2026-09-09 14:37:36.313092671 +0000 UTC m=+271.324008906 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/23259762-461b-4ee1-911b-d4a595f0ecf0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-c5lwg" (UID: "23259762-461b-4ee1-911b-d4a595f0ecf0") : configmap "kube-rbac-proxy" not found Sep 09 14:37:35.813210 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:37:35.813131 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/23259762-461b-4ee1-911b-d4a595f0ecf0-data-volume\") pod \"insights-runtime-extractor-c5lwg\" (UID: \"23259762-461b-4ee1-911b-d4a595f0ecf0\") " pod="openshift-insights/insights-runtime-extractor-c5lwg" Sep 09 14:37:35.822835 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:37:35.822818 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4wjh\" (UniqueName: \"kubernetes.io/projected/23259762-461b-4ee1-911b-d4a595f0ecf0-kube-api-access-x4wjh\") pod \"insights-runtime-extractor-c5lwg\" (UID: \"23259762-461b-4ee1-911b-d4a595f0ecf0\") " pod="openshift-insights/insights-runtime-extractor-c5lwg" Sep 09 14:37:35.913948 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:37:35.913872 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76gcv\" (UniqueName: \"kubernetes.io/projected/afa612b5-3120-49fb-a817-e85de6937bd3-kube-api-access-76gcv\") pod \"migrator-fdbb8fdff-mxjvn\" (UID: \"afa612b5-3120-49fb-a817-e85de6937bd3\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-mxjvn" Sep 09 14:37:35.922455 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:37:35.922438 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-76gcv\" (UniqueName: \"kubernetes.io/projected/afa612b5-3120-49fb-a817-e85de6937bd3-kube-api-access-76gcv\") pod \"migrator-fdbb8fdff-mxjvn\" (UID: \"afa612b5-3120-49fb-a817-e85de6937bd3\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-mxjvn" Sep 09 14:37:36.053824 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:37:36.053775 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-mxjvn" Sep 09 14:37:36.318400 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:37:36.318368 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/23259762-461b-4ee1-911b-d4a595f0ecf0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c5lwg\" (UID: \"23259762-461b-4ee1-911b-d4a595f0ecf0\") " pod="openshift-insights/insights-runtime-extractor-c5lwg" Sep 09 14:37:36.318567 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:37:36.318429 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/23259762-461b-4ee1-911b-d4a595f0ecf0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c5lwg\" (UID: \"23259762-461b-4ee1-911b-d4a595f0ecf0\") " pod="openshift-insights/insights-runtime-extractor-c5lwg" Sep 09 14:37:36.318567 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:37:36.318495 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 09 14:37:36.318567 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:37:36.318516 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 09 14:37:36.318567 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:37:36.318544 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/23259762-461b-4ee1-911b-d4a595f0ecf0-kube-rbac-proxy-cm podName:23259762-461b-4ee1-911b-d4a595f0ecf0 nodeName:}" failed. No retries permitted until 2026-09-09 14:37:37.318531038 +0000 UTC m=+272.329447266 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/23259762-461b-4ee1-911b-d4a595f0ecf0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-c5lwg" (UID: "23259762-461b-4ee1-911b-d4a595f0ecf0") : configmap "kube-rbac-proxy" not found Sep 09 14:37:36.318567 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:37:36.318564 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23259762-461b-4ee1-911b-d4a595f0ecf0-insights-runtime-extractor-tls podName:23259762-461b-4ee1-911b-d4a595f0ecf0 nodeName:}" failed. No retries permitted until 2026-09-09 14:37:37.318551819 +0000 UTC m=+272.329468048 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/23259762-461b-4ee1-911b-d4a595f0ecf0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-c5lwg" (UID: "23259762-461b-4ee1-911b-d4a595f0ecf0") : secret "insights-runtime-extractor-tls" not found Sep 09 14:37:36.324187 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:37:36.324164 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-mxjvn"] Sep 09 14:37:36.327506 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:37:36.327485 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafa612b5_3120_49fb_a817_e85de6937bd3.slice/crio-7643ae359a8ce711c0fb97b509dd090c063c3dd68e73707d37887418b34fcf62 WatchSource:0}: Error finding container 7643ae359a8ce711c0fb97b509dd090c063c3dd68e73707d37887418b34fcf62: Status 404 returned error can't find the container with id 7643ae359a8ce711c0fb97b509dd090c063c3dd68e73707d37887418b34fcf62 Sep 09 14:37:37.299117 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:37:37.299082 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-mxjvn" event={"ID":"afa612b5-3120-49fb-a817-e85de6937bd3","Type":"ContainerStarted","Data":"7643ae359a8ce711c0fb97b509dd090c063c3dd68e73707d37887418b34fcf62"} Sep 09 14:37:37.325842 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:37:37.325811 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/23259762-461b-4ee1-911b-d4a595f0ecf0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c5lwg\" (UID: \"23259762-461b-4ee1-911b-d4a595f0ecf0\") " pod="openshift-insights/insights-runtime-extractor-c5lwg" Sep 09 14:37:37.326025 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:37:37.325910 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/23259762-461b-4ee1-911b-d4a595f0ecf0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c5lwg\" (UID: \"23259762-461b-4ee1-911b-d4a595f0ecf0\") " pod="openshift-insights/insights-runtime-extractor-c5lwg" Sep 09 14:37:37.326025 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:37:37.325949 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 09 14:37:37.326025 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:37:37.326025 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/23259762-461b-4ee1-911b-d4a595f0ecf0-kube-rbac-proxy-cm podName:23259762-461b-4ee1-911b-d4a595f0ecf0 nodeName:}" failed. No retries permitted until 2026-09-09 14:37:39.32600462 +0000 UTC m=+274.336920850 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/23259762-461b-4ee1-911b-d4a595f0ecf0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-c5lwg" (UID: "23259762-461b-4ee1-911b-d4a595f0ecf0") : configmap "kube-rbac-proxy" not found Sep 09 14:37:37.326182 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:37:37.326028 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 09 14:37:37.326182 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:37:37.326087 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23259762-461b-4ee1-911b-d4a595f0ecf0-insights-runtime-extractor-tls podName:23259762-461b-4ee1-911b-d4a595f0ecf0 nodeName:}" failed. No retries permitted until 2026-09-09 14:37:39.326070549 +0000 UTC m=+274.336986784 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/23259762-461b-4ee1-911b-d4a595f0ecf0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-c5lwg" (UID: "23259762-461b-4ee1-911b-d4a595f0ecf0") : secret "insights-runtime-extractor-tls" not found Sep 09 14:37:37.409355 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:37:37.409326 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5qkp6_90785a55-a3bd-41a7-8d33-85e3b82eb41f/dns-node-resolver/0.log" Sep 09 14:37:38.303849 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:37:38.303814 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-mxjvn" event={"ID":"afa612b5-3120-49fb-a817-e85de6937bd3","Type":"ContainerStarted","Data":"c4c5bd445aa9c585b03a2448c5a8aca563ac8965920860fb3e21238381b0ac5e"} Sep 09 14:37:38.304283 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:37:38.303874 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-mxjvn" event={"ID":"afa612b5-3120-49fb-a817-e85de6937bd3","Type":"ContainerStarted","Data":"d8430da5f451e360af2b49c32bda7ae161213d61c9b960fd075db6149feb4edb"} Sep 09 14:37:38.320404 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:37:38.320352 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-mxjvn" podStartSLOduration=2.224742956 podStartE2EDuration="3.320338758s" totalImagesPullingTime="1.095595802s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 14:37:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:37:38.320157212 +0000 UTC m=+273.331073475" watchObservedRunningTime="2026-09-09 14:37:38.320338758 +0000 UTC m=+273.331255009" Sep 09 14:37:38.436316 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:37:38.436270 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-75d99c484-sdzv5"] Sep 09 14:37:38.439282 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:37:38.439262 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-sdzv5" Sep 09 14:37:38.441873 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:37:38.441849 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Sep 09 14:37:38.441873 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:37:38.441867 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Sep 09 14:37:38.442023 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:37:38.441876 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Sep 09 14:37:38.442023 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:37:38.441970 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Sep 09 14:37:38.442166 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:37:38.442153 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-tfs5k\"" Sep 09 14:37:38.447568 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:37:38.447542 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-sdzv5"] Sep 09 14:37:38.536666 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:37:38.536627 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/a94612bf-824e-47cf-a60d-ef50cfe29ca5-signing-key\") pod \"service-ca-75d99c484-sdzv5\" (UID: \"a94612bf-824e-47cf-a60d-ef50cfe29ca5\") " pod="openshift-service-ca/service-ca-75d99c484-sdzv5" Sep 09 14:37:38.536825 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:37:38.536716 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/a94612bf-824e-47cf-a60d-ef50cfe29ca5-signing-cabundle\") pod \"service-ca-75d99c484-sdzv5\" (UID: \"a94612bf-824e-47cf-a60d-ef50cfe29ca5\") " pod="openshift-service-ca/service-ca-75d99c484-sdzv5" Sep 09 14:37:38.536825 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:37:38.536810 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qz7ws\" (UniqueName: \"kubernetes.io/projected/a94612bf-824e-47cf-a60d-ef50cfe29ca5-kube-api-access-qz7ws\") pod \"service-ca-75d99c484-sdzv5\" (UID: \"a94612bf-824e-47cf-a60d-ef50cfe29ca5\") " pod="openshift-service-ca/service-ca-75d99c484-sdzv5" Sep 09 14:37:38.609751 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:37:38.609686 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-cqf8f_edb6129a-6cbf-4d83-82b0-018fd0e73f8f/node-ca/0.log" Sep 09 14:37:38.637973 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:37:38.637947 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/a94612bf-824e-47cf-a60d-ef50cfe29ca5-signing-key\") pod \"service-ca-75d99c484-sdzv5\" (UID: \"a94612bf-824e-47cf-a60d-ef50cfe29ca5\") " pod="openshift-service-ca/service-ca-75d99c484-sdzv5" Sep 09 14:37:38.638070 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:37:38.638003 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/a94612bf-824e-47cf-a60d-ef50cfe29ca5-signing-cabundle\") pod \"service-ca-75d99c484-sdzv5\" (UID: \"a94612bf-824e-47cf-a60d-ef50cfe29ca5\") " pod="openshift-service-ca/service-ca-75d99c484-sdzv5" Sep 09 14:37:38.638070 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:37:38.638066 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qz7ws\" (UniqueName: \"kubernetes.io/projected/a94612bf-824e-47cf-a60d-ef50cfe29ca5-kube-api-access-qz7ws\") pod \"service-ca-75d99c484-sdzv5\" (UID: \"a94612bf-824e-47cf-a60d-ef50cfe29ca5\") " pod="openshift-service-ca/service-ca-75d99c484-sdzv5" Sep 09 14:37:38.638655 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:37:38.638633 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/a94612bf-824e-47cf-a60d-ef50cfe29ca5-signing-cabundle\") pod \"service-ca-75d99c484-sdzv5\" (UID: \"a94612bf-824e-47cf-a60d-ef50cfe29ca5\") " pod="openshift-service-ca/service-ca-75d99c484-sdzv5" Sep 09 14:37:38.640412 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:37:38.640394 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/a94612bf-824e-47cf-a60d-ef50cfe29ca5-signing-key\") pod \"service-ca-75d99c484-sdzv5\" (UID: \"a94612bf-824e-47cf-a60d-ef50cfe29ca5\") " pod="openshift-service-ca/service-ca-75d99c484-sdzv5" Sep 09 14:37:38.646669 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:37:38.646646 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qz7ws\" (UniqueName: \"kubernetes.io/projected/a94612bf-824e-47cf-a60d-ef50cfe29ca5-kube-api-access-qz7ws\") pod \"service-ca-75d99c484-sdzv5\" (UID: \"a94612bf-824e-47cf-a60d-ef50cfe29ca5\") " pod="openshift-service-ca/service-ca-75d99c484-sdzv5" Sep 09 14:37:38.747896 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:37:38.747845 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-sdzv5" Sep 09 14:37:39.018339 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:37:39.018186 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-sdzv5"] Sep 09 14:37:39.020626 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:37:39.020602 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda94612bf_824e_47cf_a60d_ef50cfe29ca5.slice/crio-6e9802ef6866a1c805f5c9dbae84faeb491f863899383d04c65774ada7fe4cbd WatchSource:0}: Error finding container 6e9802ef6866a1c805f5c9dbae84faeb491f863899383d04c65774ada7fe4cbd: Status 404 returned error can't find the container with id 6e9802ef6866a1c805f5c9dbae84faeb491f863899383d04c65774ada7fe4cbd Sep 09 14:37:39.209811 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:37:39.209779 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-4cj2m_bb7ae7e5-fd88-4894-b4c8-4b8ea88776d4/serve-healthcheck-canary/0.log" Sep 09 14:37:39.308125 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:37:39.308044 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-sdzv5" event={"ID":"a94612bf-824e-47cf-a60d-ef50cfe29ca5","Type":"ContainerStarted","Data":"6e9802ef6866a1c805f5c9dbae84faeb491f863899383d04c65774ada7fe4cbd"} Sep 09 14:37:39.343598 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:37:39.343572 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/23259762-461b-4ee1-911b-d4a595f0ecf0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c5lwg\" (UID: \"23259762-461b-4ee1-911b-d4a595f0ecf0\") " pod="openshift-insights/insights-runtime-extractor-c5lwg" Sep 09 14:37:39.343689 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:37:39.343633 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/23259762-461b-4ee1-911b-d4a595f0ecf0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c5lwg\" (UID: \"23259762-461b-4ee1-911b-d4a595f0ecf0\") " pod="openshift-insights/insights-runtime-extractor-c5lwg" Sep 09 14:37:39.343760 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:37:39.343714 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 09 14:37:39.343760 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:37:39.343729 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 09 14:37:39.343760 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:37:39.343761 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/23259762-461b-4ee1-911b-d4a595f0ecf0-kube-rbac-proxy-cm podName:23259762-461b-4ee1-911b-d4a595f0ecf0 nodeName:}" failed. No retries permitted until 2026-09-09 14:37:43.343749161 +0000 UTC m=+278.354665389 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/23259762-461b-4ee1-911b-d4a595f0ecf0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-c5lwg" (UID: "23259762-461b-4ee1-911b-d4a595f0ecf0") : configmap "kube-rbac-proxy" not found Sep 09 14:37:39.343884 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:37:39.343785 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23259762-461b-4ee1-911b-d4a595f0ecf0-insights-runtime-extractor-tls podName:23259762-461b-4ee1-911b-d4a595f0ecf0 nodeName:}" failed. No retries permitted until 2026-09-09 14:37:43.343768781 +0000 UTC m=+278.354685010 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/23259762-461b-4ee1-911b-d4a595f0ecf0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-c5lwg" (UID: "23259762-461b-4ee1-911b-d4a595f0ecf0") : secret "insights-runtime-extractor-tls" not found Sep 09 14:37:42.317997 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:37:42.317969 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-sdzv5" event={"ID":"a94612bf-824e-47cf-a60d-ef50cfe29ca5","Type":"ContainerStarted","Data":"3989de401efa34bf9911a1b554da0b40017a58e7a6dc9445559c60ceefe735b4"} Sep 09 14:37:42.349886 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:37:42.349831 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-75d99c484-sdzv5" podStartSLOduration=1.109991879 podStartE2EDuration="4.349808586s" totalImagesPullingTime="3.239816707s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 14:37:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:37:42.347681914 +0000 UTC m=+277.358598187" watchObservedRunningTime="2026-09-09 14:37:42.349808586 +0000 UTC m=+277.360724839" Sep 09 14:37:43.379148 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:37:43.379107 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/23259762-461b-4ee1-911b-d4a595f0ecf0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c5lwg\" (UID: \"23259762-461b-4ee1-911b-d4a595f0ecf0\") " pod="openshift-insights/insights-runtime-extractor-c5lwg" Sep 09 14:37:43.379625 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:37:43.379251 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/23259762-461b-4ee1-911b-d4a595f0ecf0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c5lwg\" (UID: \"23259762-461b-4ee1-911b-d4a595f0ecf0\") " pod="openshift-insights/insights-runtime-extractor-c5lwg" Sep 09 14:37:43.379682 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:37:43.379630 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 09 14:37:43.379733 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:37:43.379696 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23259762-461b-4ee1-911b-d4a595f0ecf0-insights-runtime-extractor-tls podName:23259762-461b-4ee1-911b-d4a595f0ecf0 nodeName:}" failed. No retries permitted until 2026-09-09 14:37:51.379674335 +0000 UTC m=+286.390590580 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/23259762-461b-4ee1-911b-d4a595f0ecf0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-c5lwg" (UID: "23259762-461b-4ee1-911b-d4a595f0ecf0") : secret "insights-runtime-extractor-tls" not found Sep 09 14:37:43.379797 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:37:43.379752 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 09 14:37:43.379797 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:37:43.379792 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/23259762-461b-4ee1-911b-d4a595f0ecf0-kube-rbac-proxy-cm podName:23259762-461b-4ee1-911b-d4a595f0ecf0 nodeName:}" failed. No retries permitted until 2026-09-09 14:37:51.37977955 +0000 UTC m=+286.390695782 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/23259762-461b-4ee1-911b-d4a595f0ecf0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-c5lwg" (UID: "23259762-461b-4ee1-911b-d4a595f0ecf0") : configmap "kube-rbac-proxy" not found Sep 09 14:37:45.992783 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:37:45.992740 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-79977f9844-gpnd5" podUID="2177b959-0d75-41f6-b309-06f66757e45b" Sep 09 14:37:45.992783 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:37:45.992740 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-pkvlm" podUID="bb6068ac-f5ef-42a9-9d6e-a3ab55b63949" Sep 09 14:37:45.993221 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:37:45.992925 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-86f77b57bb-8dbpd" podUID="1fd6e104-62ef-4fb0-9bf7-66da53264398" Sep 09 14:37:46.330748 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:37:46.330719 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-gpnd5" Sep 09 14:37:46.330748 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:37:46.330747 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pkvlm" Sep 09 14:37:46.330980 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:37:46.330950 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-86f77b57bb-8dbpd" Sep 09 14:37:49.432887 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:37:49.432854 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2177b959-0d75-41f6-b309-06f66757e45b-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-gpnd5\" (UID: \"2177b959-0d75-41f6-b309-06f66757e45b\") " pod="openshift-network-console/networking-console-plugin-79977f9844-gpnd5" Sep 09 14:37:49.433456 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:37:49.432914 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bb6068ac-f5ef-42a9-9d6e-a3ab55b63949-metrics-tls\") pod \"dns-default-pkvlm\" (UID: \"bb6068ac-f5ef-42a9-9d6e-a3ab55b63949\") " pod="openshift-dns/dns-default-pkvlm" Sep 09 14:37:49.433456 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:37:49.432962 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1fd6e104-62ef-4fb0-9bf7-66da53264398-registry-tls\") pod \"image-registry-86f77b57bb-8dbpd\" (UID: \"1fd6e104-62ef-4fb0-9bf7-66da53264398\") " pod="openshift-image-registry/image-registry-86f77b57bb-8dbpd" Sep 09 14:37:49.433456 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:37:49.432996 2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 09 14:37:49.433456 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:37:49.433068 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2177b959-0d75-41f6-b309-06f66757e45b-networking-console-plugin-cert podName:2177b959-0d75-41f6-b309-06f66757e45b nodeName:}" failed. No retries permitted until 2026-09-09 14:39:51.433051186 +0000 UTC m=+406.443967416 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/2177b959-0d75-41f6-b309-06f66757e45b-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-gpnd5" (UID: "2177b959-0d75-41f6-b309-06f66757e45b") : secret "networking-console-plugin-cert" not found Sep 09 14:37:49.435322 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:37:49.435286 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1fd6e104-62ef-4fb0-9bf7-66da53264398-registry-tls\") pod \"image-registry-86f77b57bb-8dbpd\" (UID: \"1fd6e104-62ef-4fb0-9bf7-66da53264398\") " pod="openshift-image-registry/image-registry-86f77b57bb-8dbpd" Sep 09 14:37:49.435420 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:37:49.435371 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bb6068ac-f5ef-42a9-9d6e-a3ab55b63949-metrics-tls\") pod \"dns-default-pkvlm\" (UID: \"bb6068ac-f5ef-42a9-9d6e-a3ab55b63949\") " pod="openshift-dns/dns-default-pkvlm" Sep 09 14:37:49.635361 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:37:49.635330 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-m98bc\"" Sep 09 14:37:49.635361 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:37:49.635332 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-btvxq\"" Sep 09 14:37:49.642380 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:37:49.642364 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pkvlm" Sep 09 14:37:49.642488 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:37:49.642473 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-86f77b57bb-8dbpd" Sep 09 14:37:49.927031 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:37:49.927002 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-pkvlm"] Sep 09 14:37:49.930608 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:37:49.930568 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb6068ac_f5ef_42a9_9d6e_a3ab55b63949.slice/crio-00c41883789fe8446cfdf4f34566579ccccbc497427171172baf6fc016ecf82a WatchSource:0}: Error finding container 00c41883789fe8446cfdf4f34566579ccccbc497427171172baf6fc016ecf82a: Status 404 returned error can't find the container with id 00c41883789fe8446cfdf4f34566579ccccbc497427171172baf6fc016ecf82a Sep 09 14:37:49.933041 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:37:49.933020 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-86f77b57bb-8dbpd"] Sep 09 14:37:49.935869 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:37:49.935848 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fd6e104_62ef_4fb0_9bf7_66da53264398.slice/crio-4963b068c149322734ad07dd4d475706157e20f3b43b41d9bb6312cbbed34a3c WatchSource:0}: Error finding container 4963b068c149322734ad07dd4d475706157e20f3b43b41d9bb6312cbbed34a3c: Status 404 returned error can't find the container with id 4963b068c149322734ad07dd4d475706157e20f3b43b41d9bb6312cbbed34a3c Sep 09 14:37:50.346203 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:37:50.346170 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-86f77b57bb-8dbpd" event={"ID":"1fd6e104-62ef-4fb0-9bf7-66da53264398","Type":"ContainerStarted","Data":"e68a665eae35d0148d56066614004bde66e6a7b96b8c18d7669379eb971928fe"} Sep 09 14:37:50.346417 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:37:50.346232 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-86f77b57bb-8dbpd" event={"ID":"1fd6e104-62ef-4fb0-9bf7-66da53264398","Type":"ContainerStarted","Data":"4963b068c149322734ad07dd4d475706157e20f3b43b41d9bb6312cbbed34a3c"} Sep 09 14:37:50.346417 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:37:50.346261 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-86f77b57bb-8dbpd" Sep 09 14:37:50.347579 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:37:50.347550 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pkvlm" event={"ID":"bb6068ac-f5ef-42a9-9d6e-a3ab55b63949","Type":"ContainerStarted","Data":"00c41883789fe8446cfdf4f34566579ccccbc497427171172baf6fc016ecf82a"} Sep 09 14:37:50.368315 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:37:50.368258 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-86f77b57bb-8dbpd" podStartSLOduration=285.368243718 podStartE2EDuration="4m45.368243718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 14:33:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:37:50.36696174 +0000 UTC m=+285.377878010" watchObservedRunningTime="2026-09-09 14:37:50.368243718 +0000 UTC m=+285.379159969" Sep 09 14:37:51.449343 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:37:51.448880 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/23259762-461b-4ee1-911b-d4a595f0ecf0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c5lwg\" (UID: \"23259762-461b-4ee1-911b-d4a595f0ecf0\") " pod="openshift-insights/insights-runtime-extractor-c5lwg" Sep 09 14:37:51.449343 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:37:51.449016 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/23259762-461b-4ee1-911b-d4a595f0ecf0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c5lwg\" (UID: \"23259762-461b-4ee1-911b-d4a595f0ecf0\") " pod="openshift-insights/insights-runtime-extractor-c5lwg" Sep 09 14:37:51.449343 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:37:51.449112 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 09 14:37:51.449343 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:37:51.449160 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23259762-461b-4ee1-911b-d4a595f0ecf0-insights-runtime-extractor-tls podName:23259762-461b-4ee1-911b-d4a595f0ecf0 nodeName:}" failed. No retries permitted until 2026-09-09 14:38:07.449146178 +0000 UTC m=+302.460062406 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/23259762-461b-4ee1-911b-d4a595f0ecf0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-c5lwg" (UID: "23259762-461b-4ee1-911b-d4a595f0ecf0") : secret "insights-runtime-extractor-tls" not found Sep 09 14:37:51.449875 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:37:51.449597 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/23259762-461b-4ee1-911b-d4a595f0ecf0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c5lwg\" (UID: \"23259762-461b-4ee1-911b-d4a595f0ecf0\") " pod="openshift-insights/insights-runtime-extractor-c5lwg" Sep 09 14:37:52.356680 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:37:52.356650 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pkvlm" event={"ID":"bb6068ac-f5ef-42a9-9d6e-a3ab55b63949","Type":"ContainerStarted","Data":"4246ec6dbc467ebd437965c4b55c415d07bfdd820cb3a8dd9dc49557bdee63bc"} Sep 09 14:37:52.356839 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:37:52.356696 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pkvlm" event={"ID":"bb6068ac-f5ef-42a9-9d6e-a3ab55b63949","Type":"ContainerStarted","Data":"e9e9a75c20bd5002ee60913a44e68a33a01b01128586465ccea98c77f0413c81"} Sep 09 14:37:52.356839 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:37:52.356726 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-pkvlm" Sep 09 14:37:52.375693 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:37:52.375635 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-pkvlm" podStartSLOduration=251.658218276 podStartE2EDuration="4m13.375615358s" totalImagesPullingTime="1.717397082s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 14:33:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:37:52.375142917 +0000 UTC m=+287.386059186" watchObservedRunningTime="2026-09-09 14:37:52.375615358 +0000 UTC m=+287.386531606" Sep 09 14:37:59.510620 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:37:59.510583 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-wcj2p" podUID="ab36c1e4-a65c-4cf9-ada4-f8a87d4b463b" Sep 09 14:38:02.366323 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:02.366273 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-pkvlm" Sep 09 14:38:05.377475 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:05.377441 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7xcn5_186aab90-266b-4eae-977a-3f053b423ef3/ovn-acl-logging/0.log" Sep 09 14:38:05.378869 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:05.378848 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7xcn5_186aab90-266b-4eae-977a-3f053b423ef3/ovn-acl-logging/0.log" Sep 09 14:38:05.379799 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:05.379772 2576 kubelet.go:1648] "Image garbage collection succeeded" Sep 09 14:38:07.463142 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:07.463081 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/23259762-461b-4ee1-911b-d4a595f0ecf0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c5lwg\" (UID: \"23259762-461b-4ee1-911b-d4a595f0ecf0\") " pod="openshift-insights/insights-runtime-extractor-c5lwg" Sep 09 14:38:07.465722 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:07.465688 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/23259762-461b-4ee1-911b-d4a595f0ecf0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c5lwg\" (UID: \"23259762-461b-4ee1-911b-d4a595f0ecf0\") " pod="openshift-insights/insights-runtime-extractor-c5lwg" Sep 09 14:38:07.767371 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:07.767345 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-jw79l\"" Sep 09 14:38:07.774879 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:07.774852 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-c5lwg" Sep 09 14:38:08.046041 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:08.046014 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-c5lwg"] Sep 09 14:38:08.048531 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:38:08.048493 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23259762_461b_4ee1_911b_d4a595f0ecf0.slice/crio-a53c85319012aead0dd50045c5cae10fbfe99539b8dc1974b82a0825ba1b9896 WatchSource:0}: Error finding container a53c85319012aead0dd50045c5cae10fbfe99539b8dc1974b82a0825ba1b9896: Status 404 returned error can't find the container with id a53c85319012aead0dd50045c5cae10fbfe99539b8dc1974b82a0825ba1b9896 Sep 09 14:38:08.119233 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:08.119211 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 09 14:38:08.411242 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:08.411146 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-c5lwg" event={"ID":"23259762-461b-4ee1-911b-d4a595f0ecf0","Type":"ContainerStarted","Data":"720ab0a34b25cd19e7767f3f114424d3013fe6ff99e091f640abeffb699546d1"} Sep 09 14:38:08.411242 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:08.411212 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-c5lwg" event={"ID":"23259762-461b-4ee1-911b-d4a595f0ecf0","Type":"ContainerStarted","Data":"a53c85319012aead0dd50045c5cae10fbfe99539b8dc1974b82a0825ba1b9896"} Sep 09 14:38:09.416379 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:09.416336 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-c5lwg" event={"ID":"23259762-461b-4ee1-911b-d4a595f0ecf0","Type":"ContainerStarted","Data":"dd4d62b93359d520d27c03c0d1355a7677893c31cdec5c4881e5b426cb3874bf"} Sep 09 14:38:09.646688 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:09.646653 2576 patch_prober.go:28] interesting pod/image-registry-86f77b57bb-8dbpd container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Sep 09 14:38:09.646861 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:09.646706 2576 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-86f77b57bb-8dbpd" podUID="1fd6e104-62ef-4fb0-9bf7-66da53264398" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 09 14:38:11.356055 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:11.356027 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-86f77b57bb-8dbpd" Sep 09 14:38:11.426421 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:11.426381 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-c5lwg" event={"ID":"23259762-461b-4ee1-911b-d4a595f0ecf0","Type":"ContainerStarted","Data":"f33ac6b27f499032254a00724c12c85cbd699989a9c9749189ea8df477b7f565"} Sep 09 14:38:11.463736 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:11.463680 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-c5lwg" podStartSLOduration=33.990110367 podStartE2EDuration="36.463661957s" totalImagesPullingTime="2.47355159s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 14:37:35 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:38:11.463092496 +0000 UTC m=+306.474008748" watchObservedRunningTime="2026-09-09 14:38:11.463661957 +0000 UTC m=+306.474578208" Sep 09 14:38:13.509875 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:13.509845 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wcj2p" Sep 09 14:38:14.612627 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:14.612592 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-wl2ff"] Sep 09 14:38:14.616228 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:14.616211 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-wl2ff" Sep 09 14:38:14.620866 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:14.620840 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Sep 09 14:38:14.620992 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:14.620894 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Sep 09 14:38:14.621226 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:14.621074 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Sep 09 14:38:14.621226 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:14.621088 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-xmw2v\"" Sep 09 14:38:14.621226 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:14.621118 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 09 14:38:14.621226 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:14.621207 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 09 14:38:14.621503 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:14.621374 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 09 14:38:14.625552 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:14.625535 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-65dg4"] Sep 09 14:38:14.628526 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:14.628509 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-65dg4" Sep 09 14:38:14.629130 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:14.629110 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-wl2ff"] Sep 09 14:38:14.630968 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:14.630949 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 09 14:38:14.631068 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:14.630951 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 09 14:38:14.631313 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:14.631288 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-j5n54\"" Sep 09 14:38:14.631551 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:14.631531 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 09 14:38:14.717570 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:14.717538 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f3feb49b-9668-453e-a4d5-1cfe142baa3d-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-wl2ff\" (UID: \"f3feb49b-9668-453e-a4d5-1cfe142baa3d\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-wl2ff" Sep 09 14:38:14.717732 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:14.717577 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/f3feb49b-9668-453e-a4d5-1cfe142baa3d-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-wl2ff\" (UID: \"f3feb49b-9668-453e-a4d5-1cfe142baa3d\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-wl2ff" Sep 09 14:38:14.717732 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:14.717609 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c270a080-896e-4955-84a7-5ce42fdb4367-root\") pod \"node-exporter-65dg4\" (UID: \"c270a080-896e-4955-84a7-5ce42fdb4367\") " pod="openshift-monitoring/node-exporter-65dg4" Sep 09 14:38:14.717732 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:14.717671 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c270a080-896e-4955-84a7-5ce42fdb4367-node-exporter-tls\") pod \"node-exporter-65dg4\" (UID: \"c270a080-896e-4955-84a7-5ce42fdb4367\") " pod="openshift-monitoring/node-exporter-65dg4" Sep 09 14:38:14.717732 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:14.717702 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c270a080-896e-4955-84a7-5ce42fdb4367-metrics-client-ca\") pod \"node-exporter-65dg4\" (UID: \"c270a080-896e-4955-84a7-5ce42fdb4367\") " pod="openshift-monitoring/node-exporter-65dg4" Sep 09 14:38:14.717939 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:14.717731 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c270a080-896e-4955-84a7-5ce42fdb4367-node-exporter-accelerators-collector-config\") pod \"node-exporter-65dg4\" (UID: \"c270a080-896e-4955-84a7-5ce42fdb4367\") " pod="openshift-monitoring/node-exporter-65dg4" Sep 09 14:38:14.717939 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:14.717760 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/f3feb49b-9668-453e-a4d5-1cfe142baa3d-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-wl2ff\" (UID: \"f3feb49b-9668-453e-a4d5-1cfe142baa3d\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-wl2ff" Sep 09 14:38:14.717939 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:14.717803 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c270a080-896e-4955-84a7-5ce42fdb4367-sys\") pod \"node-exporter-65dg4\" (UID: \"c270a080-896e-4955-84a7-5ce42fdb4367\") " pod="openshift-monitoring/node-exporter-65dg4" Sep 09 14:38:14.717939 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:14.717834 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bmrq\" (UniqueName: \"kubernetes.io/projected/f3feb49b-9668-453e-a4d5-1cfe142baa3d-kube-api-access-4bmrq\") pod \"kube-state-metrics-74cb57df7-wl2ff\" (UID: \"f3feb49b-9668-453e-a4d5-1cfe142baa3d\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-wl2ff" Sep 09 14:38:14.717939 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:14.717875 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4qqz\" (UniqueName: \"kubernetes.io/projected/c270a080-896e-4955-84a7-5ce42fdb4367-kube-api-access-m4qqz\") pod \"node-exporter-65dg4\" (UID: \"c270a080-896e-4955-84a7-5ce42fdb4367\") " pod="openshift-monitoring/node-exporter-65dg4" Sep 09 14:38:14.717939 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:14.717895 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f3feb49b-9668-453e-a4d5-1cfe142baa3d-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-wl2ff\" (UID: \"f3feb49b-9668-453e-a4d5-1cfe142baa3d\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-wl2ff" Sep 09 14:38:14.718236 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:14.717940 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f3feb49b-9668-453e-a4d5-1cfe142baa3d-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-wl2ff\" (UID: \"f3feb49b-9668-453e-a4d5-1cfe142baa3d\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-wl2ff" Sep 09 14:38:14.718236 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:14.717960 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c270a080-896e-4955-84a7-5ce42fdb4367-node-exporter-textfile\") pod \"node-exporter-65dg4\" (UID: \"c270a080-896e-4955-84a7-5ce42fdb4367\") " pod="openshift-monitoring/node-exporter-65dg4" Sep 09 14:38:14.718236 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:14.718020 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c270a080-896e-4955-84a7-5ce42fdb4367-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-65dg4\" (UID: \"c270a080-896e-4955-84a7-5ce42fdb4367\") " pod="openshift-monitoring/node-exporter-65dg4" Sep 09 14:38:14.718236 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:14.718043 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c270a080-896e-4955-84a7-5ce42fdb4367-node-exporter-wtmp\") pod \"node-exporter-65dg4\" (UID: \"c270a080-896e-4955-84a7-5ce42fdb4367\") " pod="openshift-monitoring/node-exporter-65dg4" Sep 09 14:38:14.818493 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:14.818463 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c270a080-896e-4955-84a7-5ce42fdb4367-node-exporter-wtmp\") pod \"node-exporter-65dg4\" (UID: \"c270a080-896e-4955-84a7-5ce42fdb4367\") " pod="openshift-monitoring/node-exporter-65dg4" Sep 09 14:38:14.818671 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:14.818507 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f3feb49b-9668-453e-a4d5-1cfe142baa3d-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-wl2ff\" (UID: \"f3feb49b-9668-453e-a4d5-1cfe142baa3d\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-wl2ff" Sep 09 14:38:14.818671 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:14.818640 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c270a080-896e-4955-84a7-5ce42fdb4367-node-exporter-wtmp\") pod \"node-exporter-65dg4\" (UID: \"c270a080-896e-4955-84a7-5ce42fdb4367\") " pod="openshift-monitoring/node-exporter-65dg4" Sep 09 14:38:14.818671 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:14.818639 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/f3feb49b-9668-453e-a4d5-1cfe142baa3d-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-wl2ff\" (UID: \"f3feb49b-9668-453e-a4d5-1cfe142baa3d\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-wl2ff" Sep 09 14:38:14.818842 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:14.818715 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c270a080-896e-4955-84a7-5ce42fdb4367-root\") pod \"node-exporter-65dg4\" (UID: \"c270a080-896e-4955-84a7-5ce42fdb4367\") " pod="openshift-monitoring/node-exporter-65dg4" Sep 09 14:38:14.818842 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:14.818739 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c270a080-896e-4955-84a7-5ce42fdb4367-node-exporter-tls\") pod \"node-exporter-65dg4\" (UID: \"c270a080-896e-4955-84a7-5ce42fdb4367\") " pod="openshift-monitoring/node-exporter-65dg4" Sep 09 14:38:14.818842 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:14.818753 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c270a080-896e-4955-84a7-5ce42fdb4367-metrics-client-ca\") pod \"node-exporter-65dg4\" (UID: \"c270a080-896e-4955-84a7-5ce42fdb4367\") " pod="openshift-monitoring/node-exporter-65dg4" Sep 09 14:38:14.818842 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:14.818777 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c270a080-896e-4955-84a7-5ce42fdb4367-node-exporter-accelerators-collector-config\") pod \"node-exporter-65dg4\" (UID: \"c270a080-896e-4955-84a7-5ce42fdb4367\") " pod="openshift-monitoring/node-exporter-65dg4" Sep 09 14:38:14.818842 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:14.818803 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/f3feb49b-9668-453e-a4d5-1cfe142baa3d-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-wl2ff\" (UID: \"f3feb49b-9668-453e-a4d5-1cfe142baa3d\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-wl2ff" Sep 09 14:38:14.818842 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:14.818826 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c270a080-896e-4955-84a7-5ce42fdb4367-sys\") pod \"node-exporter-65dg4\" (UID: \"c270a080-896e-4955-84a7-5ce42fdb4367\") " pod="openshift-monitoring/node-exporter-65dg4" Sep 09 14:38:14.819191 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:14.818852 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bmrq\" (UniqueName: \"kubernetes.io/projected/f3feb49b-9668-453e-a4d5-1cfe142baa3d-kube-api-access-4bmrq\") pod \"kube-state-metrics-74cb57df7-wl2ff\" (UID: \"f3feb49b-9668-453e-a4d5-1cfe142baa3d\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-wl2ff" Sep 09 14:38:14.819191 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:14.818882 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4qqz\" (UniqueName: \"kubernetes.io/projected/c270a080-896e-4955-84a7-5ce42fdb4367-kube-api-access-m4qqz\") pod \"node-exporter-65dg4\" (UID: \"c270a080-896e-4955-84a7-5ce42fdb4367\") " pod="openshift-monitoring/node-exporter-65dg4" Sep 09 14:38:14.819191 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:14.818889 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c270a080-896e-4955-84a7-5ce42fdb4367-root\") pod \"node-exporter-65dg4\" (UID: \"c270a080-896e-4955-84a7-5ce42fdb4367\") " pod="openshift-monitoring/node-exporter-65dg4" Sep 09 14:38:14.819191 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:14.818911 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f3feb49b-9668-453e-a4d5-1cfe142baa3d-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-wl2ff\" (UID: \"f3feb49b-9668-453e-a4d5-1cfe142baa3d\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-wl2ff" Sep 09 14:38:14.819191 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:38:14.818855 2576 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Sep 09 14:38:14.819191 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:14.818935 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c270a080-896e-4955-84a7-5ce42fdb4367-sys\") pod \"node-exporter-65dg4\" (UID: \"c270a080-896e-4955-84a7-5ce42fdb4367\") " pod="openshift-monitoring/node-exporter-65dg4" Sep 09 14:38:14.819191 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:14.818961 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f3feb49b-9668-453e-a4d5-1cfe142baa3d-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-wl2ff\" (UID: \"f3feb49b-9668-453e-a4d5-1cfe142baa3d\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-wl2ff" Sep 09 14:38:14.819191 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:38:14.818997 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c270a080-896e-4955-84a7-5ce42fdb4367-node-exporter-tls podName:c270a080-896e-4955-84a7-5ce42fdb4367 nodeName:}" failed. No retries permitted until 2026-09-09 14:38:15.318973188 +0000 UTC m=+310.329889424 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/c270a080-896e-4955-84a7-5ce42fdb4367-node-exporter-tls") pod "node-exporter-65dg4" (UID: "c270a080-896e-4955-84a7-5ce42fdb4367") : secret "node-exporter-tls" not found Sep 09 14:38:14.819191 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:14.819017 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c270a080-896e-4955-84a7-5ce42fdb4367-node-exporter-textfile\") pod \"node-exporter-65dg4\" (UID: \"c270a080-896e-4955-84a7-5ce42fdb4367\") " pod="openshift-monitoring/node-exporter-65dg4" Sep 09 14:38:14.819191 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:14.819022 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/f3feb49b-9668-453e-a4d5-1cfe142baa3d-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-wl2ff\" (UID: \"f3feb49b-9668-453e-a4d5-1cfe142baa3d\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-wl2ff" Sep 09 14:38:14.819191 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:14.819095 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c270a080-896e-4955-84a7-5ce42fdb4367-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-65dg4\" (UID: \"c270a080-896e-4955-84a7-5ce42fdb4367\") " pod="openshift-monitoring/node-exporter-65dg4" Sep 09 14:38:14.819750 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:14.819324 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c270a080-896e-4955-84a7-5ce42fdb4367-node-exporter-textfile\") pod \"node-exporter-65dg4\" (UID: \"c270a080-896e-4955-84a7-5ce42fdb4367\") " pod="openshift-monitoring/node-exporter-65dg4" Sep 09 14:38:14.819750 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:14.819566 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c270a080-896e-4955-84a7-5ce42fdb4367-metrics-client-ca\") pod \"node-exporter-65dg4\" (UID: \"c270a080-896e-4955-84a7-5ce42fdb4367\") " pod="openshift-monitoring/node-exporter-65dg4" Sep 09 14:38:14.820264 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:14.820236 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c270a080-896e-4955-84a7-5ce42fdb4367-node-exporter-accelerators-collector-config\") pod \"node-exporter-65dg4\" (UID: \"c270a080-896e-4955-84a7-5ce42fdb4367\") " pod="openshift-monitoring/node-exporter-65dg4" Sep 09 14:38:14.820475 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:14.820441 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f3feb49b-9668-453e-a4d5-1cfe142baa3d-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-wl2ff\" (UID: \"f3feb49b-9668-453e-a4d5-1cfe142baa3d\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-wl2ff" Sep 09 14:38:14.820664 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:14.820641 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/f3feb49b-9668-453e-a4d5-1cfe142baa3d-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-wl2ff\" (UID: \"f3feb49b-9668-453e-a4d5-1cfe142baa3d\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-wl2ff" Sep 09 14:38:14.821703 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:14.821679 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f3feb49b-9668-453e-a4d5-1cfe142baa3d-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-wl2ff\" (UID: \"f3feb49b-9668-453e-a4d5-1cfe142baa3d\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-wl2ff" Sep 09 14:38:14.821796 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:14.821764 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c270a080-896e-4955-84a7-5ce42fdb4367-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-65dg4\" (UID: \"c270a080-896e-4955-84a7-5ce42fdb4367\") " pod="openshift-monitoring/node-exporter-65dg4" Sep 09 14:38:14.822731 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:14.822711 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f3feb49b-9668-453e-a4d5-1cfe142baa3d-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-wl2ff\" (UID: \"f3feb49b-9668-453e-a4d5-1cfe142baa3d\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-wl2ff" Sep 09 14:38:14.834543 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:14.834522 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bmrq\" (UniqueName: \"kubernetes.io/projected/f3feb49b-9668-453e-a4d5-1cfe142baa3d-kube-api-access-4bmrq\") pod \"kube-state-metrics-74cb57df7-wl2ff\" (UID: \"f3feb49b-9668-453e-a4d5-1cfe142baa3d\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-wl2ff" Sep 09 14:38:14.835145 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:14.835120 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4qqz\" (UniqueName: \"kubernetes.io/projected/c270a080-896e-4955-84a7-5ce42fdb4367-kube-api-access-m4qqz\") pod \"node-exporter-65dg4\" (UID: \"c270a080-896e-4955-84a7-5ce42fdb4367\") " pod="openshift-monitoring/node-exporter-65dg4" Sep 09 14:38:14.927945 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:14.927864 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-wl2ff" Sep 09 14:38:15.207343 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:15.207156 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-wl2ff"] Sep 09 14:38:15.209706 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:38:15.209679 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3feb49b_9668_453e_a4d5_1cfe142baa3d.slice/crio-edf547638fda7400a820a17cf10f378845dc26c98e33d2d42e0bb252d68dc1ff WatchSource:0}: Error finding container edf547638fda7400a820a17cf10f378845dc26c98e33d2d42e0bb252d68dc1ff: Status 404 returned error can't find the container with id edf547638fda7400a820a17cf10f378845dc26c98e33d2d42e0bb252d68dc1ff Sep 09 14:38:15.322844 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:15.322810 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c270a080-896e-4955-84a7-5ce42fdb4367-node-exporter-tls\") pod \"node-exporter-65dg4\" (UID: \"c270a080-896e-4955-84a7-5ce42fdb4367\") " pod="openshift-monitoring/node-exporter-65dg4" Sep 09 14:38:15.325188 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:15.325170 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c270a080-896e-4955-84a7-5ce42fdb4367-node-exporter-tls\") pod \"node-exporter-65dg4\" (UID: \"c270a080-896e-4955-84a7-5ce42fdb4367\") " pod="openshift-monitoring/node-exporter-65dg4" Sep 09 14:38:15.440337 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:15.440293 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-wl2ff" event={"ID":"f3feb49b-9668-453e-a4d5-1cfe142baa3d","Type":"ContainerStarted","Data":"edf547638fda7400a820a17cf10f378845dc26c98e33d2d42e0bb252d68dc1ff"} Sep 09 14:38:15.539045 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:15.539014 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-65dg4" Sep 09 14:38:15.551679 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:38:15.551652 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc270a080_896e_4955_84a7_5ce42fdb4367.slice/crio-8c43739d8668085d34a3d006b2e66eef0c41a81055a6bf862b036c2713ffeed9 WatchSource:0}: Error finding container 8c43739d8668085d34a3d006b2e66eef0c41a81055a6bf862b036c2713ffeed9: Status 404 returned error can't find the container with id 8c43739d8668085d34a3d006b2e66eef0c41a81055a6bf862b036c2713ffeed9 Sep 09 14:38:15.655582 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:15.655546 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 09 14:38:15.659233 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:15.659215 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 09 14:38:15.661873 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:15.661845 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Sep 09 14:38:15.662013 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:15.661848 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Sep 09 14:38:15.662013 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:15.661912 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Sep 09 14:38:15.662013 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:15.661850 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-88pzd\"" Sep 09 14:38:15.662013 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:15.661850 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Sep 09 14:38:15.662220 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:15.661850 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Sep 09 14:38:15.662220 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:15.662050 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Sep 09 14:38:15.662220 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:15.661857 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Sep 09 14:38:15.662220 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:15.662196 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Sep 09 14:38:15.662386 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:15.662363 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Sep 09 14:38:15.676258 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:15.676234 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 09 14:38:15.725348 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:15.725277 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 14:38:15.725348 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:15.725339 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 14:38:15.725584 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:15.725405 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 14:38:15.725584 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:15.725443 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6-web-config\") pod \"alertmanager-main-0\" (UID: \"cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 14:38:15.725584 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:15.725462 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 14:38:15.725584 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:15.725492 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 14:38:15.725584 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:15.725509 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 14:38:15.725584 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:15.725564 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 14:38:15.725798 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:15.725598 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6-tls-assets\") pod \"alertmanager-main-0\" (UID: \"cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 14:38:15.725798 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:15.725620 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6-config-out\") pod \"alertmanager-main-0\" (UID: \"cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 14:38:15.725798 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:15.725639 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 14:38:15.725798 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:15.725668 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7nzn\" (UniqueName: \"kubernetes.io/projected/cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6-kube-api-access-p7nzn\") pod \"alertmanager-main-0\" (UID: \"cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 14:38:15.725798 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:15.725704 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6-config-volume\") pod \"alertmanager-main-0\" (UID: \"cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 14:38:15.826766 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:15.826659 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6-config-out\") pod \"alertmanager-main-0\" (UID: \"cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 14:38:15.826766 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:15.826703 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 14:38:15.826766 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:15.826740 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7nzn\" (UniqueName: \"kubernetes.io/projected/cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6-kube-api-access-p7nzn\") pod \"alertmanager-main-0\" (UID: \"cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 14:38:15.826766 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:15.826768 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6-config-volume\") pod \"alertmanager-main-0\" (UID: \"cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 14:38:15.827091 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:15.826814 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 14:38:15.827091 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:15.826841 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 14:38:15.827091 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:15.826877 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 14:38:15.827091 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:15.826903 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6-web-config\") pod \"alertmanager-main-0\" (UID: \"cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 14:38:15.827091 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:15.826925 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 14:38:15.827091 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:15.826968 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 14:38:15.827091 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:15.826992 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 14:38:15.827091 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:15.827032 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 14:38:15.827091 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:15.827069 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6-tls-assets\") pod \"alertmanager-main-0\" (UID: \"cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 14:38:15.828371 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:38:15.827701 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6-alertmanager-trusted-ca-bundle podName:cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6 nodeName:}" failed. No retries permitted until 2026-09-09 14:38:16.327678756 +0000 UTC m=+311.338595004 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6") : configmap references non-existent config key: ca-bundle.crt Sep 09 14:38:15.828371 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:15.827934 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 14:38:15.828371 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:38:15.827990 2576 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found Sep 09 14:38:15.828371 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:38:15.828038 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6-secret-alertmanager-main-tls podName:cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6 nodeName:}" failed. No retries permitted until 2026-09-09 14:38:16.328022298 +0000 UTC m=+311.338938541 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6") : secret "alertmanager-main-tls" not found Sep 09 14:38:15.828667 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:15.828630 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 14:38:15.830290 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:15.830185 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6-config-out\") pod \"alertmanager-main-0\" (UID: \"cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 14:38:15.830290 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:15.830226 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6-tls-assets\") pod \"alertmanager-main-0\" (UID: \"cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 14:38:15.830628 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:15.830524 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6-web-config\") pod \"alertmanager-main-0\" (UID: \"cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 14:38:15.830628 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:15.830581 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 14:38:15.831373 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:15.831330 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 14:38:15.832443 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:15.832399 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 14:38:15.832547 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:15.832525 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 14:38:15.833475 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:15.833457 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6-config-volume\") pod \"alertmanager-main-0\" (UID: \"cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 14:38:15.837061 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:15.837035 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7nzn\" (UniqueName: \"kubernetes.io/projected/cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6-kube-api-access-p7nzn\") pod \"alertmanager-main-0\" (UID: \"cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 14:38:16.330429 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:16.330385 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 14:38:16.330638 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:16.330446 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 14:38:16.331494 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:16.331438 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 14:38:16.333778 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:16.333748 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 14:38:16.446817 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:16.446770 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-65dg4" event={"ID":"c270a080-896e-4955-84a7-5ce42fdb4367","Type":"ContainerStarted","Data":"8c43739d8668085d34a3d006b2e66eef0c41a81055a6bf862b036c2713ffeed9"} Sep 09 14:38:16.570292 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:16.570258 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 09 14:38:17.249894 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:17.249593 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 09 14:38:17.251835 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:38:17.251800 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcddbe9cb_b5f9_448f_8e9c_96e4ee9d66f6.slice/crio-132f7c24b91161ea8453e718513572c3c9c2569085223cee4b66a56a3f2559ab WatchSource:0}: Error finding container 132f7c24b91161ea8453e718513572c3c9c2569085223cee4b66a56a3f2559ab: Status 404 returned error can't find the container with id 132f7c24b91161ea8453e718513572c3c9c2569085223cee4b66a56a3f2559ab Sep 09 14:38:17.452896 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:17.452852 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-wl2ff" event={"ID":"f3feb49b-9668-453e-a4d5-1cfe142baa3d","Type":"ContainerStarted","Data":"69b03e63024a8633a974435e094702ad1f1c652d1dd65d2b96262a30b1c16b8b"} Sep 09 14:38:17.453074 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:17.452931 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-wl2ff" event={"ID":"f3feb49b-9668-453e-a4d5-1cfe142baa3d","Type":"ContainerStarted","Data":"1a7f8310eb0ccf6429ae06fa24582f6edc5af905a7d063e16989e0efe7a90c6e"} Sep 09 14:38:17.453074 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:17.452959 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-wl2ff" event={"ID":"f3feb49b-9668-453e-a4d5-1cfe142baa3d","Type":"ContainerStarted","Data":"8cc4d2ffa1b2cb75fa24fefca9f7a6477469e6a191dc4e328002b5b3f2fb339c"} Sep 09 14:38:17.454247 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:17.454220 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6","Type":"ContainerStarted","Data":"132f7c24b91161ea8453e718513572c3c9c2569085223cee4b66a56a3f2559ab"} Sep 09 14:38:17.455811 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:17.455786 2576 generic.go:356] "Generic (PLEG): container finished" podID="c270a080-896e-4955-84a7-5ce42fdb4367" containerID="b988a57dde3ac096550ede6d9deb54e5553eca0486dfee862a724129ea5869ce" exitCode=0 Sep 09 14:38:17.455930 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:17.455823 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-65dg4" event={"ID":"c270a080-896e-4955-84a7-5ce42fdb4367","Type":"ContainerDied","Data":"b988a57dde3ac096550ede6d9deb54e5553eca0486dfee862a724129ea5869ce"} Sep 09 14:38:17.487327 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:17.483872 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-74cb57df7-wl2ff" podStartSLOduration=1.730169236 podStartE2EDuration="3.483850171s" totalImagesPullingTime="1.753680935s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 14:38:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:38:17.482105743 +0000 UTC m=+312.493022014" watchObservedRunningTime="2026-09-09 14:38:17.483850171 +0000 UTC m=+312.494766422" Sep 09 14:38:18.460654 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:18.460622 2576 generic.go:356] "Generic (PLEG): container finished" podID="cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6" containerID="98f881fe8cab49fe2f05e1a797ce1ae9a37febca53484cc1e0dfa7b19c34bf90" exitCode=0 Sep 09 14:38:18.461146 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:18.460719 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6","Type":"ContainerDied","Data":"98f881fe8cab49fe2f05e1a797ce1ae9a37febca53484cc1e0dfa7b19c34bf90"} Sep 09 14:38:18.463167 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:18.463141 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-65dg4" event={"ID":"c270a080-896e-4955-84a7-5ce42fdb4367","Type":"ContainerStarted","Data":"2e63d0f7201af81db800c6a21d23fb673678ccc44659e1208aa24e399bc732ef"} Sep 09 14:38:18.463266 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:18.463184 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-65dg4" event={"ID":"c270a080-896e-4955-84a7-5ce42fdb4367","Type":"ContainerStarted","Data":"b30337a6c1e1a936bd0b1e46b949d3ee958d566abc4886699d356f25c8a93ebc"} Sep 09 14:38:18.513656 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:18.513597 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-65dg4" podStartSLOduration=3.017578906 podStartE2EDuration="4.513578522s" totalImagesPullingTime="1.408292703s" totalInitContainerRuntime="87.706913ms" isStatefulPod=true podCreationTimestamp="2026-09-09 14:38:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:38:18.512495856 +0000 UTC m=+313.523412108" watchObservedRunningTime="2026-09-09 14:38:18.513578522 +0000 UTC m=+313.524494772" Sep 09 14:38:18.938925 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:18.938891 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-6b959dd5d4-bjzf7"] Sep 09 14:38:18.941942 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:18.941925 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6b959dd5d4-bjzf7" Sep 09 14:38:18.945422 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:18.945395 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Sep 09 14:38:18.945544 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:18.945483 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Sep 09 14:38:18.945544 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:18.945534 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Sep 09 14:38:18.945614 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:18.945534 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-qopiji47c1v9\"" Sep 09 14:38:18.945614 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:18.945533 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-5gkqq\"" Sep 09 14:38:18.945699 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:18.945626 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Sep 09 14:38:18.957841 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:18.957814 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6b959dd5d4-bjzf7"] Sep 09 14:38:19.057637 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:19.057592 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/f7a0b560-772a-41e6-82cf-8550c30f69d2-metrics-server-audit-profiles\") pod \"metrics-server-6b959dd5d4-bjzf7\" (UID: \"f7a0b560-772a-41e6-82cf-8550c30f69d2\") " pod="openshift-monitoring/metrics-server-6b959dd5d4-bjzf7" Sep 09 14:38:19.057827 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:19.057645 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sf5s\" (UniqueName: \"kubernetes.io/projected/f7a0b560-772a-41e6-82cf-8550c30f69d2-kube-api-access-2sf5s\") pod \"metrics-server-6b959dd5d4-bjzf7\" (UID: \"f7a0b560-772a-41e6-82cf-8550c30f69d2\") " pod="openshift-monitoring/metrics-server-6b959dd5d4-bjzf7" Sep 09 14:38:19.057827 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:19.057688 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/f7a0b560-772a-41e6-82cf-8550c30f69d2-secret-metrics-server-client-certs\") pod \"metrics-server-6b959dd5d4-bjzf7\" (UID: \"f7a0b560-772a-41e6-82cf-8550c30f69d2\") " pod="openshift-monitoring/metrics-server-6b959dd5d4-bjzf7" Sep 09 14:38:19.057827 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:19.057726 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/f7a0b560-772a-41e6-82cf-8550c30f69d2-client-ca-bundle\") pod \"metrics-server-6b959dd5d4-bjzf7\" (UID: \"f7a0b560-772a-41e6-82cf-8550c30f69d2\") " pod="openshift-monitoring/metrics-server-6b959dd5d4-bjzf7" Sep 09 14:38:19.057827 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:19.057792 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/f7a0b560-772a-41e6-82cf-8550c30f69d2-audit-log\") pod \"metrics-server-6b959dd5d4-bjzf7\" (UID: \"f7a0b560-772a-41e6-82cf-8550c30f69d2\") " pod="openshift-monitoring/metrics-server-6b959dd5d4-bjzf7" Sep 09 14:38:19.057977 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:19.057829 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7a0b560-772a-41e6-82cf-8550c30f69d2-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6b959dd5d4-bjzf7\" (UID: \"f7a0b560-772a-41e6-82cf-8550c30f69d2\") " pod="openshift-monitoring/metrics-server-6b959dd5d4-bjzf7" Sep 09 14:38:19.057977 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:19.057871 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/f7a0b560-772a-41e6-82cf-8550c30f69d2-secret-metrics-server-tls\") pod \"metrics-server-6b959dd5d4-bjzf7\" (UID: \"f7a0b560-772a-41e6-82cf-8550c30f69d2\") " pod="openshift-monitoring/metrics-server-6b959dd5d4-bjzf7" Sep 09 14:38:19.119985 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:19.119940 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-ff88c649f-rpn78"] Sep 09 14:38:19.123162 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:19.123138 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-ff88c649f-rpn78" Sep 09 14:38:19.126187 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:19.126151 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 09 14:38:19.126187 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:19.126174 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 09 14:38:19.126419 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:19.126161 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 09 14:38:19.126419 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:19.126278 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-wcftc\"" Sep 09 14:38:19.126419 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:19.126334 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 09 14:38:19.126716 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:19.126697 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 09 14:38:19.126819 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:19.126782 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 09 14:38:19.126819 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:19.126805 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 09 14:38:19.132498 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:19.132476 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Sep 09 14:38:19.136909 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:19.136890 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-ff88c649f-rpn78"] Sep 09 14:38:19.159247 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:19.159221 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/f7a0b560-772a-41e6-82cf-8550c30f69d2-secret-metrics-server-tls\") pod \"metrics-server-6b959dd5d4-bjzf7\" (UID: \"f7a0b560-772a-41e6-82cf-8550c30f69d2\") " pod="openshift-monitoring/metrics-server-6b959dd5d4-bjzf7" Sep 09 14:38:19.159450 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:19.159265 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea52faf1-b940-490a-9194-a0db3dec5b3f-trusted-ca-bundle\") pod \"console-ff88c649f-rpn78\" (UID: \"ea52faf1-b940-490a-9194-a0db3dec5b3f\") " pod="openshift-console/console-ff88c649f-rpn78" Sep 09 14:38:19.159450 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:19.159286 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/f7a0b560-772a-41e6-82cf-8550c30f69d2-metrics-server-audit-profiles\") pod \"metrics-server-6b959dd5d4-bjzf7\" (UID: \"f7a0b560-772a-41e6-82cf-8550c30f69d2\") " pod="openshift-monitoring/metrics-server-6b959dd5d4-bjzf7" Sep 09 14:38:19.159450 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:19.159326 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sf5s\" (UniqueName: \"kubernetes.io/projected/f7a0b560-772a-41e6-82cf-8550c30f69d2-kube-api-access-2sf5s\") pod \"metrics-server-6b959dd5d4-bjzf7\" (UID: \"f7a0b560-772a-41e6-82cf-8550c30f69d2\") " pod="openshift-monitoring/metrics-server-6b959dd5d4-bjzf7" Sep 09 14:38:19.159450 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:19.159383 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ea52faf1-b940-490a-9194-a0db3dec5b3f-oauth-serving-cert\") pod \"console-ff88c649f-rpn78\" (UID: \"ea52faf1-b940-490a-9194-a0db3dec5b3f\") " pod="openshift-console/console-ff88c649f-rpn78" Sep 09 14:38:19.159450 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:19.159414 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ea52faf1-b940-490a-9194-a0db3dec5b3f-console-config\") pod \"console-ff88c649f-rpn78\" (UID: \"ea52faf1-b940-490a-9194-a0db3dec5b3f\") " pod="openshift-console/console-ff88c649f-rpn78" Sep 09 14:38:19.159704 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:19.159453 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/f7a0b560-772a-41e6-82cf-8550c30f69d2-secret-metrics-server-client-certs\") pod \"metrics-server-6b959dd5d4-bjzf7\" (UID: \"f7a0b560-772a-41e6-82cf-8550c30f69d2\") " pod="openshift-monitoring/metrics-server-6b959dd5d4-bjzf7" Sep 09 14:38:19.159704 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:19.159523 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/f7a0b560-772a-41e6-82cf-8550c30f69d2-client-ca-bundle\") pod \"metrics-server-6b959dd5d4-bjzf7\" (UID: \"f7a0b560-772a-41e6-82cf-8550c30f69d2\") " pod="openshift-monitoring/metrics-server-6b959dd5d4-bjzf7" Sep 09 14:38:19.159704 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:19.159565 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ea52faf1-b940-490a-9194-a0db3dec5b3f-service-ca\") pod \"console-ff88c649f-rpn78\" (UID: \"ea52faf1-b940-490a-9194-a0db3dec5b3f\") " pod="openshift-console/console-ff88c649f-rpn78" Sep 09 14:38:19.159704 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:19.159604 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/f7a0b560-772a-41e6-82cf-8550c30f69d2-audit-log\") pod \"metrics-server-6b959dd5d4-bjzf7\" (UID: \"f7a0b560-772a-41e6-82cf-8550c30f69d2\") " pod="openshift-monitoring/metrics-server-6b959dd5d4-bjzf7" Sep 09 14:38:19.159704 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:19.159645 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7a0b560-772a-41e6-82cf-8550c30f69d2-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6b959dd5d4-bjzf7\" (UID: \"f7a0b560-772a-41e6-82cf-8550c30f69d2\") " pod="openshift-monitoring/metrics-server-6b959dd5d4-bjzf7" Sep 09 14:38:19.159704 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:19.159675 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ea52faf1-b940-490a-9194-a0db3dec5b3f-console-oauth-config\") pod \"console-ff88c649f-rpn78\" (UID: \"ea52faf1-b940-490a-9194-a0db3dec5b3f\") " pod="openshift-console/console-ff88c649f-rpn78" Sep 09 14:38:19.160000 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:19.159706 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jltf\" (UniqueName: \"kubernetes.io/projected/ea52faf1-b940-490a-9194-a0db3dec5b3f-kube-api-access-4jltf\") pod \"console-ff88c649f-rpn78\" (UID: \"ea52faf1-b940-490a-9194-a0db3dec5b3f\") " pod="openshift-console/console-ff88c649f-rpn78" Sep 09 14:38:19.160000 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:19.159756 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ea52faf1-b940-490a-9194-a0db3dec5b3f-console-serving-cert\") pod \"console-ff88c649f-rpn78\" (UID: \"ea52faf1-b940-490a-9194-a0db3dec5b3f\") " pod="openshift-console/console-ff88c649f-rpn78" Sep 09 14:38:19.160000 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:19.159945 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/f7a0b560-772a-41e6-82cf-8550c30f69d2-audit-log\") pod \"metrics-server-6b959dd5d4-bjzf7\" (UID: \"f7a0b560-772a-41e6-82cf-8550c30f69d2\") " pod="openshift-monitoring/metrics-server-6b959dd5d4-bjzf7" Sep 09 14:38:19.160352 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:19.160333 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/f7a0b560-772a-41e6-82cf-8550c30f69d2-metrics-server-audit-profiles\") pod \"metrics-server-6b959dd5d4-bjzf7\" (UID: \"f7a0b560-772a-41e6-82cf-8550c30f69d2\") " pod="openshift-monitoring/metrics-server-6b959dd5d4-bjzf7" Sep 09 14:38:19.160522 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:19.160500 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7a0b560-772a-41e6-82cf-8550c30f69d2-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6b959dd5d4-bjzf7\" (UID: \"f7a0b560-772a-41e6-82cf-8550c30f69d2\") " pod="openshift-monitoring/metrics-server-6b959dd5d4-bjzf7" Sep 09 14:38:19.161834 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:19.161806 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/f7a0b560-772a-41e6-82cf-8550c30f69d2-secret-metrics-server-tls\") pod \"metrics-server-6b959dd5d4-bjzf7\" (UID: \"f7a0b560-772a-41e6-82cf-8550c30f69d2\") " pod="openshift-monitoring/metrics-server-6b959dd5d4-bjzf7" Sep 09 14:38:19.162018 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:19.162001 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/f7a0b560-772a-41e6-82cf-8550c30f69d2-client-ca-bundle\") pod \"metrics-server-6b959dd5d4-bjzf7\" (UID: \"f7a0b560-772a-41e6-82cf-8550c30f69d2\") " pod="openshift-monitoring/metrics-server-6b959dd5d4-bjzf7" Sep 09 14:38:19.162084 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:19.162021 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/f7a0b560-772a-41e6-82cf-8550c30f69d2-secret-metrics-server-client-certs\") pod \"metrics-server-6b959dd5d4-bjzf7\" (UID: \"f7a0b560-772a-41e6-82cf-8550c30f69d2\") " pod="openshift-monitoring/metrics-server-6b959dd5d4-bjzf7" Sep 09 14:38:19.170584 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:19.170564 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sf5s\" (UniqueName: \"kubernetes.io/projected/f7a0b560-772a-41e6-82cf-8550c30f69d2-kube-api-access-2sf5s\") pod \"metrics-server-6b959dd5d4-bjzf7\" (UID: \"f7a0b560-772a-41e6-82cf-8550c30f69d2\") " pod="openshift-monitoring/metrics-server-6b959dd5d4-bjzf7" Sep 09 14:38:19.250492 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:19.250410 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6b959dd5d4-bjzf7" Sep 09 14:38:19.261263 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:19.260734 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ea52faf1-b940-490a-9194-a0db3dec5b3f-oauth-serving-cert\") pod \"console-ff88c649f-rpn78\" (UID: \"ea52faf1-b940-490a-9194-a0db3dec5b3f\") " pod="openshift-console/console-ff88c649f-rpn78" Sep 09 14:38:19.261263 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:19.260792 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ea52faf1-b940-490a-9194-a0db3dec5b3f-console-config\") pod \"console-ff88c649f-rpn78\" (UID: \"ea52faf1-b940-490a-9194-a0db3dec5b3f\") " pod="openshift-console/console-ff88c649f-rpn78" Sep 09 14:38:19.261263 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:19.260868 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ea52faf1-b940-490a-9194-a0db3dec5b3f-service-ca\") pod \"console-ff88c649f-rpn78\" (UID: \"ea52faf1-b940-490a-9194-a0db3dec5b3f\") " pod="openshift-console/console-ff88c649f-rpn78" Sep 09 14:38:19.261263 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:19.260926 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ea52faf1-b940-490a-9194-a0db3dec5b3f-console-oauth-config\") pod \"console-ff88c649f-rpn78\" (UID: \"ea52faf1-b940-490a-9194-a0db3dec5b3f\") " pod="openshift-console/console-ff88c649f-rpn78" Sep 09 14:38:19.261263 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:19.260957 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jltf\" (UniqueName: \"kubernetes.io/projected/ea52faf1-b940-490a-9194-a0db3dec5b3f-kube-api-access-4jltf\") pod \"console-ff88c649f-rpn78\" (UID: \"ea52faf1-b940-490a-9194-a0db3dec5b3f\") " pod="openshift-console/console-ff88c649f-rpn78" Sep 09 14:38:19.261263 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:19.261002 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ea52faf1-b940-490a-9194-a0db3dec5b3f-console-serving-cert\") pod \"console-ff88c649f-rpn78\" (UID: \"ea52faf1-b940-490a-9194-a0db3dec5b3f\") " pod="openshift-console/console-ff88c649f-rpn78" Sep 09 14:38:19.261263 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:19.261075 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea52faf1-b940-490a-9194-a0db3dec5b3f-trusted-ca-bundle\") pod \"console-ff88c649f-rpn78\" (UID: \"ea52faf1-b940-490a-9194-a0db3dec5b3f\") " pod="openshift-console/console-ff88c649f-rpn78" Sep 09 14:38:19.262008 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:19.261489 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ea52faf1-b940-490a-9194-a0db3dec5b3f-oauth-serving-cert\") pod \"console-ff88c649f-rpn78\" (UID: \"ea52faf1-b940-490a-9194-a0db3dec5b3f\") " pod="openshift-console/console-ff88c649f-rpn78" Sep 09 14:38:19.262008 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:19.261613 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ea52faf1-b940-490a-9194-a0db3dec5b3f-service-ca\") pod \"console-ff88c649f-rpn78\" (UID: \"ea52faf1-b940-490a-9194-a0db3dec5b3f\") " pod="openshift-console/console-ff88c649f-rpn78" Sep 09 14:38:19.262008 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:19.261683 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ea52faf1-b940-490a-9194-a0db3dec5b3f-console-config\") pod \"console-ff88c649f-rpn78\" (UID: \"ea52faf1-b940-490a-9194-a0db3dec5b3f\") " pod="openshift-console/console-ff88c649f-rpn78" Sep 09 14:38:19.262732 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:19.262704 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea52faf1-b940-490a-9194-a0db3dec5b3f-trusted-ca-bundle\") pod \"console-ff88c649f-rpn78\" (UID: \"ea52faf1-b940-490a-9194-a0db3dec5b3f\") " pod="openshift-console/console-ff88c649f-rpn78" Sep 09 14:38:19.264365 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:19.264337 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ea52faf1-b940-490a-9194-a0db3dec5b3f-console-oauth-config\") pod \"console-ff88c649f-rpn78\" (UID: \"ea52faf1-b940-490a-9194-a0db3dec5b3f\") " pod="openshift-console/console-ff88c649f-rpn78" Sep 09 14:38:19.264608 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:19.264568 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ea52faf1-b940-490a-9194-a0db3dec5b3f-console-serving-cert\") pod \"console-ff88c649f-rpn78\" (UID: \"ea52faf1-b940-490a-9194-a0db3dec5b3f\") " pod="openshift-console/console-ff88c649f-rpn78" Sep 09 14:38:19.274755 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:19.274715 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jltf\" (UniqueName: \"kubernetes.io/projected/ea52faf1-b940-490a-9194-a0db3dec5b3f-kube-api-access-4jltf\") pod \"console-ff88c649f-rpn78\" (UID: \"ea52faf1-b940-490a-9194-a0db3dec5b3f\") " pod="openshift-console/console-ff88c649f-rpn78" Sep 09 14:38:19.386214 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:19.386178 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-dbt55"] Sep 09 14:38:19.390650 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:19.390628 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-dbt55" Sep 09 14:38:19.393723 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:19.393696 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Sep 09 14:38:19.393859 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:19.393804 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-5klvs\"" Sep 09 14:38:19.403994 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:19.403966 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-dbt55"] Sep 09 14:38:19.433151 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:19.433116 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-ff88c649f-rpn78" Sep 09 14:38:19.464449 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:19.464014 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bd5fd069-dc9b-401d-84ad-450b3fc43886-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-dbt55\" (UID: \"bd5fd069-dc9b-401d-84ad-450b3fc43886\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-dbt55" Sep 09 14:38:19.464449 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:19.464064 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hw5tc\" (UniqueName: \"kubernetes.io/projected/bd5fd069-dc9b-401d-84ad-450b3fc43886-kube-api-access-hw5tc\") pod \"monitoring-plugin-85bbbbbf4-dbt55\" (UID: \"bd5fd069-dc9b-401d-84ad-450b3fc43886\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-dbt55" Sep 09 14:38:19.537665 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:19.537618 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6b959dd5d4-bjzf7"] Sep 09 14:38:19.541790 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:38:19.541747 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7a0b560_772a_41e6_82cf_8550c30f69d2.slice/crio-f8f6532e7140155d2a168bd6b8d700f51d60b817e9956db022311f7410d0f0a5 WatchSource:0}: Error finding container f8f6532e7140155d2a168bd6b8d700f51d60b817e9956db022311f7410d0f0a5: Status 404 returned error can't find the container with id f8f6532e7140155d2a168bd6b8d700f51d60b817e9956db022311f7410d0f0a5 Sep 09 14:38:19.565616 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:19.565578 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bd5fd069-dc9b-401d-84ad-450b3fc43886-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-dbt55\" (UID: \"bd5fd069-dc9b-401d-84ad-450b3fc43886\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-dbt55" Sep 09 14:38:19.565816 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:19.565648 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hw5tc\" (UniqueName: \"kubernetes.io/projected/bd5fd069-dc9b-401d-84ad-450b3fc43886-kube-api-access-hw5tc\") pod \"monitoring-plugin-85bbbbbf4-dbt55\" (UID: \"bd5fd069-dc9b-401d-84ad-450b3fc43886\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-dbt55" Sep 09 14:38:19.565816 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:38:19.565748 2576 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Sep 09 14:38:19.565922 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:38:19.565819 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd5fd069-dc9b-401d-84ad-450b3fc43886-monitoring-plugin-cert podName:bd5fd069-dc9b-401d-84ad-450b3fc43886 nodeName:}" failed. No retries permitted until 2026-09-09 14:38:20.06579805 +0000 UTC m=+315.076714299 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/bd5fd069-dc9b-401d-84ad-450b3fc43886-monitoring-plugin-cert") pod "monitoring-plugin-85bbbbbf4-dbt55" (UID: "bd5fd069-dc9b-401d-84ad-450b3fc43886") : secret "monitoring-plugin-cert" not found Sep 09 14:38:19.577100 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:19.577062 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hw5tc\" (UniqueName: \"kubernetes.io/projected/bd5fd069-dc9b-401d-84ad-450b3fc43886-kube-api-access-hw5tc\") pod \"monitoring-plugin-85bbbbbf4-dbt55\" (UID: \"bd5fd069-dc9b-401d-84ad-450b3fc43886\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-dbt55" Sep 09 14:38:19.722201 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:19.722168 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-ff88c649f-rpn78"] Sep 09 14:38:19.726506 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:38:19.726479 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea52faf1_b940_490a_9194_a0db3dec5b3f.slice/crio-3ffc0ecd99c3f7f4fde09dd14f61752025003cc8bd755b848ec7b04610b9d063 WatchSource:0}: Error finding container 3ffc0ecd99c3f7f4fde09dd14f61752025003cc8bd755b848ec7b04610b9d063: Status 404 returned error can't find the container with id 3ffc0ecd99c3f7f4fde09dd14f61752025003cc8bd755b848ec7b04610b9d063 Sep 09 14:38:20.070853 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:20.070808 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bd5fd069-dc9b-401d-84ad-450b3fc43886-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-dbt55\" (UID: \"bd5fd069-dc9b-401d-84ad-450b3fc43886\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-dbt55" Sep 09 14:38:20.073314 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:20.073281 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bd5fd069-dc9b-401d-84ad-450b3fc43886-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-dbt55\" (UID: \"bd5fd069-dc9b-401d-84ad-450b3fc43886\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-dbt55" Sep 09 14:38:20.302675 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:20.302652 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-dbt55" Sep 09 14:38:20.477632 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:20.477595 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6b959dd5d4-bjzf7" event={"ID":"f7a0b560-772a-41e6-82cf-8550c30f69d2","Type":"ContainerStarted","Data":"f8f6532e7140155d2a168bd6b8d700f51d60b817e9956db022311f7410d0f0a5"} Sep 09 14:38:20.481433 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:20.481204 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6","Type":"ContainerStarted","Data":"6f9c8317538f6f89e092f9c38eaf065ad3d7d6d0b3043c8b1588c9ba8c5ee11f"} Sep 09 14:38:20.481433 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:20.481285 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6","Type":"ContainerStarted","Data":"512a509962840a16e1e592d45fe921461276455b6af9d358b267d853d0185a8a"} Sep 09 14:38:20.481433 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:20.481348 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6","Type":"ContainerStarted","Data":"7703bb5317e77b1c338f67fafd294eed049e16d243da6283bc495505411c8b4a"} Sep 09 14:38:20.481433 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:20.481389 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6","Type":"ContainerStarted","Data":"d5a1f20e62599e56b16eb2a1dfe1a08b5d9cc99636ea81680ad422195f4d5ae3"} Sep 09 14:38:20.482822 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:20.482776 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-ff88c649f-rpn78" event={"ID":"ea52faf1-b940-490a-9194-a0db3dec5b3f","Type":"ContainerStarted","Data":"3ffc0ecd99c3f7f4fde09dd14f61752025003cc8bd755b848ec7b04610b9d063"} Sep 09 14:38:20.585889 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:20.585812 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-86f77b57bb-8dbpd"] Sep 09 14:38:20.588818 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:20.588752 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-dbt55"] Sep 09 14:38:20.592147 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:38:20.592120 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd5fd069_dc9b_401d_84ad_450b3fc43886.slice/crio-450ab597774ea74be4f85336a7e0e867bdd58a9d6739338095705a6ec65c2a12 WatchSource:0}: Error finding container 450ab597774ea74be4f85336a7e0e867bdd58a9d6739338095705a6ec65c2a12: Status 404 returned error can't find the container with id 450ab597774ea74be4f85336a7e0e867bdd58a9d6739338095705a6ec65c2a12 Sep 09 14:38:20.937960 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:20.937869 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 09 14:38:20.942523 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:20.942501 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 09 14:38:20.945851 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:20.945822 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Sep 09 14:38:20.946001 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:20.945894 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Sep 09 14:38:20.946160 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:20.946135 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-9r2zg\"" Sep 09 14:38:20.946270 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:20.946250 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Sep 09 14:38:20.946683 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:20.946647 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Sep 09 14:38:20.946683 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:20.946678 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-6cc0nabvvn998\"" Sep 09 14:38:20.946852 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:20.946745 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Sep 09 14:38:20.947084 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:20.947065 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Sep 09 14:38:20.947177 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:20.947096 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Sep 09 14:38:20.948160 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:20.947641 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Sep 09 14:38:20.948160 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:20.947705 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Sep 09 14:38:20.948160 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:20.947711 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Sep 09 14:38:20.948160 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:20.947906 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Sep 09 14:38:20.948160 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:20.947989 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Sep 09 14:38:20.949845 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:20.949822 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Sep 09 14:38:20.951516 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:20.951455 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Sep 09 14:38:20.960841 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:20.960814 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 09 14:38:20.980818 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:20.980785 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5a424647-68c0-4d35-87e8-496494c0b8f0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5a424647-68c0-4d35-87e8-496494c0b8f0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 14:38:20.980818 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:20.980831 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5a424647-68c0-4d35-87e8-496494c0b8f0-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5a424647-68c0-4d35-87e8-496494c0b8f0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 14:38:20.981065 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:20.980925 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5a424647-68c0-4d35-87e8-496494c0b8f0-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5a424647-68c0-4d35-87e8-496494c0b8f0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 14:38:20.981065 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:20.980978 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5a424647-68c0-4d35-87e8-496494c0b8f0-web-config\") pod \"prometheus-k8s-0\" (UID: \"5a424647-68c0-4d35-87e8-496494c0b8f0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 14:38:20.981065 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:20.981002 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qgzb\" (UniqueName: \"kubernetes.io/projected/5a424647-68c0-4d35-87e8-496494c0b8f0-kube-api-access-7qgzb\") pod \"prometheus-k8s-0\" (UID: \"5a424647-68c0-4d35-87e8-496494c0b8f0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 14:38:20.981222 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:20.981057 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a424647-68c0-4d35-87e8-496494c0b8f0-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5a424647-68c0-4d35-87e8-496494c0b8f0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 14:38:20.981222 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:20.981099 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5a424647-68c0-4d35-87e8-496494c0b8f0-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5a424647-68c0-4d35-87e8-496494c0b8f0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 14:38:20.981222 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:20.981141 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5a424647-68c0-4d35-87e8-496494c0b8f0-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5a424647-68c0-4d35-87e8-496494c0b8f0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 14:38:20.981222 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:20.981159 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a424647-68c0-4d35-87e8-496494c0b8f0-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5a424647-68c0-4d35-87e8-496494c0b8f0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 14:38:20.981222 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:20.981190 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5a424647-68c0-4d35-87e8-496494c0b8f0-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5a424647-68c0-4d35-87e8-496494c0b8f0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 14:38:20.981484 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:20.981233 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5a424647-68c0-4d35-87e8-496494c0b8f0-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5a424647-68c0-4d35-87e8-496494c0b8f0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 14:38:20.981484 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:20.981277 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5a424647-68c0-4d35-87e8-496494c0b8f0-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5a424647-68c0-4d35-87e8-496494c0b8f0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 14:38:20.981484 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:20.981340 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5a424647-68c0-4d35-87e8-496494c0b8f0-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5a424647-68c0-4d35-87e8-496494c0b8f0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 14:38:20.981484 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:20.981378 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/5a424647-68c0-4d35-87e8-496494c0b8f0-config\") pod \"prometheus-k8s-0\" (UID: \"5a424647-68c0-4d35-87e8-496494c0b8f0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 14:38:20.981484 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:20.981405 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5a424647-68c0-4d35-87e8-496494c0b8f0-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5a424647-68c0-4d35-87e8-496494c0b8f0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 14:38:20.981484 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:20.981442 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5a424647-68c0-4d35-87e8-496494c0b8f0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5a424647-68c0-4d35-87e8-496494c0b8f0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 14:38:20.981484 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:20.981458 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a424647-68c0-4d35-87e8-496494c0b8f0-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5a424647-68c0-4d35-87e8-496494c0b8f0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 14:38:20.981484 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:20.981478 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5a424647-68c0-4d35-87e8-496494c0b8f0-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5a424647-68c0-4d35-87e8-496494c0b8f0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 14:38:20.981810 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:20.981502 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5a424647-68c0-4d35-87e8-496494c0b8f0-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5a424647-68c0-4d35-87e8-496494c0b8f0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 14:38:20.981810 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:20.981593 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5a424647-68c0-4d35-87e8-496494c0b8f0-config-out\") pod \"prometheus-k8s-0\" (UID: \"5a424647-68c0-4d35-87e8-496494c0b8f0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 14:38:21.082523 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:21.082481 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5a424647-68c0-4d35-87e8-496494c0b8f0-config-out\") pod \"prometheus-k8s-0\" (UID: \"5a424647-68c0-4d35-87e8-496494c0b8f0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 14:38:21.082698 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:21.082534 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5a424647-68c0-4d35-87e8-496494c0b8f0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5a424647-68c0-4d35-87e8-496494c0b8f0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 14:38:21.082698 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:21.082570 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5a424647-68c0-4d35-87e8-496494c0b8f0-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5a424647-68c0-4d35-87e8-496494c0b8f0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 14:38:21.082698 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:21.082601 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5a424647-68c0-4d35-87e8-496494c0b8f0-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5a424647-68c0-4d35-87e8-496494c0b8f0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 14:38:21.082698 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:21.082637 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5a424647-68c0-4d35-87e8-496494c0b8f0-web-config\") pod \"prometheus-k8s-0\" (UID: \"5a424647-68c0-4d35-87e8-496494c0b8f0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 14:38:21.082698 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:21.082658 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qgzb\" (UniqueName: \"kubernetes.io/projected/5a424647-68c0-4d35-87e8-496494c0b8f0-kube-api-access-7qgzb\") pod \"prometheus-k8s-0\" (UID: \"5a424647-68c0-4d35-87e8-496494c0b8f0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 14:38:21.082698 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:21.082680 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a424647-68c0-4d35-87e8-496494c0b8f0-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5a424647-68c0-4d35-87e8-496494c0b8f0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 14:38:21.082992 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:21.082706 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5a424647-68c0-4d35-87e8-496494c0b8f0-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5a424647-68c0-4d35-87e8-496494c0b8f0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 14:38:21.083100 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:21.083067 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5a424647-68c0-4d35-87e8-496494c0b8f0-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5a424647-68c0-4d35-87e8-496494c0b8f0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 14:38:21.083247 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:21.083115 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a424647-68c0-4d35-87e8-496494c0b8f0-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5a424647-68c0-4d35-87e8-496494c0b8f0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 14:38:21.083247 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:21.083154 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5a424647-68c0-4d35-87e8-496494c0b8f0-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5a424647-68c0-4d35-87e8-496494c0b8f0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 14:38:21.083247 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:21.083187 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5a424647-68c0-4d35-87e8-496494c0b8f0-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5a424647-68c0-4d35-87e8-496494c0b8f0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 14:38:21.083247 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:21.083221 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5a424647-68c0-4d35-87e8-496494c0b8f0-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5a424647-68c0-4d35-87e8-496494c0b8f0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 14:38:21.083247 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:21.083246 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5a424647-68c0-4d35-87e8-496494c0b8f0-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5a424647-68c0-4d35-87e8-496494c0b8f0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 14:38:21.083582 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:21.083290 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/5a424647-68c0-4d35-87e8-496494c0b8f0-config\") pod \"prometheus-k8s-0\" (UID: \"5a424647-68c0-4d35-87e8-496494c0b8f0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 14:38:21.083582 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:21.083341 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5a424647-68c0-4d35-87e8-496494c0b8f0-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5a424647-68c0-4d35-87e8-496494c0b8f0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 14:38:21.083582 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:21.083385 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5a424647-68c0-4d35-87e8-496494c0b8f0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5a424647-68c0-4d35-87e8-496494c0b8f0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 14:38:21.083582 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:21.083410 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a424647-68c0-4d35-87e8-496494c0b8f0-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5a424647-68c0-4d35-87e8-496494c0b8f0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 14:38:21.083582 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:21.083434 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5a424647-68c0-4d35-87e8-496494c0b8f0-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5a424647-68c0-4d35-87e8-496494c0b8f0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 14:38:21.083582 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:21.083458 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5a424647-68c0-4d35-87e8-496494c0b8f0-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5a424647-68c0-4d35-87e8-496494c0b8f0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 14:38:21.085899 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:21.084645 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5a424647-68c0-4d35-87e8-496494c0b8f0-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5a424647-68c0-4d35-87e8-496494c0b8f0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 14:38:21.085899 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:21.085500 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5a424647-68c0-4d35-87e8-496494c0b8f0-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5a424647-68c0-4d35-87e8-496494c0b8f0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 14:38:21.086085 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:21.085934 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a424647-68c0-4d35-87e8-496494c0b8f0-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5a424647-68c0-4d35-87e8-496494c0b8f0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 14:38:21.086775 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:21.086239 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5a424647-68c0-4d35-87e8-496494c0b8f0-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5a424647-68c0-4d35-87e8-496494c0b8f0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 14:38:21.086775 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:21.086481 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5a424647-68c0-4d35-87e8-496494c0b8f0-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5a424647-68c0-4d35-87e8-496494c0b8f0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 14:38:21.086775 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:21.086676 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a424647-68c0-4d35-87e8-496494c0b8f0-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5a424647-68c0-4d35-87e8-496494c0b8f0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 14:38:21.087843 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:21.087721 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5a424647-68c0-4d35-87e8-496494c0b8f0-config-out\") pod \"prometheus-k8s-0\" (UID: \"5a424647-68c0-4d35-87e8-496494c0b8f0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 14:38:21.087957 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:21.087928 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a424647-68c0-4d35-87e8-496494c0b8f0-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5a424647-68c0-4d35-87e8-496494c0b8f0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 14:38:21.088155 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:21.088106 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5a424647-68c0-4d35-87e8-496494c0b8f0-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5a424647-68c0-4d35-87e8-496494c0b8f0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 14:38:21.088562 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:21.088410 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5a424647-68c0-4d35-87e8-496494c0b8f0-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5a424647-68c0-4d35-87e8-496494c0b8f0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 14:38:21.088562 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:21.088451 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5a424647-68c0-4d35-87e8-496494c0b8f0-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5a424647-68c0-4d35-87e8-496494c0b8f0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 14:38:21.088562 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:21.088451 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5a424647-68c0-4d35-87e8-496494c0b8f0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5a424647-68c0-4d35-87e8-496494c0b8f0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 14:38:21.089442 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:21.089397 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5a424647-68c0-4d35-87e8-496494c0b8f0-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5a424647-68c0-4d35-87e8-496494c0b8f0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 14:38:21.090722 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:21.090681 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5a424647-68c0-4d35-87e8-496494c0b8f0-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5a424647-68c0-4d35-87e8-496494c0b8f0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 14:38:21.090900 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:21.090834 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5a424647-68c0-4d35-87e8-496494c0b8f0-web-config\") pod \"prometheus-k8s-0\" (UID: \"5a424647-68c0-4d35-87e8-496494c0b8f0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 14:38:21.091324 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:21.091275 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/5a424647-68c0-4d35-87e8-496494c0b8f0-config\") pod \"prometheus-k8s-0\" (UID: \"5a424647-68c0-4d35-87e8-496494c0b8f0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 14:38:21.092189 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:21.092166 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5a424647-68c0-4d35-87e8-496494c0b8f0-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5a424647-68c0-4d35-87e8-496494c0b8f0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 14:38:21.092419 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:21.092398 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5a424647-68c0-4d35-87e8-496494c0b8f0-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5a424647-68c0-4d35-87e8-496494c0b8f0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 14:38:21.092528 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:21.092508 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5a424647-68c0-4d35-87e8-496494c0b8f0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5a424647-68c0-4d35-87e8-496494c0b8f0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 14:38:21.100947 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:21.100922 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qgzb\" (UniqueName: \"kubernetes.io/projected/5a424647-68c0-4d35-87e8-496494c0b8f0-kube-api-access-7qgzb\") pod \"prometheus-k8s-0\" (UID: \"5a424647-68c0-4d35-87e8-496494c0b8f0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 14:38:21.256412 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:21.256377 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 09 14:38:21.489052 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:21.489016 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-dbt55" event={"ID":"bd5fd069-dc9b-401d-84ad-450b3fc43886","Type":"ContainerStarted","Data":"450ab597774ea74be4f85336a7e0e867bdd58a9d6739338095705a6ec65c2a12"} Sep 09 14:38:21.494998 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:21.494936 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6","Type":"ContainerStarted","Data":"4c45967013db8112c49277a4660228c14fe79b1bbdfe23f1cd734e3f8f8bc58c"} Sep 09 14:38:21.756390 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:21.756359 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 09 14:38:22.502513 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:22.502460 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6b959dd5d4-bjzf7" event={"ID":"f7a0b560-772a-41e6-82cf-8550c30f69d2","Type":"ContainerStarted","Data":"78b23f7f01454024d45172ddeabbd06cb96e53be971b647e2b3d6153ca4ded11"} Sep 09 14:38:22.529286 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:22.529223 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-6b959dd5d4-bjzf7" podStartSLOduration=2.5568183270000002 podStartE2EDuration="4.529186889s" totalImagesPullingTime="1.972368562s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 14:38:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:38:22.527837713 +0000 UTC m=+317.538753983" watchObservedRunningTime="2026-09-09 14:38:22.529186889 +0000 UTC m=+317.540103140" Sep 09 14:38:22.881167 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:38:22.881131 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a424647_68c0_4d35_87e8_496494c0b8f0.slice/crio-5667b4bcc993461106a20c2e2d7c13f4394211276ee3517d7ac748bdb94d8ead WatchSource:0}: Error finding container 5667b4bcc993461106a20c2e2d7c13f4394211276ee3517d7ac748bdb94d8ead: Status 404 returned error can't find the container with id 5667b4bcc993461106a20c2e2d7c13f4394211276ee3517d7ac748bdb94d8ead Sep 09 14:38:23.322861 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:23.322830 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-ff88c649f-rpn78"] Sep 09 14:38:23.508075 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:23.507979 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-dbt55" event={"ID":"bd5fd069-dc9b-401d-84ad-450b3fc43886","Type":"ContainerStarted","Data":"048e8c16f59e542cd1f972699c85804b946ddaf92ed7486d054eab53d48dc9a8"} Sep 09 14:38:23.508706 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:23.508615 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-dbt55" Sep 09 14:38:23.519907 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:23.519882 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5a424647-68c0-4d35-87e8-496494c0b8f0","Type":"ContainerStarted","Data":"e7cf15d15c5614d31d998f0a6ade67c50f68137f7411c8d982ca83c0653e7906"} Sep 09 14:38:23.520106 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:23.520089 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5a424647-68c0-4d35-87e8-496494c0b8f0","Type":"ContainerStarted","Data":"5667b4bcc993461106a20c2e2d7c13f4394211276ee3517d7ac748bdb94d8ead"} Sep 09 14:38:23.520346 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:23.520331 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-dbt55" Sep 09 14:38:23.524220 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:23.524193 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6","Type":"ContainerStarted","Data":"1044d18d11fec3692265db487d33daf37b9a23183052d9ed61f78d20da34a5dd"} Sep 09 14:38:23.525681 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:23.525659 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-ff88c649f-rpn78" event={"ID":"ea52faf1-b940-490a-9194-a0db3dec5b3f","Type":"ContainerStarted","Data":"798c3390d1b6bd28fd77033818c6a433e84f1acb567c0bfe7723434ac657080c"} Sep 09 14:38:23.529552 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:23.529513 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-dbt55" podStartSLOduration=1.7230977360000002 podStartE2EDuration="4.529500677s" totalImagesPullingTime="2.806402941s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 14:38:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:38:23.528867564 +0000 UTC m=+318.539783832" watchObservedRunningTime="2026-09-09 14:38:23.529500677 +0000 UTC m=+318.540416948" Sep 09 14:38:23.579146 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:23.579045 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.944836358 podStartE2EDuration="8.579023086s" totalImagesPullingTime="5.618617836s" totalInitContainerRuntime="15.568892ms" isStatefulPod=false podCreationTimestamp="2026-09-09 14:38:15 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:38:23.576565646 +0000 UTC m=+318.587481915" watchObservedRunningTime="2026-09-09 14:38:23.579023086 +0000 UTC m=+318.589939338" Sep 09 14:38:23.595017 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:23.594970 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-ff88c649f-rpn78" podStartSLOduration=0.924294982 podStartE2EDuration="4.59495513s" totalImagesPullingTime="3.670660148s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 14:38:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:38:23.594147117 +0000 UTC m=+318.605063372" watchObservedRunningTime="2026-09-09 14:38:23.59495513 +0000 UTC m=+318.605871382" Sep 09 14:38:24.530928 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:24.530889 2576 generic.go:356] "Generic (PLEG): container finished" podID="5a424647-68c0-4d35-87e8-496494c0b8f0" containerID="e7cf15d15c5614d31d998f0a6ade67c50f68137f7411c8d982ca83c0653e7906" exitCode=0 Sep 09 14:38:24.531472 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:24.530973 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5a424647-68c0-4d35-87e8-496494c0b8f0","Type":"ContainerDied","Data":"e7cf15d15c5614d31d998f0a6ade67c50f68137f7411c8d982ca83c0653e7906"} Sep 09 14:38:28.547913 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:28.547882 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5a424647-68c0-4d35-87e8-496494c0b8f0","Type":"ContainerStarted","Data":"2621fdbf311cc0e64590b9500d3a3d05efe0dc2de7110165fe20a5d87b4808b7"} Sep 09 14:38:28.548262 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:28.547941 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5a424647-68c0-4d35-87e8-496494c0b8f0","Type":"ContainerStarted","Data":"43d1404b45d1b5b7d2c6e4480bc5808400fb3f6ed1d9c08b4219598103cd6159"} Sep 09 14:38:29.435389 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:29.435356 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-ff88c649f-rpn78" Sep 09 14:38:30.559874 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:30.559838 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5a424647-68c0-4d35-87e8-496494c0b8f0","Type":"ContainerStarted","Data":"b20fe58fae1d2c347b64af3a0db85da556a11cbd87377af08dadda0d7c76abe1"} Sep 09 14:38:30.560351 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:30.559912 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5a424647-68c0-4d35-87e8-496494c0b8f0","Type":"ContainerStarted","Data":"abe29843fa4a9ed94df2de59408d6645211f313e98636b037e11dae09c781ff9"} Sep 09 14:38:30.560351 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:30.559957 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5a424647-68c0-4d35-87e8-496494c0b8f0","Type":"ContainerStarted","Data":"3aa2add971bc20e83acb8526e058a06ca6bcdccf4df036920dd19ad361ffa5e4"} Sep 09 14:38:30.560351 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:30.560001 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5a424647-68c0-4d35-87e8-496494c0b8f0","Type":"ContainerStarted","Data":"413b924324e8ddab89a80df797d617dc6dce0133b6c7d79bc03bb66adaa08055"} Sep 09 14:38:30.591049 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:30.590999 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=5.327552447 podStartE2EDuration="10.590981834s" totalImagesPullingTime="5.240032535s" totalInitContainerRuntime="23.396852ms" isStatefulPod=false podCreationTimestamp="2026-09-09 14:38:20 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:38:30.588197454 +0000 UTC m=+325.599113752" watchObservedRunningTime="2026-09-09 14:38:30.590981834 +0000 UTC m=+325.601898118" Sep 09 14:38:31.257233 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:31.257200 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 09 14:38:39.251240 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:39.251208 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-6b959dd5d4-bjzf7" Sep 09 14:38:39.251826 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:39.251259 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-6b959dd5d4-bjzf7" Sep 09 14:38:45.617032 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:45.616989 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-86f77b57bb-8dbpd" podUID="1fd6e104-62ef-4fb0-9bf7-66da53264398" containerName="registry" containerID="cri-o://e68a665eae35d0148d56066614004bde66e6a7b96b8c18d7669379eb971928fe" gracePeriod=30 Sep 09 14:38:45.883523 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:45.883449 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-86f77b57bb-8dbpd" Sep 09 14:38:46.008155 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:46.008120 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1fd6e104-62ef-4fb0-9bf7-66da53264398-ca-trust-extracted\") pod \"1fd6e104-62ef-4fb0-9bf7-66da53264398\" (UID: \"1fd6e104-62ef-4fb0-9bf7-66da53264398\") " Sep 09 14:38:46.008363 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:46.008189 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j8zzf\" (UniqueName: \"kubernetes.io/projected/1fd6e104-62ef-4fb0-9bf7-66da53264398-kube-api-access-j8zzf\") pod \"1fd6e104-62ef-4fb0-9bf7-66da53264398\" (UID: \"1fd6e104-62ef-4fb0-9bf7-66da53264398\") " Sep 09 14:38:46.008363 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:46.008210 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1fd6e104-62ef-4fb0-9bf7-66da53264398-registry-certificates\") pod \"1fd6e104-62ef-4fb0-9bf7-66da53264398\" (UID: \"1fd6e104-62ef-4fb0-9bf7-66da53264398\") " Sep 09 14:38:46.008363 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:46.008233 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1fd6e104-62ef-4fb0-9bf7-66da53264398-bound-sa-token\") pod \"1fd6e104-62ef-4fb0-9bf7-66da53264398\" (UID: \"1fd6e104-62ef-4fb0-9bf7-66da53264398\") " Sep 09 14:38:46.008363 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:46.008264 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1fd6e104-62ef-4fb0-9bf7-66da53264398-image-registry-private-configuration\") pod \"1fd6e104-62ef-4fb0-9bf7-66da53264398\" (UID: \"1fd6e104-62ef-4fb0-9bf7-66da53264398\") " Sep 09 14:38:46.008584 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:46.008377 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1fd6e104-62ef-4fb0-9bf7-66da53264398-trusted-ca\") pod \"1fd6e104-62ef-4fb0-9bf7-66da53264398\" (UID: \"1fd6e104-62ef-4fb0-9bf7-66da53264398\") " Sep 09 14:38:46.008584 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:46.008456 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1fd6e104-62ef-4fb0-9bf7-66da53264398-registry-tls\") pod \"1fd6e104-62ef-4fb0-9bf7-66da53264398\" (UID: \"1fd6e104-62ef-4fb0-9bf7-66da53264398\") " Sep 09 14:38:46.008584 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:46.008493 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1fd6e104-62ef-4fb0-9bf7-66da53264398-installation-pull-secrets\") pod \"1fd6e104-62ef-4fb0-9bf7-66da53264398\" (UID: \"1fd6e104-62ef-4fb0-9bf7-66da53264398\") " Sep 09 14:38:46.008780 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:46.008748 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fd6e104-62ef-4fb0-9bf7-66da53264398-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "1fd6e104-62ef-4fb0-9bf7-66da53264398" (UID: "1fd6e104-62ef-4fb0-9bf7-66da53264398"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 14:38:46.008859 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:46.008832 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fd6e104-62ef-4fb0-9bf7-66da53264398-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "1fd6e104-62ef-4fb0-9bf7-66da53264398" (UID: "1fd6e104-62ef-4fb0-9bf7-66da53264398"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 14:38:46.011049 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:46.010978 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fd6e104-62ef-4fb0-9bf7-66da53264398-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "1fd6e104-62ef-4fb0-9bf7-66da53264398" (UID: "1fd6e104-62ef-4fb0-9bf7-66da53264398"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 14:38:46.011263 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:46.011243 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fd6e104-62ef-4fb0-9bf7-66da53264398-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "1fd6e104-62ef-4fb0-9bf7-66da53264398" (UID: "1fd6e104-62ef-4fb0-9bf7-66da53264398"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 14:38:46.011359 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:46.011270 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fd6e104-62ef-4fb0-9bf7-66da53264398-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "1fd6e104-62ef-4fb0-9bf7-66da53264398" (UID: "1fd6e104-62ef-4fb0-9bf7-66da53264398"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 14:38:46.011359 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:46.011283 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fd6e104-62ef-4fb0-9bf7-66da53264398-kube-api-access-j8zzf" (OuterVolumeSpecName: "kube-api-access-j8zzf") pod "1fd6e104-62ef-4fb0-9bf7-66da53264398" (UID: "1fd6e104-62ef-4fb0-9bf7-66da53264398"). InnerVolumeSpecName "kube-api-access-j8zzf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 14:38:46.011453 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:46.011384 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fd6e104-62ef-4fb0-9bf7-66da53264398-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "1fd6e104-62ef-4fb0-9bf7-66da53264398" (UID: "1fd6e104-62ef-4fb0-9bf7-66da53264398"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 14:38:46.017079 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:46.017053 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fd6e104-62ef-4fb0-9bf7-66da53264398-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "1fd6e104-62ef-4fb0-9bf7-66da53264398" (UID: "1fd6e104-62ef-4fb0-9bf7-66da53264398"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 14:38:46.109688 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:46.109657 2576 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1fd6e104-62ef-4fb0-9bf7-66da53264398-image-registry-private-configuration\") on node \"ip-10-0-133-66.ec2.internal\" DevicePath \"\"" Sep 09 14:38:46.109688 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:46.109682 2576 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1fd6e104-62ef-4fb0-9bf7-66da53264398-trusted-ca\") on node \"ip-10-0-133-66.ec2.internal\" DevicePath \"\"" Sep 09 14:38:46.109688 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:46.109691 2576 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1fd6e104-62ef-4fb0-9bf7-66da53264398-registry-tls\") on node \"ip-10-0-133-66.ec2.internal\" DevicePath \"\"" Sep 09 14:38:46.109876 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:46.109699 2576 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1fd6e104-62ef-4fb0-9bf7-66da53264398-installation-pull-secrets\") on node \"ip-10-0-133-66.ec2.internal\" DevicePath \"\"" Sep 09 14:38:46.109876 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:46.109708 2576 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1fd6e104-62ef-4fb0-9bf7-66da53264398-ca-trust-extracted\") on node \"ip-10-0-133-66.ec2.internal\" DevicePath \"\"" Sep 09 14:38:46.109876 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:46.109716 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8zzf\" (UniqueName: \"kubernetes.io/projected/1fd6e104-62ef-4fb0-9bf7-66da53264398-kube-api-access-j8zzf\") on node \"ip-10-0-133-66.ec2.internal\" DevicePath \"\"" Sep 09 14:38:46.109876 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:46.109725 2576 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1fd6e104-62ef-4fb0-9bf7-66da53264398-registry-certificates\") on node \"ip-10-0-133-66.ec2.internal\" DevicePath \"\"" Sep 09 14:38:46.109876 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:46.109733 2576 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1fd6e104-62ef-4fb0-9bf7-66da53264398-bound-sa-token\") on node \"ip-10-0-133-66.ec2.internal\" DevicePath \"\"" Sep 09 14:38:46.627351 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:46.627287 2576 generic.go:356] "Generic (PLEG): container finished" podID="1fd6e104-62ef-4fb0-9bf7-66da53264398" containerID="e68a665eae35d0148d56066614004bde66e6a7b96b8c18d7669379eb971928fe" exitCode=0 Sep 09 14:38:46.627351 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:46.627357 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-86f77b57bb-8dbpd" Sep 09 14:38:46.627882 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:46.627327 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-86f77b57bb-8dbpd" event={"ID":"1fd6e104-62ef-4fb0-9bf7-66da53264398","Type":"ContainerDied","Data":"e68a665eae35d0148d56066614004bde66e6a7b96b8c18d7669379eb971928fe"} Sep 09 14:38:46.627882 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:46.627414 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-86f77b57bb-8dbpd" event={"ID":"1fd6e104-62ef-4fb0-9bf7-66da53264398","Type":"ContainerDied","Data":"4963b068c149322734ad07dd4d475706157e20f3b43b41d9bb6312cbbed34a3c"} Sep 09 14:38:46.627882 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:46.627461 2576 scope.go:117] "RemoveContainer" containerID="e68a665eae35d0148d56066614004bde66e6a7b96b8c18d7669379eb971928fe" Sep 09 14:38:46.647037 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:46.647008 2576 scope.go:117] "RemoveContainer" containerID="e68a665eae35d0148d56066614004bde66e6a7b96b8c18d7669379eb971928fe" Sep 09 14:38:46.647389 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:38:46.647361 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e68a665eae35d0148d56066614004bde66e6a7b96b8c18d7669379eb971928fe\": container with ID starting with e68a665eae35d0148d56066614004bde66e6a7b96b8c18d7669379eb971928fe not found: ID does not exist" containerID="e68a665eae35d0148d56066614004bde66e6a7b96b8c18d7669379eb971928fe" Sep 09 14:38:46.647492 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:46.647398 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e68a665eae35d0148d56066614004bde66e6a7b96b8c18d7669379eb971928fe"} err="failed to get container status \"e68a665eae35d0148d56066614004bde66e6a7b96b8c18d7669379eb971928fe\": rpc error: code = NotFound desc = could not find container \"e68a665eae35d0148d56066614004bde66e6a7b96b8c18d7669379eb971928fe\": container with ID starting with e68a665eae35d0148d56066614004bde66e6a7b96b8c18d7669379eb971928fe not found: ID does not exist" Sep 09 14:38:46.649893 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:46.649874 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-86f77b57bb-8dbpd"] Sep 09 14:38:46.655376 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:46.655358 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-86f77b57bb-8dbpd"] Sep 09 14:38:47.514519 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:47.514487 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1fd6e104-62ef-4fb0-9bf7-66da53264398" path="/var/lib/kubelet/pods/1fd6e104-62ef-4fb0-9bf7-66da53264398/volumes" Sep 09 14:38:48.547175 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:48.547120 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-ff88c649f-rpn78" podUID="ea52faf1-b940-490a-9194-a0db3dec5b3f" containerName="console" containerID="cri-o://798c3390d1b6bd28fd77033818c6a433e84f1acb567c0bfe7723434ac657080c" gracePeriod=15 Sep 09 14:38:48.814176 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:48.814110 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-ff88c649f-rpn78" Sep 09 14:38:48.834696 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:48.834671 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea52faf1-b940-490a-9194-a0db3dec5b3f-trusted-ca-bundle\") pod \"ea52faf1-b940-490a-9194-a0db3dec5b3f\" (UID: \"ea52faf1-b940-490a-9194-a0db3dec5b3f\") " Sep 09 14:38:48.834858 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:48.834761 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ea52faf1-b940-490a-9194-a0db3dec5b3f-console-oauth-config\") pod \"ea52faf1-b940-490a-9194-a0db3dec5b3f\" (UID: \"ea52faf1-b940-490a-9194-a0db3dec5b3f\") " Sep 09 14:38:48.834858 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:48.834792 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ea52faf1-b940-490a-9194-a0db3dec5b3f-console-serving-cert\") pod \"ea52faf1-b940-490a-9194-a0db3dec5b3f\" (UID: \"ea52faf1-b940-490a-9194-a0db3dec5b3f\") " Sep 09 14:38:48.834858 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:48.834819 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ea52faf1-b940-490a-9194-a0db3dec5b3f-service-ca\") pod \"ea52faf1-b940-490a-9194-a0db3dec5b3f\" (UID: \"ea52faf1-b940-490a-9194-a0db3dec5b3f\") " Sep 09 14:38:48.835009 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:48.834873 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ea52faf1-b940-490a-9194-a0db3dec5b3f-oauth-serving-cert\") pod \"ea52faf1-b940-490a-9194-a0db3dec5b3f\" (UID: \"ea52faf1-b940-490a-9194-a0db3dec5b3f\") " Sep 09 14:38:48.835009 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:48.834895 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ea52faf1-b940-490a-9194-a0db3dec5b3f-console-config\") pod \"ea52faf1-b940-490a-9194-a0db3dec5b3f\" (UID: \"ea52faf1-b940-490a-9194-a0db3dec5b3f\") " Sep 09 14:38:48.835009 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:48.834921 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4jltf\" (UniqueName: \"kubernetes.io/projected/ea52faf1-b940-490a-9194-a0db3dec5b3f-kube-api-access-4jltf\") pod \"ea52faf1-b940-490a-9194-a0db3dec5b3f\" (UID: \"ea52faf1-b940-490a-9194-a0db3dec5b3f\") " Sep 09 14:38:48.835410 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:48.835377 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea52faf1-b940-490a-9194-a0db3dec5b3f-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "ea52faf1-b940-490a-9194-a0db3dec5b3f" (UID: "ea52faf1-b940-490a-9194-a0db3dec5b3f"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 14:38:48.835668 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:48.835471 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea52faf1-b940-490a-9194-a0db3dec5b3f-service-ca" (OuterVolumeSpecName: "service-ca") pod "ea52faf1-b940-490a-9194-a0db3dec5b3f" (UID: "ea52faf1-b940-490a-9194-a0db3dec5b3f"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 14:38:48.837407 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:48.837380 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea52faf1-b940-490a-9194-a0db3dec5b3f-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "ea52faf1-b940-490a-9194-a0db3dec5b3f" (UID: "ea52faf1-b940-490a-9194-a0db3dec5b3f"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 14:38:48.837548 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:48.837525 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea52faf1-b940-490a-9194-a0db3dec5b3f-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "ea52faf1-b940-490a-9194-a0db3dec5b3f" (UID: "ea52faf1-b940-490a-9194-a0db3dec5b3f"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 14:38:48.837588 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:48.837548 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea52faf1-b940-490a-9194-a0db3dec5b3f-kube-api-access-4jltf" (OuterVolumeSpecName: "kube-api-access-4jltf") pod "ea52faf1-b940-490a-9194-a0db3dec5b3f" (UID: "ea52faf1-b940-490a-9194-a0db3dec5b3f"). InnerVolumeSpecName "kube-api-access-4jltf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 14:38:48.840117 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:48.840093 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea52faf1-b940-490a-9194-a0db3dec5b3f-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "ea52faf1-b940-490a-9194-a0db3dec5b3f" (UID: "ea52faf1-b940-490a-9194-a0db3dec5b3f"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 14:38:48.844512 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:48.844479 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea52faf1-b940-490a-9194-a0db3dec5b3f-console-config" (OuterVolumeSpecName: "console-config") pod "ea52faf1-b940-490a-9194-a0db3dec5b3f" (UID: "ea52faf1-b940-490a-9194-a0db3dec5b3f"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 14:38:48.936040 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:48.935998 2576 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ea52faf1-b940-490a-9194-a0db3dec5b3f-oauth-serving-cert\") on node \"ip-10-0-133-66.ec2.internal\" DevicePath \"\"" Sep 09 14:38:48.936040 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:48.936031 2576 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ea52faf1-b940-490a-9194-a0db3dec5b3f-console-config\") on node \"ip-10-0-133-66.ec2.internal\" DevicePath \"\"" Sep 09 14:38:48.936040 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:48.936045 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jltf\" (UniqueName: \"kubernetes.io/projected/ea52faf1-b940-490a-9194-a0db3dec5b3f-kube-api-access-4jltf\") on node \"ip-10-0-133-66.ec2.internal\" DevicePath \"\"" Sep 09 14:38:48.936257 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:48.936057 2576 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea52faf1-b940-490a-9194-a0db3dec5b3f-trusted-ca-bundle\") on node \"ip-10-0-133-66.ec2.internal\" DevicePath \"\"" Sep 09 14:38:48.936257 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:48.936068 2576 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ea52faf1-b940-490a-9194-a0db3dec5b3f-console-oauth-config\") on node \"ip-10-0-133-66.ec2.internal\" DevicePath \"\"" Sep 09 14:38:48.936257 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:48.936079 2576 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ea52faf1-b940-490a-9194-a0db3dec5b3f-console-serving-cert\") on node \"ip-10-0-133-66.ec2.internal\" DevicePath \"\"" Sep 09 14:38:48.936257 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:48.936089 2576 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ea52faf1-b940-490a-9194-a0db3dec5b3f-service-ca\") on node \"ip-10-0-133-66.ec2.internal\" DevicePath \"\"" Sep 09 14:38:49.641710 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:49.641673 2576 generic.go:356] "Generic (PLEG): container finished" podID="ea52faf1-b940-490a-9194-a0db3dec5b3f" containerID="798c3390d1b6bd28fd77033818c6a433e84f1acb567c0bfe7723434ac657080c" exitCode=0 Sep 09 14:38:49.642117 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:49.641717 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-ff88c649f-rpn78" Sep 09 14:38:49.642117 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:49.641745 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-ff88c649f-rpn78" event={"ID":"ea52faf1-b940-490a-9194-a0db3dec5b3f","Type":"ContainerDied","Data":"798c3390d1b6bd28fd77033818c6a433e84f1acb567c0bfe7723434ac657080c"} Sep 09 14:38:49.642117 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:49.641789 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-ff88c649f-rpn78" event={"ID":"ea52faf1-b940-490a-9194-a0db3dec5b3f","Type":"ContainerDied","Data":"3ffc0ecd99c3f7f4fde09dd14f61752025003cc8bd755b848ec7b04610b9d063"} Sep 09 14:38:49.642117 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:49.641813 2576 scope.go:117] "RemoveContainer" containerID="798c3390d1b6bd28fd77033818c6a433e84f1acb567c0bfe7723434ac657080c" Sep 09 14:38:49.656981 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:49.656962 2576 scope.go:117] "RemoveContainer" containerID="798c3390d1b6bd28fd77033818c6a433e84f1acb567c0bfe7723434ac657080c" Sep 09 14:38:49.657292 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:38:49.657265 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"798c3390d1b6bd28fd77033818c6a433e84f1acb567c0bfe7723434ac657080c\": container with ID starting with 798c3390d1b6bd28fd77033818c6a433e84f1acb567c0bfe7723434ac657080c not found: ID does not exist" containerID="798c3390d1b6bd28fd77033818c6a433e84f1acb567c0bfe7723434ac657080c" Sep 09 14:38:49.657420 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:49.657317 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"798c3390d1b6bd28fd77033818c6a433e84f1acb567c0bfe7723434ac657080c"} err="failed to get container status \"798c3390d1b6bd28fd77033818c6a433e84f1acb567c0bfe7723434ac657080c\": rpc error: code = NotFound desc = could not find container \"798c3390d1b6bd28fd77033818c6a433e84f1acb567c0bfe7723434ac657080c\": container with ID starting with 798c3390d1b6bd28fd77033818c6a433e84f1acb567c0bfe7723434ac657080c not found: ID does not exist" Sep 09 14:38:49.659109 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:49.659090 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-ff88c649f-rpn78"] Sep 09 14:38:49.662573 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:49.662554 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-ff88c649f-rpn78"] Sep 09 14:38:51.515576 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:51.515541 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea52faf1-b940-490a-9194-a0db3dec5b3f" path="/var/lib/kubelet/pods/ea52faf1-b940-490a-9194-a0db3dec5b3f/volumes" Sep 09 14:38:59.255701 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:59.255628 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-6b959dd5d4-bjzf7" Sep 09 14:38:59.259311 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:38:59.259277 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-6b959dd5d4-bjzf7" Sep 09 14:39:05.703721 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:05.703630 2576 generic.go:356] "Generic (PLEG): container finished" podID="342ca2d8-8fa0-45b6-9aaf-4bcf7117b021" containerID="a6f0fd786d30092b32a6efef89eb8509bcc02759037876d91f7ec93deb8aea92" exitCode=0 Sep 09 14:39:05.703721 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:05.703704 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-xtkrm" event={"ID":"342ca2d8-8fa0-45b6-9aaf-4bcf7117b021","Type":"ContainerDied","Data":"a6f0fd786d30092b32a6efef89eb8509bcc02759037876d91f7ec93deb8aea92"} Sep 09 14:39:05.704134 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:05.703990 2576 scope.go:117] "RemoveContainer" containerID="a6f0fd786d30092b32a6efef89eb8509bcc02759037876d91f7ec93deb8aea92" Sep 09 14:39:06.709019 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:06.708984 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-xtkrm" event={"ID":"342ca2d8-8fa0-45b6-9aaf-4bcf7117b021","Type":"ContainerStarted","Data":"0b1b9b2d4643485e6bd51c27f2f8fdda75904ebbad1bf70a5fc901d18d72b9c9"} Sep 09 14:39:19.388873 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:19.388839 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab36c1e4-a65c-4cf9-ada4-f8a87d4b463b-metrics-certs\") pod \"network-metrics-daemon-wcj2p\" (UID: \"ab36c1e4-a65c-4cf9-ada4-f8a87d4b463b\") " pod="openshift-multus/network-metrics-daemon-wcj2p" Sep 09 14:39:19.391212 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:19.391194 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab36c1e4-a65c-4cf9-ada4-f8a87d4b463b-metrics-certs\") pod \"network-metrics-daemon-wcj2p\" (UID: \"ab36c1e4-a65c-4cf9-ada4-f8a87d4b463b\") " pod="openshift-multus/network-metrics-daemon-wcj2p" Sep 09 14:39:19.513474 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:19.513453 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-bjh9b\"" Sep 09 14:39:19.520679 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:19.520663 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wcj2p" Sep 09 14:39:19.791968 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:19.791943 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-wcj2p"] Sep 09 14:39:19.794338 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:39:19.794285 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab36c1e4_a65c_4cf9_ada4_f8a87d4b463b.slice/crio-aaca858c836768c1cfbfefd5524a0438583a524a791e4b33b89901fd41d64b79 WatchSource:0}: Error finding container aaca858c836768c1cfbfefd5524a0438583a524a791e4b33b89901fd41d64b79: Status 404 returned error can't find the container with id aaca858c836768c1cfbfefd5524a0438583a524a791e4b33b89901fd41d64b79 Sep 09 14:39:20.768193 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:20.768154 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wcj2p" event={"ID":"ab36c1e4-a65c-4cf9-ada4-f8a87d4b463b","Type":"ContainerStarted","Data":"aaca858c836768c1cfbfefd5524a0438583a524a791e4b33b89901fd41d64b79"} Sep 09 14:39:21.257209 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:21.257179 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Sep 09 14:39:21.277447 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:21.277423 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Sep 09 14:39:21.773408 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:21.773374 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wcj2p" event={"ID":"ab36c1e4-a65c-4cf9-ada4-f8a87d4b463b","Type":"ContainerStarted","Data":"f8147268980c4fc4d8a101b4ca482413cae22f088c707e8311d5e0a504b5d13b"} Sep 09 14:39:21.773810 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:21.773430 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wcj2p" event={"ID":"ab36c1e4-a65c-4cf9-ada4-f8a87d4b463b","Type":"ContainerStarted","Data":"d092b26333bb502143ac30d91fdbb1448db1091d077355319bdc1e23f984c6ab"} Sep 09 14:39:21.788813 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:21.788788 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 09 14:39:21.791219 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:21.791173 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-wcj2p" podStartSLOduration=375.870727509 podStartE2EDuration="6m16.791158309s" totalImagesPullingTime="920.4308ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 14:33:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:39:21.789598835 +0000 UTC m=+376.800515099" watchObservedRunningTime="2026-09-09 14:39:21.791158309 +0000 UTC m=+376.802074559" Sep 09 14:39:35.283795 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:35.283751 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 09 14:39:35.284294 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:35.284032 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6" containerName="alertmanager" containerID="cri-o://d5a1f20e62599e56b16eb2a1dfe1a08b5d9cc99636ea81680ad422195f4d5ae3" gracePeriod=120 Sep 09 14:39:35.284294 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:35.284110 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6" containerName="kube-rbac-proxy-metric" containerID="cri-o://4c45967013db8112c49277a4660228c14fe79b1bbdfe23f1cd734e3f8f8bc58c" gracePeriod=120 Sep 09 14:39:35.284294 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:35.284126 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6" containerName="prom-label-proxy" containerID="cri-o://1044d18d11fec3692265db487d33daf37b9a23183052d9ed61f78d20da34a5dd" gracePeriod=120 Sep 09 14:39:35.284294 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:35.284155 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6" containerName="kube-rbac-proxy-web" containerID="cri-o://512a509962840a16e1e592d45fe921461276455b6af9d358b267d853d0185a8a" gracePeriod=120 Sep 09 14:39:35.284294 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:35.284172 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6" containerName="config-reloader" containerID="cri-o://7703bb5317e77b1c338f67fafd294eed049e16d243da6283bc495505411c8b4a" gracePeriod=120 Sep 09 14:39:35.284294 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:35.284248 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6" containerName="kube-rbac-proxy" containerID="cri-o://6f9c8317538f6f89e092f9c38eaf065ad3d7d6d0b3043c8b1588c9ba8c5ee11f" gracePeriod=120 Sep 09 14:39:35.831175 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:35.831138 2576 generic.go:356] "Generic (PLEG): container finished" podID="cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6" containerID="1044d18d11fec3692265db487d33daf37b9a23183052d9ed61f78d20da34a5dd" exitCode=0 Sep 09 14:39:35.831175 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:35.831162 2576 generic.go:356] "Generic (PLEG): container finished" podID="cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6" containerID="6f9c8317538f6f89e092f9c38eaf065ad3d7d6d0b3043c8b1588c9ba8c5ee11f" exitCode=0 Sep 09 14:39:35.831175 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:35.831168 2576 generic.go:356] "Generic (PLEG): container finished" podID="cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6" containerID="7703bb5317e77b1c338f67fafd294eed049e16d243da6283bc495505411c8b4a" exitCode=0 Sep 09 14:39:35.831175 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:35.831173 2576 generic.go:356] "Generic (PLEG): container finished" podID="cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6" containerID="d5a1f20e62599e56b16eb2a1dfe1a08b5d9cc99636ea81680ad422195f4d5ae3" exitCode=0 Sep 09 14:39:35.831460 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:35.831207 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6","Type":"ContainerDied","Data":"1044d18d11fec3692265db487d33daf37b9a23183052d9ed61f78d20da34a5dd"} Sep 09 14:39:35.831460 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:35.831268 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6","Type":"ContainerDied","Data":"6f9c8317538f6f89e092f9c38eaf065ad3d7d6d0b3043c8b1588c9ba8c5ee11f"} Sep 09 14:39:35.831460 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:35.831294 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6","Type":"ContainerDied","Data":"7703bb5317e77b1c338f67fafd294eed049e16d243da6283bc495505411c8b4a"} Sep 09 14:39:35.831460 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:35.831335 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6","Type":"ContainerDied","Data":"d5a1f20e62599e56b16eb2a1dfe1a08b5d9cc99636ea81680ad422195f4d5ae3"} Sep 09 14:39:36.533349 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:36.533326 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 09 14:39:36.637884 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:36.637810 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6-web-config\") pod \"cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6\" (UID: \"cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6\") " Sep 09 14:39:36.637884 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:36.637850 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6-metrics-client-ca\") pod \"cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6\" (UID: \"cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6\") " Sep 09 14:39:36.637884 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:36.637873 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6-cluster-tls-config\") pod \"cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6\" (UID: \"cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6\") " Sep 09 14:39:36.638141 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:36.637892 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6-secret-alertmanager-kube-rbac-proxy\") pod \"cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6\" (UID: \"cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6\") " Sep 09 14:39:36.638141 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:36.637928 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p7nzn\" (UniqueName: \"kubernetes.io/projected/cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6-kube-api-access-p7nzn\") pod \"cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6\" (UID: \"cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6\") " Sep 09 14:39:36.638141 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:36.637957 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6-tls-assets\") pod \"cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6\" (UID: \"cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6\") " Sep 09 14:39:36.638141 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:36.637985 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6-config-out\") pod \"cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6\" (UID: \"cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6\") " Sep 09 14:39:36.638141 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:36.638016 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6-secret-alertmanager-main-tls\") pod \"cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6\" (UID: \"cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6\") " Sep 09 14:39:36.638141 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:36.638048 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6-alertmanager-main-db\") pod \"cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6\" (UID: \"cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6\") " Sep 09 14:39:36.638141 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:36.638071 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6-config-volume\") pod \"cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6\" (UID: \"cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6\") " Sep 09 14:39:36.638141 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:36.638139 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6-secret-alertmanager-kube-rbac-proxy-metric\") pod \"cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6\" (UID: \"cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6\") " Sep 09 14:39:36.638596 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:36.638177 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6-secret-alertmanager-kube-rbac-proxy-web\") pod \"cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6\" (UID: \"cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6\") " Sep 09 14:39:36.638596 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:36.638274 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6" (UID: "cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 14:39:36.638596 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:36.638365 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6-alertmanager-trusted-ca-bundle\") pod \"cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6\" (UID: \"cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6\") " Sep 09 14:39:36.638742 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:36.638676 2576 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6-metrics-client-ca\") on node \"ip-10-0-133-66.ec2.internal\" DevicePath \"\"" Sep 09 14:39:36.639547 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:36.639057 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6" (UID: "cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 14:39:36.641332 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:36.641285 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6" (UID: "cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 14:39:36.642475 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:36.642436 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6" (UID: "cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 14:39:36.642596 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:36.642561 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6-config-volume" (OuterVolumeSpecName: "config-volume") pod "cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6" (UID: "cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 14:39:36.642665 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:36.642591 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6" (UID: "cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 14:39:36.642822 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:36.642795 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6-kube-api-access-p7nzn" (OuterVolumeSpecName: "kube-api-access-p7nzn") pod "cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6" (UID: "cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6"). InnerVolumeSpecName "kube-api-access-p7nzn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 14:39:36.643065 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:36.643015 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6" (UID: "cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 14:39:36.643065 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:36.643031 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6" (UID: "cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 14:39:36.643609 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:36.643574 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6" (UID: "cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 14:39:36.644023 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:36.644004 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6-config-out" (OuterVolumeSpecName: "config-out") pod "cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6" (UID: "cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 14:39:36.646840 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:36.646745 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6" (UID: "cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 14:39:36.653197 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:36.653171 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6-web-config" (OuterVolumeSpecName: "web-config") pod "cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6" (UID: "cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 14:39:36.739656 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:36.739613 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7nzn\" (UniqueName: \"kubernetes.io/projected/cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6-kube-api-access-p7nzn\") on node \"ip-10-0-133-66.ec2.internal\" DevicePath \"\"" Sep 09 14:39:36.739656 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:36.739653 2576 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6-tls-assets\") on node \"ip-10-0-133-66.ec2.internal\" DevicePath \"\"" Sep 09 14:39:36.739656 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:36.739665 2576 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6-config-out\") on node \"ip-10-0-133-66.ec2.internal\" DevicePath \"\"" Sep 09 14:39:36.739911 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:36.739679 2576 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6-secret-alertmanager-main-tls\") on node \"ip-10-0-133-66.ec2.internal\" DevicePath \"\"" Sep 09 14:39:36.739911 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:36.739691 2576 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6-alertmanager-main-db\") on node \"ip-10-0-133-66.ec2.internal\" DevicePath \"\"" Sep 09 14:39:36.739911 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:36.739702 2576 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6-config-volume\") on node \"ip-10-0-133-66.ec2.internal\" DevicePath \"\"" Sep 09 14:39:36.739911 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:36.739714 2576 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-133-66.ec2.internal\" DevicePath \"\"" Sep 09 14:39:36.739911 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:36.739725 2576 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-133-66.ec2.internal\" DevicePath \"\"" Sep 09 14:39:36.739911 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:36.739740 2576 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-133-66.ec2.internal\" DevicePath \"\"" Sep 09 14:39:36.739911 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:36.739753 2576 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6-web-config\") on node \"ip-10-0-133-66.ec2.internal\" DevicePath \"\"" Sep 09 14:39:36.739911 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:36.739763 2576 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6-cluster-tls-config\") on node \"ip-10-0-133-66.ec2.internal\" DevicePath \"\"" Sep 09 14:39:36.739911 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:36.739774 2576 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-133-66.ec2.internal\" DevicePath \"\"" Sep 09 14:39:36.838066 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:36.838034 2576 generic.go:356] "Generic (PLEG): container finished" podID="cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6" containerID="4c45967013db8112c49277a4660228c14fe79b1bbdfe23f1cd734e3f8f8bc58c" exitCode=0 Sep 09 14:39:36.838066 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:36.838059 2576 generic.go:356] "Generic (PLEG): container finished" podID="cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6" containerID="512a509962840a16e1e592d45fe921461276455b6af9d358b267d853d0185a8a" exitCode=0 Sep 09 14:39:36.838258 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:36.838100 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6","Type":"ContainerDied","Data":"4c45967013db8112c49277a4660228c14fe79b1bbdfe23f1cd734e3f8f8bc58c"} Sep 09 14:39:36.838258 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:36.838119 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 09 14:39:36.838258 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:36.838178 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6","Type":"ContainerDied","Data":"512a509962840a16e1e592d45fe921461276455b6af9d358b267d853d0185a8a"} Sep 09 14:39:36.838258 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:36.838216 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6","Type":"ContainerDied","Data":"132f7c24b91161ea8453e718513572c3c9c2569085223cee4b66a56a3f2559ab"} Sep 09 14:39:36.838258 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:36.838251 2576 scope.go:117] "RemoveContainer" containerID="1044d18d11fec3692265db487d33daf37b9a23183052d9ed61f78d20da34a5dd" Sep 09 14:39:36.846264 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:36.846246 2576 scope.go:117] "RemoveContainer" containerID="4c45967013db8112c49277a4660228c14fe79b1bbdfe23f1cd734e3f8f8bc58c" Sep 09 14:39:36.853057 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:36.853041 2576 scope.go:117] "RemoveContainer" containerID="6f9c8317538f6f89e092f9c38eaf065ad3d7d6d0b3043c8b1588c9ba8c5ee11f" Sep 09 14:39:36.859543 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:36.859528 2576 scope.go:117] "RemoveContainer" containerID="512a509962840a16e1e592d45fe921461276455b6af9d358b267d853d0185a8a" Sep 09 14:39:36.866778 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:36.866752 2576 scope.go:117] "RemoveContainer" containerID="7703bb5317e77b1c338f67fafd294eed049e16d243da6283bc495505411c8b4a" Sep 09 14:39:36.870535 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:36.870507 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 09 14:39:36.872324 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:36.872283 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 09 14:39:36.874773 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:36.874758 2576 scope.go:117] "RemoveContainer" containerID="d5a1f20e62599e56b16eb2a1dfe1a08b5d9cc99636ea81680ad422195f4d5ae3" Sep 09 14:39:36.881605 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:36.881582 2576 scope.go:117] "RemoveContainer" containerID="98f881fe8cab49fe2f05e1a797ce1ae9a37febca53484cc1e0dfa7b19c34bf90" Sep 09 14:39:36.888437 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:36.888419 2576 scope.go:117] "RemoveContainer" containerID="1044d18d11fec3692265db487d33daf37b9a23183052d9ed61f78d20da34a5dd" Sep 09 14:39:36.888693 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:39:36.888675 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1044d18d11fec3692265db487d33daf37b9a23183052d9ed61f78d20da34a5dd\": container with ID starting with 1044d18d11fec3692265db487d33daf37b9a23183052d9ed61f78d20da34a5dd not found: ID does not exist" containerID="1044d18d11fec3692265db487d33daf37b9a23183052d9ed61f78d20da34a5dd" Sep 09 14:39:36.888762 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:36.888718 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1044d18d11fec3692265db487d33daf37b9a23183052d9ed61f78d20da34a5dd"} err="failed to get container status \"1044d18d11fec3692265db487d33daf37b9a23183052d9ed61f78d20da34a5dd\": rpc error: code = NotFound desc = could not find container \"1044d18d11fec3692265db487d33daf37b9a23183052d9ed61f78d20da34a5dd\": container with ID starting with 1044d18d11fec3692265db487d33daf37b9a23183052d9ed61f78d20da34a5dd not found: ID does not exist" Sep 09 14:39:36.888762 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:36.888747 2576 scope.go:117] "RemoveContainer" containerID="4c45967013db8112c49277a4660228c14fe79b1bbdfe23f1cd734e3f8f8bc58c" Sep 09 14:39:36.889023 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:39:36.889007 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c45967013db8112c49277a4660228c14fe79b1bbdfe23f1cd734e3f8f8bc58c\": container with ID starting with 4c45967013db8112c49277a4660228c14fe79b1bbdfe23f1cd734e3f8f8bc58c not found: ID does not exist" containerID="4c45967013db8112c49277a4660228c14fe79b1bbdfe23f1cd734e3f8f8bc58c" Sep 09 14:39:36.889068 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:36.889030 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c45967013db8112c49277a4660228c14fe79b1bbdfe23f1cd734e3f8f8bc58c"} err="failed to get container status \"4c45967013db8112c49277a4660228c14fe79b1bbdfe23f1cd734e3f8f8bc58c\": rpc error: code = NotFound desc = could not find container \"4c45967013db8112c49277a4660228c14fe79b1bbdfe23f1cd734e3f8f8bc58c\": container with ID starting with 4c45967013db8112c49277a4660228c14fe79b1bbdfe23f1cd734e3f8f8bc58c not found: ID does not exist" Sep 09 14:39:36.889068 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:36.889045 2576 scope.go:117] "RemoveContainer" containerID="6f9c8317538f6f89e092f9c38eaf065ad3d7d6d0b3043c8b1588c9ba8c5ee11f" Sep 09 14:39:36.889275 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:39:36.889258 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f9c8317538f6f89e092f9c38eaf065ad3d7d6d0b3043c8b1588c9ba8c5ee11f\": container with ID starting with 6f9c8317538f6f89e092f9c38eaf065ad3d7d6d0b3043c8b1588c9ba8c5ee11f not found: ID does not exist" containerID="6f9c8317538f6f89e092f9c38eaf065ad3d7d6d0b3043c8b1588c9ba8c5ee11f" Sep 09 14:39:36.889345 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:36.889280 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f9c8317538f6f89e092f9c38eaf065ad3d7d6d0b3043c8b1588c9ba8c5ee11f"} err="failed to get container status \"6f9c8317538f6f89e092f9c38eaf065ad3d7d6d0b3043c8b1588c9ba8c5ee11f\": rpc error: code = NotFound desc = could not find container \"6f9c8317538f6f89e092f9c38eaf065ad3d7d6d0b3043c8b1588c9ba8c5ee11f\": container with ID starting with 6f9c8317538f6f89e092f9c38eaf065ad3d7d6d0b3043c8b1588c9ba8c5ee11f not found: ID does not exist" Sep 09 14:39:36.889345 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:36.889294 2576 scope.go:117] "RemoveContainer" containerID="512a509962840a16e1e592d45fe921461276455b6af9d358b267d853d0185a8a" Sep 09 14:39:36.889534 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:39:36.889519 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"512a509962840a16e1e592d45fe921461276455b6af9d358b267d853d0185a8a\": container with ID starting with 512a509962840a16e1e592d45fe921461276455b6af9d358b267d853d0185a8a not found: ID does not exist" containerID="512a509962840a16e1e592d45fe921461276455b6af9d358b267d853d0185a8a" Sep 09 14:39:36.889573 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:36.889539 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"512a509962840a16e1e592d45fe921461276455b6af9d358b267d853d0185a8a"} err="failed to get container status \"512a509962840a16e1e592d45fe921461276455b6af9d358b267d853d0185a8a\": rpc error: code = NotFound desc = could not find container \"512a509962840a16e1e592d45fe921461276455b6af9d358b267d853d0185a8a\": container with ID starting with 512a509962840a16e1e592d45fe921461276455b6af9d358b267d853d0185a8a not found: ID does not exist" Sep 09 14:39:36.889573 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:36.889551 2576 scope.go:117] "RemoveContainer" containerID="7703bb5317e77b1c338f67fafd294eed049e16d243da6283bc495505411c8b4a" Sep 09 14:39:36.889778 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:39:36.889763 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7703bb5317e77b1c338f67fafd294eed049e16d243da6283bc495505411c8b4a\": container with ID starting with 7703bb5317e77b1c338f67fafd294eed049e16d243da6283bc495505411c8b4a not found: ID does not exist" containerID="7703bb5317e77b1c338f67fafd294eed049e16d243da6283bc495505411c8b4a" Sep 09 14:39:36.889821 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:36.889791 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7703bb5317e77b1c338f67fafd294eed049e16d243da6283bc495505411c8b4a"} err="failed to get container status \"7703bb5317e77b1c338f67fafd294eed049e16d243da6283bc495505411c8b4a\": rpc error: code = NotFound desc = could not find container \"7703bb5317e77b1c338f67fafd294eed049e16d243da6283bc495505411c8b4a\": container with ID starting with 7703bb5317e77b1c338f67fafd294eed049e16d243da6283bc495505411c8b4a not found: ID does not exist" Sep 09 14:39:36.889821 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:36.889805 2576 scope.go:117] "RemoveContainer" containerID="d5a1f20e62599e56b16eb2a1dfe1a08b5d9cc99636ea81680ad422195f4d5ae3" Sep 09 14:39:36.890053 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:39:36.890037 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5a1f20e62599e56b16eb2a1dfe1a08b5d9cc99636ea81680ad422195f4d5ae3\": container with ID starting with d5a1f20e62599e56b16eb2a1dfe1a08b5d9cc99636ea81680ad422195f4d5ae3 not found: ID does not exist" containerID="d5a1f20e62599e56b16eb2a1dfe1a08b5d9cc99636ea81680ad422195f4d5ae3" Sep 09 14:39:36.890096 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:36.890056 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5a1f20e62599e56b16eb2a1dfe1a08b5d9cc99636ea81680ad422195f4d5ae3"} err="failed to get container status \"d5a1f20e62599e56b16eb2a1dfe1a08b5d9cc99636ea81680ad422195f4d5ae3\": rpc error: code = NotFound desc = could not find container \"d5a1f20e62599e56b16eb2a1dfe1a08b5d9cc99636ea81680ad422195f4d5ae3\": container with ID starting with d5a1f20e62599e56b16eb2a1dfe1a08b5d9cc99636ea81680ad422195f4d5ae3 not found: ID does not exist" Sep 09 14:39:36.890096 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:36.890068 2576 scope.go:117] "RemoveContainer" containerID="98f881fe8cab49fe2f05e1a797ce1ae9a37febca53484cc1e0dfa7b19c34bf90" Sep 09 14:39:36.890339 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:39:36.890324 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98f881fe8cab49fe2f05e1a797ce1ae9a37febca53484cc1e0dfa7b19c34bf90\": container with ID starting with 98f881fe8cab49fe2f05e1a797ce1ae9a37febca53484cc1e0dfa7b19c34bf90 not found: ID does not exist" containerID="98f881fe8cab49fe2f05e1a797ce1ae9a37febca53484cc1e0dfa7b19c34bf90" Sep 09 14:39:36.890383 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:36.890343 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98f881fe8cab49fe2f05e1a797ce1ae9a37febca53484cc1e0dfa7b19c34bf90"} err="failed to get container status \"98f881fe8cab49fe2f05e1a797ce1ae9a37febca53484cc1e0dfa7b19c34bf90\": rpc error: code = NotFound desc = could not find container \"98f881fe8cab49fe2f05e1a797ce1ae9a37febca53484cc1e0dfa7b19c34bf90\": container with ID starting with 98f881fe8cab49fe2f05e1a797ce1ae9a37febca53484cc1e0dfa7b19c34bf90 not found: ID does not exist" Sep 09 14:39:36.890383 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:36.890356 2576 scope.go:117] "RemoveContainer" containerID="1044d18d11fec3692265db487d33daf37b9a23183052d9ed61f78d20da34a5dd" Sep 09 14:39:36.890592 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:36.890574 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1044d18d11fec3692265db487d33daf37b9a23183052d9ed61f78d20da34a5dd"} err="failed to get container status \"1044d18d11fec3692265db487d33daf37b9a23183052d9ed61f78d20da34a5dd\": rpc error: code = NotFound desc = could not find container \"1044d18d11fec3692265db487d33daf37b9a23183052d9ed61f78d20da34a5dd\": container with ID starting with 1044d18d11fec3692265db487d33daf37b9a23183052d9ed61f78d20da34a5dd not found: ID does not exist" Sep 09 14:39:36.890632 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:36.890593 2576 scope.go:117] "RemoveContainer" containerID="4c45967013db8112c49277a4660228c14fe79b1bbdfe23f1cd734e3f8f8bc58c" Sep 09 14:39:36.890825 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:36.890804 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c45967013db8112c49277a4660228c14fe79b1bbdfe23f1cd734e3f8f8bc58c"} err="failed to get container status \"4c45967013db8112c49277a4660228c14fe79b1bbdfe23f1cd734e3f8f8bc58c\": rpc error: code = NotFound desc = could not find container \"4c45967013db8112c49277a4660228c14fe79b1bbdfe23f1cd734e3f8f8bc58c\": container with ID starting with 4c45967013db8112c49277a4660228c14fe79b1bbdfe23f1cd734e3f8f8bc58c not found: ID does not exist" Sep 09 14:39:36.890869 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:36.890827 2576 scope.go:117] "RemoveContainer" containerID="6f9c8317538f6f89e092f9c38eaf065ad3d7d6d0b3043c8b1588c9ba8c5ee11f" Sep 09 14:39:36.891033 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:36.891016 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f9c8317538f6f89e092f9c38eaf065ad3d7d6d0b3043c8b1588c9ba8c5ee11f"} err="failed to get container status \"6f9c8317538f6f89e092f9c38eaf065ad3d7d6d0b3043c8b1588c9ba8c5ee11f\": rpc error: code = NotFound desc = could not find container \"6f9c8317538f6f89e092f9c38eaf065ad3d7d6d0b3043c8b1588c9ba8c5ee11f\": container with ID starting with 6f9c8317538f6f89e092f9c38eaf065ad3d7d6d0b3043c8b1588c9ba8c5ee11f not found: ID does not exist" Sep 09 14:39:36.891074 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:36.891033 2576 scope.go:117] "RemoveContainer" containerID="512a509962840a16e1e592d45fe921461276455b6af9d358b267d853d0185a8a" Sep 09 14:39:36.891230 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:36.891215 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"512a509962840a16e1e592d45fe921461276455b6af9d358b267d853d0185a8a"} err="failed to get container status \"512a509962840a16e1e592d45fe921461276455b6af9d358b267d853d0185a8a\": rpc error: code = NotFound desc = could not find container \"512a509962840a16e1e592d45fe921461276455b6af9d358b267d853d0185a8a\": container with ID starting with 512a509962840a16e1e592d45fe921461276455b6af9d358b267d853d0185a8a not found: ID does not exist" Sep 09 14:39:36.891271 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:36.891230 2576 scope.go:117] "RemoveContainer" containerID="7703bb5317e77b1c338f67fafd294eed049e16d243da6283bc495505411c8b4a" Sep 09 14:39:36.891473 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:36.891457 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7703bb5317e77b1c338f67fafd294eed049e16d243da6283bc495505411c8b4a"} err="failed to get container status \"7703bb5317e77b1c338f67fafd294eed049e16d243da6283bc495505411c8b4a\": rpc error: code = NotFound desc = could not find container \"7703bb5317e77b1c338f67fafd294eed049e16d243da6283bc495505411c8b4a\": container with ID starting with 7703bb5317e77b1c338f67fafd294eed049e16d243da6283bc495505411c8b4a not found: ID does not exist" Sep 09 14:39:36.891517 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:36.891473 2576 scope.go:117] "RemoveContainer" containerID="d5a1f20e62599e56b16eb2a1dfe1a08b5d9cc99636ea81680ad422195f4d5ae3" Sep 09 14:39:36.891709 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:36.891692 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5a1f20e62599e56b16eb2a1dfe1a08b5d9cc99636ea81680ad422195f4d5ae3"} err="failed to get container status \"d5a1f20e62599e56b16eb2a1dfe1a08b5d9cc99636ea81680ad422195f4d5ae3\": rpc error: code = NotFound desc = could not find container \"d5a1f20e62599e56b16eb2a1dfe1a08b5d9cc99636ea81680ad422195f4d5ae3\": container with ID starting with d5a1f20e62599e56b16eb2a1dfe1a08b5d9cc99636ea81680ad422195f4d5ae3 not found: ID does not exist" Sep 09 14:39:36.891748 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:36.891709 2576 scope.go:117] "RemoveContainer" containerID="98f881fe8cab49fe2f05e1a797ce1ae9a37febca53484cc1e0dfa7b19c34bf90" Sep 09 14:39:36.891909 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:36.891892 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98f881fe8cab49fe2f05e1a797ce1ae9a37febca53484cc1e0dfa7b19c34bf90"} err="failed to get container status \"98f881fe8cab49fe2f05e1a797ce1ae9a37febca53484cc1e0dfa7b19c34bf90\": rpc error: code = NotFound desc = could not find container \"98f881fe8cab49fe2f05e1a797ce1ae9a37febca53484cc1e0dfa7b19c34bf90\": container with ID starting with 98f881fe8cab49fe2f05e1a797ce1ae9a37febca53484cc1e0dfa7b19c34bf90 not found: ID does not exist" Sep 09 14:39:37.515161 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:37.515131 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6" path="/var/lib/kubelet/pods/cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6/volumes" Sep 09 14:39:39.550652 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.550620 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 09 14:39:39.551083 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.550881 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="5a424647-68c0-4d35-87e8-496494c0b8f0" containerName="prometheus" containerID="cri-o://43d1404b45d1b5b7d2c6e4480bc5808400fb3f6ed1d9c08b4219598103cd6159" gracePeriod=600 Sep 09 14:39:39.551083 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.550884 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="5a424647-68c0-4d35-87e8-496494c0b8f0" containerName="kube-rbac-proxy" containerID="cri-o://abe29843fa4a9ed94df2de59408d6645211f313e98636b037e11dae09c781ff9" gracePeriod=600 Sep 09 14:39:39.551083 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.550962 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="5a424647-68c0-4d35-87e8-496494c0b8f0" containerName="config-reloader" containerID="cri-o://2621fdbf311cc0e64590b9500d3a3d05efe0dc2de7110165fe20a5d87b4808b7" gracePeriod=600 Sep 09 14:39:39.551083 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.550978 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="5a424647-68c0-4d35-87e8-496494c0b8f0" containerName="kube-rbac-proxy-web" containerID="cri-o://3aa2add971bc20e83acb8526e058a06ca6bcdccf4df036920dd19ad361ffa5e4" gracePeriod=600 Sep 09 14:39:39.551083 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.550963 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="5a424647-68c0-4d35-87e8-496494c0b8f0" containerName="kube-rbac-proxy-thanos" containerID="cri-o://b20fe58fae1d2c347b64af3a0db85da556a11cbd87377af08dadda0d7c76abe1" gracePeriod=600 Sep 09 14:39:39.551083 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.550977 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="5a424647-68c0-4d35-87e8-496494c0b8f0" containerName="thanos-sidecar" containerID="cri-o://413b924324e8ddab89a80df797d617dc6dce0133b6c7d79bc03bb66adaa08055" gracePeriod=600 Sep 09 14:39:39.787086 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.787062 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 09 14:39:39.854461 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.854367 2576 generic.go:356] "Generic (PLEG): container finished" podID="5a424647-68c0-4d35-87e8-496494c0b8f0" containerID="b20fe58fae1d2c347b64af3a0db85da556a11cbd87377af08dadda0d7c76abe1" exitCode=0 Sep 09 14:39:39.854461 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.854396 2576 generic.go:356] "Generic (PLEG): container finished" podID="5a424647-68c0-4d35-87e8-496494c0b8f0" containerID="abe29843fa4a9ed94df2de59408d6645211f313e98636b037e11dae09c781ff9" exitCode=0 Sep 09 14:39:39.854461 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.854405 2576 generic.go:356] "Generic (PLEG): container finished" podID="5a424647-68c0-4d35-87e8-496494c0b8f0" containerID="3aa2add971bc20e83acb8526e058a06ca6bcdccf4df036920dd19ad361ffa5e4" exitCode=0 Sep 09 14:39:39.854461 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.854413 2576 generic.go:356] "Generic (PLEG): container finished" podID="5a424647-68c0-4d35-87e8-496494c0b8f0" containerID="413b924324e8ddab89a80df797d617dc6dce0133b6c7d79bc03bb66adaa08055" exitCode=0 Sep 09 14:39:39.854461 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.854421 2576 generic.go:356] "Generic (PLEG): container finished" podID="5a424647-68c0-4d35-87e8-496494c0b8f0" containerID="2621fdbf311cc0e64590b9500d3a3d05efe0dc2de7110165fe20a5d87b4808b7" exitCode=0 Sep 09 14:39:39.854461 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.854429 2576 generic.go:356] "Generic (PLEG): container finished" podID="5a424647-68c0-4d35-87e8-496494c0b8f0" containerID="43d1404b45d1b5b7d2c6e4480bc5808400fb3f6ed1d9c08b4219598103cd6159" exitCode=0 Sep 09 14:39:39.854461 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.854443 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5a424647-68c0-4d35-87e8-496494c0b8f0","Type":"ContainerDied","Data":"b20fe58fae1d2c347b64af3a0db85da556a11cbd87377af08dadda0d7c76abe1"} Sep 09 14:39:39.854461 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.854461 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 09 14:39:39.854975 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.854512 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5a424647-68c0-4d35-87e8-496494c0b8f0","Type":"ContainerDied","Data":"abe29843fa4a9ed94df2de59408d6645211f313e98636b037e11dae09c781ff9"} Sep 09 14:39:39.854975 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.854541 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5a424647-68c0-4d35-87e8-496494c0b8f0","Type":"ContainerDied","Data":"3aa2add971bc20e83acb8526e058a06ca6bcdccf4df036920dd19ad361ffa5e4"} Sep 09 14:39:39.854975 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.854567 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5a424647-68c0-4d35-87e8-496494c0b8f0","Type":"ContainerDied","Data":"413b924324e8ddab89a80df797d617dc6dce0133b6c7d79bc03bb66adaa08055"} Sep 09 14:39:39.854975 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.854584 2576 scope.go:117] "RemoveContainer" containerID="b20fe58fae1d2c347b64af3a0db85da556a11cbd87377af08dadda0d7c76abe1" Sep 09 14:39:39.854975 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.854597 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5a424647-68c0-4d35-87e8-496494c0b8f0","Type":"ContainerDied","Data":"2621fdbf311cc0e64590b9500d3a3d05efe0dc2de7110165fe20a5d87b4808b7"} Sep 09 14:39:39.854975 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.854624 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5a424647-68c0-4d35-87e8-496494c0b8f0","Type":"ContainerDied","Data":"43d1404b45d1b5b7d2c6e4480bc5808400fb3f6ed1d9c08b4219598103cd6159"} Sep 09 14:39:39.854975 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.854653 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5a424647-68c0-4d35-87e8-496494c0b8f0","Type":"ContainerDied","Data":"5667b4bcc993461106a20c2e2d7c13f4394211276ee3517d7ac748bdb94d8ead"} Sep 09 14:39:39.863151 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.863134 2576 scope.go:117] "RemoveContainer" containerID="abe29843fa4a9ed94df2de59408d6645211f313e98636b037e11dae09c781ff9" Sep 09 14:39:39.864101 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.864057 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5a424647-68c0-4d35-87e8-496494c0b8f0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"5a424647-68c0-4d35-87e8-496494c0b8f0\" (UID: \"5a424647-68c0-4d35-87e8-496494c0b8f0\") " Sep 09 14:39:39.864101 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.864084 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5a424647-68c0-4d35-87e8-496494c0b8f0-prometheus-k8s-rulefiles-1\") pod \"5a424647-68c0-4d35-87e8-496494c0b8f0\" (UID: \"5a424647-68c0-4d35-87e8-496494c0b8f0\") " Sep 09 14:39:39.864194 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.864107 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5a424647-68c0-4d35-87e8-496494c0b8f0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"5a424647-68c0-4d35-87e8-496494c0b8f0\" (UID: \"5a424647-68c0-4d35-87e8-496494c0b8f0\") " Sep 09 14:39:39.864194 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.864135 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a424647-68c0-4d35-87e8-496494c0b8f0-configmap-serving-certs-ca-bundle\") pod \"5a424647-68c0-4d35-87e8-496494c0b8f0\" (UID: \"5a424647-68c0-4d35-87e8-496494c0b8f0\") " Sep 09 14:39:39.864194 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.864170 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5a424647-68c0-4d35-87e8-496494c0b8f0-tls-assets\") pod \"5a424647-68c0-4d35-87e8-496494c0b8f0\" (UID: \"5a424647-68c0-4d35-87e8-496494c0b8f0\") " Sep 09 14:39:39.864365 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.864196 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5a424647-68c0-4d35-87e8-496494c0b8f0-config-out\") pod \"5a424647-68c0-4d35-87e8-496494c0b8f0\" (UID: \"5a424647-68c0-4d35-87e8-496494c0b8f0\") " Sep 09 14:39:39.864365 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.864237 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5a424647-68c0-4d35-87e8-496494c0b8f0-thanos-prometheus-http-client-file\") pod \"5a424647-68c0-4d35-87e8-496494c0b8f0\" (UID: \"5a424647-68c0-4d35-87e8-496494c0b8f0\") " Sep 09 14:39:39.864365 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.864279 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5a424647-68c0-4d35-87e8-496494c0b8f0-prometheus-k8s-rulefiles-0\") pod \"5a424647-68c0-4d35-87e8-496494c0b8f0\" (UID: \"5a424647-68c0-4d35-87e8-496494c0b8f0\") " Sep 09 14:39:39.864365 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.864334 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7qgzb\" (UniqueName: \"kubernetes.io/projected/5a424647-68c0-4d35-87e8-496494c0b8f0-kube-api-access-7qgzb\") pod \"5a424647-68c0-4d35-87e8-496494c0b8f0\" (UID: \"5a424647-68c0-4d35-87e8-496494c0b8f0\") " Sep 09 14:39:39.864365 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.864355 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5a424647-68c0-4d35-87e8-496494c0b8f0-web-config\") pod \"5a424647-68c0-4d35-87e8-496494c0b8f0\" (UID: \"5a424647-68c0-4d35-87e8-496494c0b8f0\") " Sep 09 14:39:39.864775 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.864376 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5a424647-68c0-4d35-87e8-496494c0b8f0-secret-kube-rbac-proxy\") pod \"5a424647-68c0-4d35-87e8-496494c0b8f0\" (UID: \"5a424647-68c0-4d35-87e8-496494c0b8f0\") " Sep 09 14:39:39.864775 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.864400 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5a424647-68c0-4d35-87e8-496494c0b8f0-secret-prometheus-k8s-tls\") pod \"5a424647-68c0-4d35-87e8-496494c0b8f0\" (UID: \"5a424647-68c0-4d35-87e8-496494c0b8f0\") " Sep 09 14:39:39.864775 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.864440 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5a424647-68c0-4d35-87e8-496494c0b8f0-prometheus-k8s-db\") pod \"5a424647-68c0-4d35-87e8-496494c0b8f0\" (UID: \"5a424647-68c0-4d35-87e8-496494c0b8f0\") " Sep 09 14:39:39.864775 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.864471 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a424647-68c0-4d35-87e8-496494c0b8f0-configmap-kubelet-serving-ca-bundle\") pod \"5a424647-68c0-4d35-87e8-496494c0b8f0\" (UID: \"5a424647-68c0-4d35-87e8-496494c0b8f0\") " Sep 09 14:39:39.864775 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.864496 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5a424647-68c0-4d35-87e8-496494c0b8f0-secret-grpc-tls\") pod \"5a424647-68c0-4d35-87e8-496494c0b8f0\" (UID: \"5a424647-68c0-4d35-87e8-496494c0b8f0\") " Sep 09 14:39:39.864775 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.864494 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a424647-68c0-4d35-87e8-496494c0b8f0-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "5a424647-68c0-4d35-87e8-496494c0b8f0" (UID: "5a424647-68c0-4d35-87e8-496494c0b8f0"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 14:39:39.864775 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.864528 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5a424647-68c0-4d35-87e8-496494c0b8f0-prometheus-k8s-rulefiles-2\") pod \"5a424647-68c0-4d35-87e8-496494c0b8f0\" (UID: \"5a424647-68c0-4d35-87e8-496494c0b8f0\") " Sep 09 14:39:39.864775 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.864568 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5a424647-68c0-4d35-87e8-496494c0b8f0-secret-metrics-client-certs\") pod \"5a424647-68c0-4d35-87e8-496494c0b8f0\" (UID: \"5a424647-68c0-4d35-87e8-496494c0b8f0\") " Sep 09 14:39:39.864775 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.864596 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5a424647-68c0-4d35-87e8-496494c0b8f0-configmap-metrics-client-ca\") pod \"5a424647-68c0-4d35-87e8-496494c0b8f0\" (UID: \"5a424647-68c0-4d35-87e8-496494c0b8f0\") " Sep 09 14:39:39.864775 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.864620 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a424647-68c0-4d35-87e8-496494c0b8f0-prometheus-trusted-ca-bundle\") pod \"5a424647-68c0-4d35-87e8-496494c0b8f0\" (UID: \"5a424647-68c0-4d35-87e8-496494c0b8f0\") " Sep 09 14:39:39.864775 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.864649 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/5a424647-68c0-4d35-87e8-496494c0b8f0-config\") pod \"5a424647-68c0-4d35-87e8-496494c0b8f0\" (UID: \"5a424647-68c0-4d35-87e8-496494c0b8f0\") " Sep 09 14:39:39.865327 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.864909 2576 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5a424647-68c0-4d35-87e8-496494c0b8f0-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-133-66.ec2.internal\" DevicePath \"\"" Sep 09 14:39:39.865327 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.865065 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a424647-68c0-4d35-87e8-496494c0b8f0-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "5a424647-68c0-4d35-87e8-496494c0b8f0" (UID: "5a424647-68c0-4d35-87e8-496494c0b8f0"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 14:39:39.867653 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.865446 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a424647-68c0-4d35-87e8-496494c0b8f0-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "5a424647-68c0-4d35-87e8-496494c0b8f0" (UID: "5a424647-68c0-4d35-87e8-496494c0b8f0"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 14:39:39.867653 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.866356 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a424647-68c0-4d35-87e8-496494c0b8f0-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "5a424647-68c0-4d35-87e8-496494c0b8f0" (UID: "5a424647-68c0-4d35-87e8-496494c0b8f0"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 14:39:39.867653 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.867019 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a424647-68c0-4d35-87e8-496494c0b8f0-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "5a424647-68c0-4d35-87e8-496494c0b8f0" (UID: "5a424647-68c0-4d35-87e8-496494c0b8f0"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 14:39:39.867653 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.867042 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a424647-68c0-4d35-87e8-496494c0b8f0-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "5a424647-68c0-4d35-87e8-496494c0b8f0" (UID: "5a424647-68c0-4d35-87e8-496494c0b8f0"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 14:39:39.867653 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.867259 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a424647-68c0-4d35-87e8-496494c0b8f0-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "5a424647-68c0-4d35-87e8-496494c0b8f0" (UID: "5a424647-68c0-4d35-87e8-496494c0b8f0"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 14:39:39.867653 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.867620 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a424647-68c0-4d35-87e8-496494c0b8f0-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "5a424647-68c0-4d35-87e8-496494c0b8f0" (UID: "5a424647-68c0-4d35-87e8-496494c0b8f0"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 14:39:39.869818 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.869686 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a424647-68c0-4d35-87e8-496494c0b8f0-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "5a424647-68c0-4d35-87e8-496494c0b8f0" (UID: "5a424647-68c0-4d35-87e8-496494c0b8f0"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 14:39:39.869818 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.869782 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a424647-68c0-4d35-87e8-496494c0b8f0-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "5a424647-68c0-4d35-87e8-496494c0b8f0" (UID: "5a424647-68c0-4d35-87e8-496494c0b8f0"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 14:39:39.870075 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.870028 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a424647-68c0-4d35-87e8-496494c0b8f0-config" (OuterVolumeSpecName: "config") pod "5a424647-68c0-4d35-87e8-496494c0b8f0" (UID: "5a424647-68c0-4d35-87e8-496494c0b8f0"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 14:39:39.870714 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.870256 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a424647-68c0-4d35-87e8-496494c0b8f0-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "5a424647-68c0-4d35-87e8-496494c0b8f0" (UID: "5a424647-68c0-4d35-87e8-496494c0b8f0"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 14:39:39.870714 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.870477 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a424647-68c0-4d35-87e8-496494c0b8f0-kube-api-access-7qgzb" (OuterVolumeSpecName: "kube-api-access-7qgzb") pod "5a424647-68c0-4d35-87e8-496494c0b8f0" (UID: "5a424647-68c0-4d35-87e8-496494c0b8f0"). InnerVolumeSpecName "kube-api-access-7qgzb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 14:39:39.870714 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.870558 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a424647-68c0-4d35-87e8-496494c0b8f0-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "5a424647-68c0-4d35-87e8-496494c0b8f0" (UID: "5a424647-68c0-4d35-87e8-496494c0b8f0"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 14:39:39.870714 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.870668 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a424647-68c0-4d35-87e8-496494c0b8f0-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "5a424647-68c0-4d35-87e8-496494c0b8f0" (UID: "5a424647-68c0-4d35-87e8-496494c0b8f0"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 14:39:39.871026 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.870813 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a424647-68c0-4d35-87e8-496494c0b8f0-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "5a424647-68c0-4d35-87e8-496494c0b8f0" (UID: "5a424647-68c0-4d35-87e8-496494c0b8f0"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 14:39:39.871269 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.871245 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a424647-68c0-4d35-87e8-496494c0b8f0-config-out" (OuterVolumeSpecName: "config-out") pod "5a424647-68c0-4d35-87e8-496494c0b8f0" (UID: "5a424647-68c0-4d35-87e8-496494c0b8f0"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 14:39:39.871990 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.871968 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a424647-68c0-4d35-87e8-496494c0b8f0-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "5a424647-68c0-4d35-87e8-496494c0b8f0" (UID: "5a424647-68c0-4d35-87e8-496494c0b8f0"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 14:39:39.872085 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.872074 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a424647-68c0-4d35-87e8-496494c0b8f0-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "5a424647-68c0-4d35-87e8-496494c0b8f0" (UID: "5a424647-68c0-4d35-87e8-496494c0b8f0"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 14:39:39.873192 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.873123 2576 scope.go:117] "RemoveContainer" containerID="3aa2add971bc20e83acb8526e058a06ca6bcdccf4df036920dd19ad361ffa5e4" Sep 09 14:39:39.881084 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.881068 2576 scope.go:117] "RemoveContainer" containerID="413b924324e8ddab89a80df797d617dc6dce0133b6c7d79bc03bb66adaa08055" Sep 09 14:39:39.881766 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.881745 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a424647-68c0-4d35-87e8-496494c0b8f0-web-config" (OuterVolumeSpecName: "web-config") pod "5a424647-68c0-4d35-87e8-496494c0b8f0" (UID: "5a424647-68c0-4d35-87e8-496494c0b8f0"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 14:39:39.887900 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.887884 2576 scope.go:117] "RemoveContainer" containerID="2621fdbf311cc0e64590b9500d3a3d05efe0dc2de7110165fe20a5d87b4808b7" Sep 09 14:39:39.894544 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.894530 2576 scope.go:117] "RemoveContainer" containerID="43d1404b45d1b5b7d2c6e4480bc5808400fb3f6ed1d9c08b4219598103cd6159" Sep 09 14:39:39.901313 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.901286 2576 scope.go:117] "RemoveContainer" containerID="e7cf15d15c5614d31d998f0a6ade67c50f68137f7411c8d982ca83c0653e7906" Sep 09 14:39:39.907588 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.907573 2576 scope.go:117] "RemoveContainer" containerID="b20fe58fae1d2c347b64af3a0db85da556a11cbd87377af08dadda0d7c76abe1" Sep 09 14:39:39.907834 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:39:39.907817 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b20fe58fae1d2c347b64af3a0db85da556a11cbd87377af08dadda0d7c76abe1\": container with ID starting with b20fe58fae1d2c347b64af3a0db85da556a11cbd87377af08dadda0d7c76abe1 not found: ID does not exist" containerID="b20fe58fae1d2c347b64af3a0db85da556a11cbd87377af08dadda0d7c76abe1" Sep 09 14:39:39.907880 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.907842 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b20fe58fae1d2c347b64af3a0db85da556a11cbd87377af08dadda0d7c76abe1"} err="failed to get container status \"b20fe58fae1d2c347b64af3a0db85da556a11cbd87377af08dadda0d7c76abe1\": rpc error: code = NotFound desc = could not find container \"b20fe58fae1d2c347b64af3a0db85da556a11cbd87377af08dadda0d7c76abe1\": container with ID starting with b20fe58fae1d2c347b64af3a0db85da556a11cbd87377af08dadda0d7c76abe1 not found: ID does not exist" Sep 09 14:39:39.907880 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.907858 2576 scope.go:117] "RemoveContainer" containerID="abe29843fa4a9ed94df2de59408d6645211f313e98636b037e11dae09c781ff9" Sep 09 14:39:39.908105 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:39:39.908089 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abe29843fa4a9ed94df2de59408d6645211f313e98636b037e11dae09c781ff9\": container with ID starting with abe29843fa4a9ed94df2de59408d6645211f313e98636b037e11dae09c781ff9 not found: ID does not exist" containerID="abe29843fa4a9ed94df2de59408d6645211f313e98636b037e11dae09c781ff9" Sep 09 14:39:39.908148 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.908110 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abe29843fa4a9ed94df2de59408d6645211f313e98636b037e11dae09c781ff9"} err="failed to get container status \"abe29843fa4a9ed94df2de59408d6645211f313e98636b037e11dae09c781ff9\": rpc error: code = NotFound desc = could not find container \"abe29843fa4a9ed94df2de59408d6645211f313e98636b037e11dae09c781ff9\": container with ID starting with abe29843fa4a9ed94df2de59408d6645211f313e98636b037e11dae09c781ff9 not found: ID does not exist" Sep 09 14:39:39.908148 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.908125 2576 scope.go:117] "RemoveContainer" containerID="3aa2add971bc20e83acb8526e058a06ca6bcdccf4df036920dd19ad361ffa5e4" Sep 09 14:39:39.908391 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:39:39.908368 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3aa2add971bc20e83acb8526e058a06ca6bcdccf4df036920dd19ad361ffa5e4\": container with ID starting with 3aa2add971bc20e83acb8526e058a06ca6bcdccf4df036920dd19ad361ffa5e4 not found: ID does not exist" containerID="3aa2add971bc20e83acb8526e058a06ca6bcdccf4df036920dd19ad361ffa5e4" Sep 09 14:39:39.908506 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.908398 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3aa2add971bc20e83acb8526e058a06ca6bcdccf4df036920dd19ad361ffa5e4"} err="failed to get container status \"3aa2add971bc20e83acb8526e058a06ca6bcdccf4df036920dd19ad361ffa5e4\": rpc error: code = NotFound desc = could not find container \"3aa2add971bc20e83acb8526e058a06ca6bcdccf4df036920dd19ad361ffa5e4\": container with ID starting with 3aa2add971bc20e83acb8526e058a06ca6bcdccf4df036920dd19ad361ffa5e4 not found: ID does not exist" Sep 09 14:39:39.908506 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.908420 2576 scope.go:117] "RemoveContainer" containerID="413b924324e8ddab89a80df797d617dc6dce0133b6c7d79bc03bb66adaa08055" Sep 09 14:39:39.908653 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:39:39.908639 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"413b924324e8ddab89a80df797d617dc6dce0133b6c7d79bc03bb66adaa08055\": container with ID starting with 413b924324e8ddab89a80df797d617dc6dce0133b6c7d79bc03bb66adaa08055 not found: ID does not exist" containerID="413b924324e8ddab89a80df797d617dc6dce0133b6c7d79bc03bb66adaa08055" Sep 09 14:39:39.908708 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.908657 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"413b924324e8ddab89a80df797d617dc6dce0133b6c7d79bc03bb66adaa08055"} err="failed to get container status \"413b924324e8ddab89a80df797d617dc6dce0133b6c7d79bc03bb66adaa08055\": rpc error: code = NotFound desc = could not find container \"413b924324e8ddab89a80df797d617dc6dce0133b6c7d79bc03bb66adaa08055\": container with ID starting with 413b924324e8ddab89a80df797d617dc6dce0133b6c7d79bc03bb66adaa08055 not found: ID does not exist" Sep 09 14:39:39.908708 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.908669 2576 scope.go:117] "RemoveContainer" containerID="2621fdbf311cc0e64590b9500d3a3d05efe0dc2de7110165fe20a5d87b4808b7" Sep 09 14:39:39.908926 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:39:39.908909 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2621fdbf311cc0e64590b9500d3a3d05efe0dc2de7110165fe20a5d87b4808b7\": container with ID starting with 2621fdbf311cc0e64590b9500d3a3d05efe0dc2de7110165fe20a5d87b4808b7 not found: ID does not exist" containerID="2621fdbf311cc0e64590b9500d3a3d05efe0dc2de7110165fe20a5d87b4808b7" Sep 09 14:39:39.908992 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.908932 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2621fdbf311cc0e64590b9500d3a3d05efe0dc2de7110165fe20a5d87b4808b7"} err="failed to get container status \"2621fdbf311cc0e64590b9500d3a3d05efe0dc2de7110165fe20a5d87b4808b7\": rpc error: code = NotFound desc = could not find container \"2621fdbf311cc0e64590b9500d3a3d05efe0dc2de7110165fe20a5d87b4808b7\": container with ID starting with 2621fdbf311cc0e64590b9500d3a3d05efe0dc2de7110165fe20a5d87b4808b7 not found: ID does not exist" Sep 09 14:39:39.908992 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.908953 2576 scope.go:117] "RemoveContainer" containerID="43d1404b45d1b5b7d2c6e4480bc5808400fb3f6ed1d9c08b4219598103cd6159" Sep 09 14:39:39.909196 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:39:39.909180 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43d1404b45d1b5b7d2c6e4480bc5808400fb3f6ed1d9c08b4219598103cd6159\": container with ID starting with 43d1404b45d1b5b7d2c6e4480bc5808400fb3f6ed1d9c08b4219598103cd6159 not found: ID does not exist" containerID="43d1404b45d1b5b7d2c6e4480bc5808400fb3f6ed1d9c08b4219598103cd6159" Sep 09 14:39:39.909287 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.909199 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43d1404b45d1b5b7d2c6e4480bc5808400fb3f6ed1d9c08b4219598103cd6159"} err="failed to get container status \"43d1404b45d1b5b7d2c6e4480bc5808400fb3f6ed1d9c08b4219598103cd6159\": rpc error: code = NotFound desc = could not find container \"43d1404b45d1b5b7d2c6e4480bc5808400fb3f6ed1d9c08b4219598103cd6159\": container with ID starting with 43d1404b45d1b5b7d2c6e4480bc5808400fb3f6ed1d9c08b4219598103cd6159 not found: ID does not exist" Sep 09 14:39:39.909287 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.909211 2576 scope.go:117] "RemoveContainer" containerID="e7cf15d15c5614d31d998f0a6ade67c50f68137f7411c8d982ca83c0653e7906" Sep 09 14:39:39.909469 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:39:39.909451 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7cf15d15c5614d31d998f0a6ade67c50f68137f7411c8d982ca83c0653e7906\": container with ID starting with e7cf15d15c5614d31d998f0a6ade67c50f68137f7411c8d982ca83c0653e7906 not found: ID does not exist" containerID="e7cf15d15c5614d31d998f0a6ade67c50f68137f7411c8d982ca83c0653e7906" Sep 09 14:39:39.909507 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.909474 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7cf15d15c5614d31d998f0a6ade67c50f68137f7411c8d982ca83c0653e7906"} err="failed to get container status \"e7cf15d15c5614d31d998f0a6ade67c50f68137f7411c8d982ca83c0653e7906\": rpc error: code = NotFound desc = could not find container \"e7cf15d15c5614d31d998f0a6ade67c50f68137f7411c8d982ca83c0653e7906\": container with ID starting with e7cf15d15c5614d31d998f0a6ade67c50f68137f7411c8d982ca83c0653e7906 not found: ID does not exist" Sep 09 14:39:39.909507 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.909489 2576 scope.go:117] "RemoveContainer" containerID="b20fe58fae1d2c347b64af3a0db85da556a11cbd87377af08dadda0d7c76abe1" Sep 09 14:39:39.909745 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.909726 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b20fe58fae1d2c347b64af3a0db85da556a11cbd87377af08dadda0d7c76abe1"} err="failed to get container status \"b20fe58fae1d2c347b64af3a0db85da556a11cbd87377af08dadda0d7c76abe1\": rpc error: code = NotFound desc = could not find container \"b20fe58fae1d2c347b64af3a0db85da556a11cbd87377af08dadda0d7c76abe1\": container with ID starting with b20fe58fae1d2c347b64af3a0db85da556a11cbd87377af08dadda0d7c76abe1 not found: ID does not exist" Sep 09 14:39:39.909813 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.909747 2576 scope.go:117] "RemoveContainer" containerID="abe29843fa4a9ed94df2de59408d6645211f313e98636b037e11dae09c781ff9" Sep 09 14:39:39.910004 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.909978 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abe29843fa4a9ed94df2de59408d6645211f313e98636b037e11dae09c781ff9"} err="failed to get container status \"abe29843fa4a9ed94df2de59408d6645211f313e98636b037e11dae09c781ff9\": rpc error: code = NotFound desc = could not find container \"abe29843fa4a9ed94df2de59408d6645211f313e98636b037e11dae09c781ff9\": container with ID starting with abe29843fa4a9ed94df2de59408d6645211f313e98636b037e11dae09c781ff9 not found: ID does not exist" Sep 09 14:39:39.910004 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.909995 2576 scope.go:117] "RemoveContainer" containerID="3aa2add971bc20e83acb8526e058a06ca6bcdccf4df036920dd19ad361ffa5e4" Sep 09 14:39:39.910253 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.910224 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3aa2add971bc20e83acb8526e058a06ca6bcdccf4df036920dd19ad361ffa5e4"} err="failed to get container status \"3aa2add971bc20e83acb8526e058a06ca6bcdccf4df036920dd19ad361ffa5e4\": rpc error: code = NotFound desc = could not find container \"3aa2add971bc20e83acb8526e058a06ca6bcdccf4df036920dd19ad361ffa5e4\": container with ID starting with 3aa2add971bc20e83acb8526e058a06ca6bcdccf4df036920dd19ad361ffa5e4 not found: ID does not exist" Sep 09 14:39:39.910253 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.910242 2576 scope.go:117] "RemoveContainer" containerID="413b924324e8ddab89a80df797d617dc6dce0133b6c7d79bc03bb66adaa08055" Sep 09 14:39:39.910541 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.910521 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"413b924324e8ddab89a80df797d617dc6dce0133b6c7d79bc03bb66adaa08055"} err="failed to get container status \"413b924324e8ddab89a80df797d617dc6dce0133b6c7d79bc03bb66adaa08055\": rpc error: code = NotFound desc = could not find container \"413b924324e8ddab89a80df797d617dc6dce0133b6c7d79bc03bb66adaa08055\": container with ID starting with 413b924324e8ddab89a80df797d617dc6dce0133b6c7d79bc03bb66adaa08055 not found: ID does not exist" Sep 09 14:39:39.910604 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.910542 2576 scope.go:117] "RemoveContainer" containerID="2621fdbf311cc0e64590b9500d3a3d05efe0dc2de7110165fe20a5d87b4808b7" Sep 09 14:39:39.910783 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.910766 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2621fdbf311cc0e64590b9500d3a3d05efe0dc2de7110165fe20a5d87b4808b7"} err="failed to get container status \"2621fdbf311cc0e64590b9500d3a3d05efe0dc2de7110165fe20a5d87b4808b7\": rpc error: code = NotFound desc = could not find container \"2621fdbf311cc0e64590b9500d3a3d05efe0dc2de7110165fe20a5d87b4808b7\": container with ID starting with 2621fdbf311cc0e64590b9500d3a3d05efe0dc2de7110165fe20a5d87b4808b7 not found: ID does not exist" Sep 09 14:39:39.910828 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.910786 2576 scope.go:117] "RemoveContainer" containerID="43d1404b45d1b5b7d2c6e4480bc5808400fb3f6ed1d9c08b4219598103cd6159" Sep 09 14:39:39.911006 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.910987 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43d1404b45d1b5b7d2c6e4480bc5808400fb3f6ed1d9c08b4219598103cd6159"} err="failed to get container status \"43d1404b45d1b5b7d2c6e4480bc5808400fb3f6ed1d9c08b4219598103cd6159\": rpc error: code = NotFound desc = could not find container \"43d1404b45d1b5b7d2c6e4480bc5808400fb3f6ed1d9c08b4219598103cd6159\": container with ID starting with 43d1404b45d1b5b7d2c6e4480bc5808400fb3f6ed1d9c08b4219598103cd6159 not found: ID does not exist" Sep 09 14:39:39.911006 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.911004 2576 scope.go:117] "RemoveContainer" containerID="e7cf15d15c5614d31d998f0a6ade67c50f68137f7411c8d982ca83c0653e7906" Sep 09 14:39:39.911237 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.911220 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7cf15d15c5614d31d998f0a6ade67c50f68137f7411c8d982ca83c0653e7906"} err="failed to get container status \"e7cf15d15c5614d31d998f0a6ade67c50f68137f7411c8d982ca83c0653e7906\": rpc error: code = NotFound desc = could not find container \"e7cf15d15c5614d31d998f0a6ade67c50f68137f7411c8d982ca83c0653e7906\": container with ID starting with e7cf15d15c5614d31d998f0a6ade67c50f68137f7411c8d982ca83c0653e7906 not found: ID does not exist" Sep 09 14:39:39.911237 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.911238 2576 scope.go:117] "RemoveContainer" containerID="b20fe58fae1d2c347b64af3a0db85da556a11cbd87377af08dadda0d7c76abe1" Sep 09 14:39:39.911444 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.911426 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b20fe58fae1d2c347b64af3a0db85da556a11cbd87377af08dadda0d7c76abe1"} err="failed to get container status \"b20fe58fae1d2c347b64af3a0db85da556a11cbd87377af08dadda0d7c76abe1\": rpc error: code = NotFound desc = could not find container \"b20fe58fae1d2c347b64af3a0db85da556a11cbd87377af08dadda0d7c76abe1\": container with ID starting with b20fe58fae1d2c347b64af3a0db85da556a11cbd87377af08dadda0d7c76abe1 not found: ID does not exist" Sep 09 14:39:39.911492 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.911445 2576 scope.go:117] "RemoveContainer" containerID="abe29843fa4a9ed94df2de59408d6645211f313e98636b037e11dae09c781ff9" Sep 09 14:39:39.911668 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.911651 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abe29843fa4a9ed94df2de59408d6645211f313e98636b037e11dae09c781ff9"} err="failed to get container status \"abe29843fa4a9ed94df2de59408d6645211f313e98636b037e11dae09c781ff9\": rpc error: code = NotFound desc = could not find container \"abe29843fa4a9ed94df2de59408d6645211f313e98636b037e11dae09c781ff9\": container with ID starting with abe29843fa4a9ed94df2de59408d6645211f313e98636b037e11dae09c781ff9 not found: ID does not exist" Sep 09 14:39:39.911720 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.911668 2576 scope.go:117] "RemoveContainer" containerID="3aa2add971bc20e83acb8526e058a06ca6bcdccf4df036920dd19ad361ffa5e4" Sep 09 14:39:39.911919 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.911902 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3aa2add971bc20e83acb8526e058a06ca6bcdccf4df036920dd19ad361ffa5e4"} err="failed to get container status \"3aa2add971bc20e83acb8526e058a06ca6bcdccf4df036920dd19ad361ffa5e4\": rpc error: code = NotFound desc = could not find container \"3aa2add971bc20e83acb8526e058a06ca6bcdccf4df036920dd19ad361ffa5e4\": container with ID starting with 3aa2add971bc20e83acb8526e058a06ca6bcdccf4df036920dd19ad361ffa5e4 not found: ID does not exist" Sep 09 14:39:39.911979 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.911920 2576 scope.go:117] "RemoveContainer" containerID="413b924324e8ddab89a80df797d617dc6dce0133b6c7d79bc03bb66adaa08055" Sep 09 14:39:39.912160 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.912140 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"413b924324e8ddab89a80df797d617dc6dce0133b6c7d79bc03bb66adaa08055"} err="failed to get container status \"413b924324e8ddab89a80df797d617dc6dce0133b6c7d79bc03bb66adaa08055\": rpc error: code = NotFound desc = could not find container \"413b924324e8ddab89a80df797d617dc6dce0133b6c7d79bc03bb66adaa08055\": container with ID starting with 413b924324e8ddab89a80df797d617dc6dce0133b6c7d79bc03bb66adaa08055 not found: ID does not exist" Sep 09 14:39:39.912203 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.912161 2576 scope.go:117] "RemoveContainer" containerID="2621fdbf311cc0e64590b9500d3a3d05efe0dc2de7110165fe20a5d87b4808b7" Sep 09 14:39:39.912408 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.912389 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2621fdbf311cc0e64590b9500d3a3d05efe0dc2de7110165fe20a5d87b4808b7"} err="failed to get container status \"2621fdbf311cc0e64590b9500d3a3d05efe0dc2de7110165fe20a5d87b4808b7\": rpc error: code = NotFound desc = could not find container \"2621fdbf311cc0e64590b9500d3a3d05efe0dc2de7110165fe20a5d87b4808b7\": container with ID starting with 2621fdbf311cc0e64590b9500d3a3d05efe0dc2de7110165fe20a5d87b4808b7 not found: ID does not exist" Sep 09 14:39:39.912459 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.912408 2576 scope.go:117] "RemoveContainer" containerID="43d1404b45d1b5b7d2c6e4480bc5808400fb3f6ed1d9c08b4219598103cd6159" Sep 09 14:39:39.912623 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.912605 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43d1404b45d1b5b7d2c6e4480bc5808400fb3f6ed1d9c08b4219598103cd6159"} err="failed to get container status \"43d1404b45d1b5b7d2c6e4480bc5808400fb3f6ed1d9c08b4219598103cd6159\": rpc error: code = NotFound desc = could not find container \"43d1404b45d1b5b7d2c6e4480bc5808400fb3f6ed1d9c08b4219598103cd6159\": container with ID starting with 43d1404b45d1b5b7d2c6e4480bc5808400fb3f6ed1d9c08b4219598103cd6159 not found: ID does not exist" Sep 09 14:39:39.912661 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.912624 2576 scope.go:117] "RemoveContainer" containerID="e7cf15d15c5614d31d998f0a6ade67c50f68137f7411c8d982ca83c0653e7906" Sep 09 14:39:39.912828 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.912812 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7cf15d15c5614d31d998f0a6ade67c50f68137f7411c8d982ca83c0653e7906"} err="failed to get container status \"e7cf15d15c5614d31d998f0a6ade67c50f68137f7411c8d982ca83c0653e7906\": rpc error: code = NotFound desc = could not find container \"e7cf15d15c5614d31d998f0a6ade67c50f68137f7411c8d982ca83c0653e7906\": container with ID starting with e7cf15d15c5614d31d998f0a6ade67c50f68137f7411c8d982ca83c0653e7906 not found: ID does not exist" Sep 09 14:39:39.912828 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.912826 2576 scope.go:117] "RemoveContainer" containerID="b20fe58fae1d2c347b64af3a0db85da556a11cbd87377af08dadda0d7c76abe1" Sep 09 14:39:39.913010 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.912995 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b20fe58fae1d2c347b64af3a0db85da556a11cbd87377af08dadda0d7c76abe1"} err="failed to get container status \"b20fe58fae1d2c347b64af3a0db85da556a11cbd87377af08dadda0d7c76abe1\": rpc error: code = NotFound desc = could not find container \"b20fe58fae1d2c347b64af3a0db85da556a11cbd87377af08dadda0d7c76abe1\": container with ID starting with b20fe58fae1d2c347b64af3a0db85da556a11cbd87377af08dadda0d7c76abe1 not found: ID does not exist" Sep 09 14:39:39.913010 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.913008 2576 scope.go:117] "RemoveContainer" containerID="abe29843fa4a9ed94df2de59408d6645211f313e98636b037e11dae09c781ff9" Sep 09 14:39:39.913238 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.913219 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abe29843fa4a9ed94df2de59408d6645211f313e98636b037e11dae09c781ff9"} err="failed to get container status \"abe29843fa4a9ed94df2de59408d6645211f313e98636b037e11dae09c781ff9\": rpc error: code = NotFound desc = could not find container \"abe29843fa4a9ed94df2de59408d6645211f313e98636b037e11dae09c781ff9\": container with ID starting with abe29843fa4a9ed94df2de59408d6645211f313e98636b037e11dae09c781ff9 not found: ID does not exist" Sep 09 14:39:39.913294 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.913238 2576 scope.go:117] "RemoveContainer" containerID="3aa2add971bc20e83acb8526e058a06ca6bcdccf4df036920dd19ad361ffa5e4" Sep 09 14:39:39.913498 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.913481 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3aa2add971bc20e83acb8526e058a06ca6bcdccf4df036920dd19ad361ffa5e4"} err="failed to get container status \"3aa2add971bc20e83acb8526e058a06ca6bcdccf4df036920dd19ad361ffa5e4\": rpc error: code = NotFound desc = could not find container \"3aa2add971bc20e83acb8526e058a06ca6bcdccf4df036920dd19ad361ffa5e4\": container with ID starting with 3aa2add971bc20e83acb8526e058a06ca6bcdccf4df036920dd19ad361ffa5e4 not found: ID does not exist" Sep 09 14:39:39.913555 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.913499 2576 scope.go:117] "RemoveContainer" containerID="413b924324e8ddab89a80df797d617dc6dce0133b6c7d79bc03bb66adaa08055" Sep 09 14:39:39.913729 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.913709 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"413b924324e8ddab89a80df797d617dc6dce0133b6c7d79bc03bb66adaa08055"} err="failed to get container status \"413b924324e8ddab89a80df797d617dc6dce0133b6c7d79bc03bb66adaa08055\": rpc error: code = NotFound desc = could not find container \"413b924324e8ddab89a80df797d617dc6dce0133b6c7d79bc03bb66adaa08055\": container with ID starting with 413b924324e8ddab89a80df797d617dc6dce0133b6c7d79bc03bb66adaa08055 not found: ID does not exist" Sep 09 14:39:39.913766 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.913729 2576 scope.go:117] "RemoveContainer" containerID="2621fdbf311cc0e64590b9500d3a3d05efe0dc2de7110165fe20a5d87b4808b7" Sep 09 14:39:39.913934 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.913918 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2621fdbf311cc0e64590b9500d3a3d05efe0dc2de7110165fe20a5d87b4808b7"} err="failed to get container status \"2621fdbf311cc0e64590b9500d3a3d05efe0dc2de7110165fe20a5d87b4808b7\": rpc error: code = NotFound desc = could not find container \"2621fdbf311cc0e64590b9500d3a3d05efe0dc2de7110165fe20a5d87b4808b7\": container with ID starting with 2621fdbf311cc0e64590b9500d3a3d05efe0dc2de7110165fe20a5d87b4808b7 not found: ID does not exist" Sep 09 14:39:39.913975 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.913934 2576 scope.go:117] "RemoveContainer" containerID="43d1404b45d1b5b7d2c6e4480bc5808400fb3f6ed1d9c08b4219598103cd6159" Sep 09 14:39:39.914153 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.914138 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43d1404b45d1b5b7d2c6e4480bc5808400fb3f6ed1d9c08b4219598103cd6159"} err="failed to get container status \"43d1404b45d1b5b7d2c6e4480bc5808400fb3f6ed1d9c08b4219598103cd6159\": rpc error: code = NotFound desc = could not find container \"43d1404b45d1b5b7d2c6e4480bc5808400fb3f6ed1d9c08b4219598103cd6159\": container with ID starting with 43d1404b45d1b5b7d2c6e4480bc5808400fb3f6ed1d9c08b4219598103cd6159 not found: ID does not exist" Sep 09 14:39:39.914209 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.914153 2576 scope.go:117] "RemoveContainer" containerID="e7cf15d15c5614d31d998f0a6ade67c50f68137f7411c8d982ca83c0653e7906" Sep 09 14:39:39.914420 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.914401 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7cf15d15c5614d31d998f0a6ade67c50f68137f7411c8d982ca83c0653e7906"} err="failed to get container status \"e7cf15d15c5614d31d998f0a6ade67c50f68137f7411c8d982ca83c0653e7906\": rpc error: code = NotFound desc = could not find container \"e7cf15d15c5614d31d998f0a6ade67c50f68137f7411c8d982ca83c0653e7906\": container with ID starting with e7cf15d15c5614d31d998f0a6ade67c50f68137f7411c8d982ca83c0653e7906 not found: ID does not exist" Sep 09 14:39:39.914420 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.914420 2576 scope.go:117] "RemoveContainer" containerID="b20fe58fae1d2c347b64af3a0db85da556a11cbd87377af08dadda0d7c76abe1" Sep 09 14:39:39.914645 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.914629 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b20fe58fae1d2c347b64af3a0db85da556a11cbd87377af08dadda0d7c76abe1"} err="failed to get container status \"b20fe58fae1d2c347b64af3a0db85da556a11cbd87377af08dadda0d7c76abe1\": rpc error: code = NotFound desc = could not find container \"b20fe58fae1d2c347b64af3a0db85da556a11cbd87377af08dadda0d7c76abe1\": container with ID starting with b20fe58fae1d2c347b64af3a0db85da556a11cbd87377af08dadda0d7c76abe1 not found: ID does not exist" Sep 09 14:39:39.914686 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.914646 2576 scope.go:117] "RemoveContainer" containerID="abe29843fa4a9ed94df2de59408d6645211f313e98636b037e11dae09c781ff9" Sep 09 14:39:39.914851 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.914836 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abe29843fa4a9ed94df2de59408d6645211f313e98636b037e11dae09c781ff9"} err="failed to get container status \"abe29843fa4a9ed94df2de59408d6645211f313e98636b037e11dae09c781ff9\": rpc error: code = NotFound desc = could not find container \"abe29843fa4a9ed94df2de59408d6645211f313e98636b037e11dae09c781ff9\": container with ID starting with abe29843fa4a9ed94df2de59408d6645211f313e98636b037e11dae09c781ff9 not found: ID does not exist" Sep 09 14:39:39.914908 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.914852 2576 scope.go:117] "RemoveContainer" containerID="3aa2add971bc20e83acb8526e058a06ca6bcdccf4df036920dd19ad361ffa5e4" Sep 09 14:39:39.915072 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.915055 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3aa2add971bc20e83acb8526e058a06ca6bcdccf4df036920dd19ad361ffa5e4"} err="failed to get container status \"3aa2add971bc20e83acb8526e058a06ca6bcdccf4df036920dd19ad361ffa5e4\": rpc error: code = NotFound desc = could not find container \"3aa2add971bc20e83acb8526e058a06ca6bcdccf4df036920dd19ad361ffa5e4\": container with ID starting with 3aa2add971bc20e83acb8526e058a06ca6bcdccf4df036920dd19ad361ffa5e4 not found: ID does not exist" Sep 09 14:39:39.915109 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.915073 2576 scope.go:117] "RemoveContainer" containerID="413b924324e8ddab89a80df797d617dc6dce0133b6c7d79bc03bb66adaa08055" Sep 09 14:39:39.915349 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.915286 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"413b924324e8ddab89a80df797d617dc6dce0133b6c7d79bc03bb66adaa08055"} err="failed to get container status \"413b924324e8ddab89a80df797d617dc6dce0133b6c7d79bc03bb66adaa08055\": rpc error: code = NotFound desc = could not find container \"413b924324e8ddab89a80df797d617dc6dce0133b6c7d79bc03bb66adaa08055\": container with ID starting with 413b924324e8ddab89a80df797d617dc6dce0133b6c7d79bc03bb66adaa08055 not found: ID does not exist" Sep 09 14:39:39.915403 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.915353 2576 scope.go:117] "RemoveContainer" containerID="2621fdbf311cc0e64590b9500d3a3d05efe0dc2de7110165fe20a5d87b4808b7" Sep 09 14:39:39.915603 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.915587 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2621fdbf311cc0e64590b9500d3a3d05efe0dc2de7110165fe20a5d87b4808b7"} err="failed to get container status \"2621fdbf311cc0e64590b9500d3a3d05efe0dc2de7110165fe20a5d87b4808b7\": rpc error: code = NotFound desc = could not find container \"2621fdbf311cc0e64590b9500d3a3d05efe0dc2de7110165fe20a5d87b4808b7\": container with ID starting with 2621fdbf311cc0e64590b9500d3a3d05efe0dc2de7110165fe20a5d87b4808b7 not found: ID does not exist" Sep 09 14:39:39.915649 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.915604 2576 scope.go:117] "RemoveContainer" containerID="43d1404b45d1b5b7d2c6e4480bc5808400fb3f6ed1d9c08b4219598103cd6159" Sep 09 14:39:39.915819 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.915802 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43d1404b45d1b5b7d2c6e4480bc5808400fb3f6ed1d9c08b4219598103cd6159"} err="failed to get container status \"43d1404b45d1b5b7d2c6e4480bc5808400fb3f6ed1d9c08b4219598103cd6159\": rpc error: code = NotFound desc = could not find container \"43d1404b45d1b5b7d2c6e4480bc5808400fb3f6ed1d9c08b4219598103cd6159\": container with ID starting with 43d1404b45d1b5b7d2c6e4480bc5808400fb3f6ed1d9c08b4219598103cd6159 not found: ID does not exist" Sep 09 14:39:39.915871 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.915821 2576 scope.go:117] "RemoveContainer" containerID="e7cf15d15c5614d31d998f0a6ade67c50f68137f7411c8d982ca83c0653e7906" Sep 09 14:39:39.916034 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.916013 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7cf15d15c5614d31d998f0a6ade67c50f68137f7411c8d982ca83c0653e7906"} err="failed to get container status \"e7cf15d15c5614d31d998f0a6ade67c50f68137f7411c8d982ca83c0653e7906\": rpc error: code = NotFound desc = could not find container \"e7cf15d15c5614d31d998f0a6ade67c50f68137f7411c8d982ca83c0653e7906\": container with ID starting with e7cf15d15c5614d31d998f0a6ade67c50f68137f7411c8d982ca83c0653e7906 not found: ID does not exist" Sep 09 14:39:39.916034 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.916028 2576 scope.go:117] "RemoveContainer" containerID="b20fe58fae1d2c347b64af3a0db85da556a11cbd87377af08dadda0d7c76abe1" Sep 09 14:39:39.916265 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.916248 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b20fe58fae1d2c347b64af3a0db85da556a11cbd87377af08dadda0d7c76abe1"} err="failed to get container status \"b20fe58fae1d2c347b64af3a0db85da556a11cbd87377af08dadda0d7c76abe1\": rpc error: code = NotFound desc = could not find container \"b20fe58fae1d2c347b64af3a0db85da556a11cbd87377af08dadda0d7c76abe1\": container with ID starting with b20fe58fae1d2c347b64af3a0db85da556a11cbd87377af08dadda0d7c76abe1 not found: ID does not exist" Sep 09 14:39:39.916329 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.916265 2576 scope.go:117] "RemoveContainer" containerID="abe29843fa4a9ed94df2de59408d6645211f313e98636b037e11dae09c781ff9" Sep 09 14:39:39.916574 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.916556 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abe29843fa4a9ed94df2de59408d6645211f313e98636b037e11dae09c781ff9"} err="failed to get container status \"abe29843fa4a9ed94df2de59408d6645211f313e98636b037e11dae09c781ff9\": rpc error: code = NotFound desc = could not find container \"abe29843fa4a9ed94df2de59408d6645211f313e98636b037e11dae09c781ff9\": container with ID starting with abe29843fa4a9ed94df2de59408d6645211f313e98636b037e11dae09c781ff9 not found: ID does not exist" Sep 09 14:39:39.916574 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.916574 2576 scope.go:117] "RemoveContainer" containerID="3aa2add971bc20e83acb8526e058a06ca6bcdccf4df036920dd19ad361ffa5e4" Sep 09 14:39:39.916809 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.916791 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3aa2add971bc20e83acb8526e058a06ca6bcdccf4df036920dd19ad361ffa5e4"} err="failed to get container status \"3aa2add971bc20e83acb8526e058a06ca6bcdccf4df036920dd19ad361ffa5e4\": rpc error: code = NotFound desc = could not find container \"3aa2add971bc20e83acb8526e058a06ca6bcdccf4df036920dd19ad361ffa5e4\": container with ID starting with 3aa2add971bc20e83acb8526e058a06ca6bcdccf4df036920dd19ad361ffa5e4 not found: ID does not exist" Sep 09 14:39:39.916859 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.916809 2576 scope.go:117] "RemoveContainer" containerID="413b924324e8ddab89a80df797d617dc6dce0133b6c7d79bc03bb66adaa08055" Sep 09 14:39:39.917042 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.917024 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"413b924324e8ddab89a80df797d617dc6dce0133b6c7d79bc03bb66adaa08055"} err="failed to get container status \"413b924324e8ddab89a80df797d617dc6dce0133b6c7d79bc03bb66adaa08055\": rpc error: code = NotFound desc = could not find container \"413b924324e8ddab89a80df797d617dc6dce0133b6c7d79bc03bb66adaa08055\": container with ID starting with 413b924324e8ddab89a80df797d617dc6dce0133b6c7d79bc03bb66adaa08055 not found: ID does not exist" Sep 09 14:39:39.917113 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.917044 2576 scope.go:117] "RemoveContainer" containerID="2621fdbf311cc0e64590b9500d3a3d05efe0dc2de7110165fe20a5d87b4808b7" Sep 09 14:39:39.917323 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.917292 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2621fdbf311cc0e64590b9500d3a3d05efe0dc2de7110165fe20a5d87b4808b7"} err="failed to get container status \"2621fdbf311cc0e64590b9500d3a3d05efe0dc2de7110165fe20a5d87b4808b7\": rpc error: code = NotFound desc = could not find container \"2621fdbf311cc0e64590b9500d3a3d05efe0dc2de7110165fe20a5d87b4808b7\": container with ID starting with 2621fdbf311cc0e64590b9500d3a3d05efe0dc2de7110165fe20a5d87b4808b7 not found: ID does not exist" Sep 09 14:39:39.917366 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.917324 2576 scope.go:117] "RemoveContainer" containerID="43d1404b45d1b5b7d2c6e4480bc5808400fb3f6ed1d9c08b4219598103cd6159" Sep 09 14:39:39.917553 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.917536 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43d1404b45d1b5b7d2c6e4480bc5808400fb3f6ed1d9c08b4219598103cd6159"} err="failed to get container status \"43d1404b45d1b5b7d2c6e4480bc5808400fb3f6ed1d9c08b4219598103cd6159\": rpc error: code = NotFound desc = could not find container \"43d1404b45d1b5b7d2c6e4480bc5808400fb3f6ed1d9c08b4219598103cd6159\": container with ID starting with 43d1404b45d1b5b7d2c6e4480bc5808400fb3f6ed1d9c08b4219598103cd6159 not found: ID does not exist" Sep 09 14:39:39.917602 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.917553 2576 scope.go:117] "RemoveContainer" containerID="e7cf15d15c5614d31d998f0a6ade67c50f68137f7411c8d982ca83c0653e7906" Sep 09 14:39:39.917785 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.917767 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7cf15d15c5614d31d998f0a6ade67c50f68137f7411c8d982ca83c0653e7906"} err="failed to get container status \"e7cf15d15c5614d31d998f0a6ade67c50f68137f7411c8d982ca83c0653e7906\": rpc error: code = NotFound desc = could not find container \"e7cf15d15c5614d31d998f0a6ade67c50f68137f7411c8d982ca83c0653e7906\": container with ID starting with e7cf15d15c5614d31d998f0a6ade67c50f68137f7411c8d982ca83c0653e7906 not found: ID does not exist" Sep 09 14:39:39.965215 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.965190 2576 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5a424647-68c0-4d35-87e8-496494c0b8f0-thanos-prometheus-http-client-file\") on node \"ip-10-0-133-66.ec2.internal\" DevicePath \"\"" Sep 09 14:39:39.965215 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.965213 2576 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5a424647-68c0-4d35-87e8-496494c0b8f0-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-133-66.ec2.internal\" DevicePath \"\"" Sep 09 14:39:39.965409 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.965227 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qgzb\" (UniqueName: \"kubernetes.io/projected/5a424647-68c0-4d35-87e8-496494c0b8f0-kube-api-access-7qgzb\") on node \"ip-10-0-133-66.ec2.internal\" DevicePath \"\"" Sep 09 14:39:39.965409 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.965240 2576 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5a424647-68c0-4d35-87e8-496494c0b8f0-web-config\") on node \"ip-10-0-133-66.ec2.internal\" DevicePath \"\"" Sep 09 14:39:39.965409 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.965251 2576 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5a424647-68c0-4d35-87e8-496494c0b8f0-secret-kube-rbac-proxy\") on node \"ip-10-0-133-66.ec2.internal\" DevicePath \"\"" Sep 09 14:39:39.965409 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.965262 2576 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5a424647-68c0-4d35-87e8-496494c0b8f0-secret-prometheus-k8s-tls\") on node \"ip-10-0-133-66.ec2.internal\" DevicePath \"\"" Sep 09 14:39:39.965409 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.965274 2576 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5a424647-68c0-4d35-87e8-496494c0b8f0-prometheus-k8s-db\") on node \"ip-10-0-133-66.ec2.internal\" DevicePath \"\"" Sep 09 14:39:39.965409 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.965287 2576 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a424647-68c0-4d35-87e8-496494c0b8f0-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-133-66.ec2.internal\" DevicePath \"\"" Sep 09 14:39:39.965409 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.965325 2576 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5a424647-68c0-4d35-87e8-496494c0b8f0-secret-grpc-tls\") on node \"ip-10-0-133-66.ec2.internal\" DevicePath \"\"" Sep 09 14:39:39.965409 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.965339 2576 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5a424647-68c0-4d35-87e8-496494c0b8f0-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-133-66.ec2.internal\" DevicePath \"\"" Sep 09 14:39:39.965409 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.965351 2576 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5a424647-68c0-4d35-87e8-496494c0b8f0-secret-metrics-client-certs\") on node \"ip-10-0-133-66.ec2.internal\" DevicePath \"\"" Sep 09 14:39:39.965409 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.965363 2576 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5a424647-68c0-4d35-87e8-496494c0b8f0-configmap-metrics-client-ca\") on node \"ip-10-0-133-66.ec2.internal\" DevicePath \"\"" Sep 09 14:39:39.965409 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.965377 2576 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a424647-68c0-4d35-87e8-496494c0b8f0-prometheus-trusted-ca-bundle\") on node \"ip-10-0-133-66.ec2.internal\" DevicePath \"\"" Sep 09 14:39:39.965409 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.965390 2576 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/5a424647-68c0-4d35-87e8-496494c0b8f0-config\") on node \"ip-10-0-133-66.ec2.internal\" DevicePath \"\"" Sep 09 14:39:39.965409 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.965403 2576 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5a424647-68c0-4d35-87e8-496494c0b8f0-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-133-66.ec2.internal\" DevicePath \"\"" Sep 09 14:39:39.965849 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.965417 2576 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5a424647-68c0-4d35-87e8-496494c0b8f0-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-133-66.ec2.internal\" DevicePath \"\"" Sep 09 14:39:39.965849 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.965430 2576 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a424647-68c0-4d35-87e8-496494c0b8f0-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-133-66.ec2.internal\" DevicePath \"\"" Sep 09 14:39:39.965849 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.965442 2576 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5a424647-68c0-4d35-87e8-496494c0b8f0-tls-assets\") on node \"ip-10-0-133-66.ec2.internal\" DevicePath \"\"" Sep 09 14:39:39.965849 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:39.965455 2576 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5a424647-68c0-4d35-87e8-496494c0b8f0-config-out\") on node \"ip-10-0-133-66.ec2.internal\" DevicePath \"\"" Sep 09 14:39:40.178598 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:40.178568 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 09 14:39:40.185263 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:40.185241 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 09 14:39:40.215229 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:40.215201 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 09 14:39:40.215492 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:40.215478 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6" containerName="kube-rbac-proxy-web" Sep 09 14:39:40.215533 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:40.215496 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6" containerName="kube-rbac-proxy-web" Sep 09 14:39:40.215533 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:40.215506 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a424647-68c0-4d35-87e8-496494c0b8f0" containerName="init-config-reloader" Sep 09 14:39:40.215533 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:40.215511 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a424647-68c0-4d35-87e8-496494c0b8f0" containerName="init-config-reloader" Sep 09 14:39:40.215533 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:40.215528 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6" containerName="prom-label-proxy" Sep 09 14:39:40.215645 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:40.215535 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6" containerName="prom-label-proxy" Sep 09 14:39:40.215645 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:40.215543 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6" containerName="alertmanager" Sep 09 14:39:40.215645 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:40.215548 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6" containerName="alertmanager" Sep 09 14:39:40.215645 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:40.215553 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a424647-68c0-4d35-87e8-496494c0b8f0" containerName="kube-rbac-proxy" Sep 09 14:39:40.215645 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:40.215559 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a424647-68c0-4d35-87e8-496494c0b8f0" containerName="kube-rbac-proxy" Sep 09 14:39:40.215645 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:40.215566 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a424647-68c0-4d35-87e8-496494c0b8f0" containerName="config-reloader" Sep 09 14:39:40.215645 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:40.215571 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a424647-68c0-4d35-87e8-496494c0b8f0" containerName="config-reloader" Sep 09 14:39:40.215645 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:40.215576 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a424647-68c0-4d35-87e8-496494c0b8f0" containerName="kube-rbac-proxy-thanos" Sep 09 14:39:40.215645 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:40.215581 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a424647-68c0-4d35-87e8-496494c0b8f0" containerName="kube-rbac-proxy-thanos" Sep 09 14:39:40.215645 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:40.215588 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1fd6e104-62ef-4fb0-9bf7-66da53264398" containerName="registry" Sep 09 14:39:40.215645 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:40.215594 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="1fd6e104-62ef-4fb0-9bf7-66da53264398" containerName="registry" Sep 09 14:39:40.215645 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:40.215602 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a424647-68c0-4d35-87e8-496494c0b8f0" containerName="prometheus" Sep 09 14:39:40.215645 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:40.215610 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a424647-68c0-4d35-87e8-496494c0b8f0" containerName="prometheus" Sep 09 14:39:40.215645 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:40.215616 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a424647-68c0-4d35-87e8-496494c0b8f0" containerName="thanos-sidecar" Sep 09 14:39:40.215645 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:40.215621 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a424647-68c0-4d35-87e8-496494c0b8f0" containerName="thanos-sidecar" Sep 09 14:39:40.215645 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:40.215626 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6" containerName="kube-rbac-proxy-metric" Sep 09 14:39:40.215645 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:40.215631 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6" containerName="kube-rbac-proxy-metric" Sep 09 14:39:40.215645 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:40.215640 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6" containerName="init-config-reloader" Sep 09 14:39:40.215645 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:40.215645 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6" containerName="init-config-reloader" Sep 09 14:39:40.216159 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:40.215654 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6" containerName="kube-rbac-proxy" Sep 09 14:39:40.216159 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:40.215659 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6" containerName="kube-rbac-proxy" Sep 09 14:39:40.216159 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:40.215664 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6" containerName="config-reloader" Sep 09 14:39:40.216159 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:40.215670 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6" containerName="config-reloader" Sep 09 14:39:40.216159 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:40.215679 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea52faf1-b940-490a-9194-a0db3dec5b3f" containerName="console" Sep 09 14:39:40.216159 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:40.215688 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea52faf1-b940-490a-9194-a0db3dec5b3f" containerName="console" Sep 09 14:39:40.216159 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:40.215694 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a424647-68c0-4d35-87e8-496494c0b8f0" containerName="kube-rbac-proxy-web" Sep 09 14:39:40.216159 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:40.215699 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a424647-68c0-4d35-87e8-496494c0b8f0" containerName="kube-rbac-proxy-web" Sep 09 14:39:40.216159 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:40.215736 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a424647-68c0-4d35-87e8-496494c0b8f0" containerName="kube-rbac-proxy" Sep 09 14:39:40.216159 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:40.215744 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a424647-68c0-4d35-87e8-496494c0b8f0" containerName="thanos-sidecar" Sep 09 14:39:40.216159 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:40.215750 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a424647-68c0-4d35-87e8-496494c0b8f0" containerName="kube-rbac-proxy-thanos" Sep 09 14:39:40.216159 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:40.215760 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6" containerName="kube-rbac-proxy-metric" Sep 09 14:39:40.216159 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:40.215768 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a424647-68c0-4d35-87e8-496494c0b8f0" containerName="config-reloader" Sep 09 14:39:40.216159 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:40.215774 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6" containerName="alertmanager" Sep 09 14:39:40.216159 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:40.215779 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6" containerName="kube-rbac-proxy" Sep 09 14:39:40.216159 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:40.215784 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6" containerName="kube-rbac-proxy-web" Sep 09 14:39:40.216159 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:40.215790 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a424647-68c0-4d35-87e8-496494c0b8f0" containerName="prometheus" Sep 09 14:39:40.216159 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:40.215794 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea52faf1-b940-490a-9194-a0db3dec5b3f" containerName="console" Sep 09 14:39:40.216159 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:40.215798 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a424647-68c0-4d35-87e8-496494c0b8f0" containerName="kube-rbac-proxy-web" Sep 09 14:39:40.216159 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:40.215803 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6" containerName="prom-label-proxy" Sep 09 14:39:40.216159 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:40.215808 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="1fd6e104-62ef-4fb0-9bf7-66da53264398" containerName="registry" Sep 09 14:39:40.216159 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:40.215812 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="cddbe9cb-b5f9-448f-8e9c-96e4ee9d66f6" containerName="config-reloader" Sep 09 14:39:40.221052 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:40.221035 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 09 14:39:40.223756 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:40.223738 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Sep 09 14:39:40.223856 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:40.223740 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Sep 09 14:39:40.224033 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:40.224016 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Sep 09 14:39:40.224154 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:40.224089 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Sep 09 14:39:40.224154 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:40.224137 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Sep 09 14:39:40.224257 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:40.224228 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Sep 09 14:39:40.224380 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:40.224364 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Sep 09 14:39:40.224484 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:40.224467 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-6cc0nabvvn998\"" Sep 09 14:39:40.224534 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:40.224471 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-9r2zg\"" Sep 09 14:39:40.224728 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:40.224714 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Sep 09 14:39:40.224728 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:40.224723 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Sep 09 14:39:40.224843 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:40.224742 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Sep 09 14:39:40.224962 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:40.224947 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Sep 09 14:39:40.225559 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:40.225541 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Sep 09 14:39:40.227673 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:40.227651 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Sep 09 14:39:40.230488 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:40.230471 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Sep 09 14:39:40.235559 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:40.235539 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 09 14:39:40.268094 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:40.268063 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a5f6e2c-2708-4f25-94bd-bc638555dec2-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2a5f6e2c-2708-4f25-94bd-bc638555dec2\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 14:39:40.268227 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:40.268102 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2a5f6e2c-2708-4f25-94bd-bc638555dec2-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"2a5f6e2c-2708-4f25-94bd-bc638555dec2\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 14:39:40.268227 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:40.268124 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2a5f6e2c-2708-4f25-94bd-bc638555dec2-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"2a5f6e2c-2708-4f25-94bd-bc638555dec2\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 14:39:40.268227 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:40.268140 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a5f6e2c-2708-4f25-94bd-bc638555dec2-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2a5f6e2c-2708-4f25-94bd-bc638555dec2\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 14:39:40.268227 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:40.268156 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjr5f\" (UniqueName: \"kubernetes.io/projected/2a5f6e2c-2708-4f25-94bd-bc638555dec2-kube-api-access-vjr5f\") pod \"prometheus-k8s-0\" (UID: \"2a5f6e2c-2708-4f25-94bd-bc638555dec2\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 14:39:40.268445 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:40.268221 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2a5f6e2c-2708-4f25-94bd-bc638555dec2-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"2a5f6e2c-2708-4f25-94bd-bc638555dec2\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 14:39:40.268445 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:40.268256 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2a5f6e2c-2708-4f25-94bd-bc638555dec2-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"2a5f6e2c-2708-4f25-94bd-bc638555dec2\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 14:39:40.268445 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:40.268351 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2a5f6e2c-2708-4f25-94bd-bc638555dec2-config-out\") pod \"prometheus-k8s-0\" (UID: \"2a5f6e2c-2708-4f25-94bd-bc638555dec2\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 14:39:40.268445 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:40.268382 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2a5f6e2c-2708-4f25-94bd-bc638555dec2-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"2a5f6e2c-2708-4f25-94bd-bc638555dec2\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 14:39:40.268445 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:40.268419 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2a5f6e2c-2708-4f25-94bd-bc638555dec2-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"2a5f6e2c-2708-4f25-94bd-bc638555dec2\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 14:39:40.268631 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:40.268447 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2a5f6e2c-2708-4f25-94bd-bc638555dec2-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"2a5f6e2c-2708-4f25-94bd-bc638555dec2\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 14:39:40.268631 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:40.268484 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2a5f6e2c-2708-4f25-94bd-bc638555dec2-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"2a5f6e2c-2708-4f25-94bd-bc638555dec2\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 14:39:40.268631 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:40.268510 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2a5f6e2c-2708-4f25-94bd-bc638555dec2-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"2a5f6e2c-2708-4f25-94bd-bc638555dec2\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 14:39:40.268631 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:40.268532 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2a5f6e2c-2708-4f25-94bd-bc638555dec2-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"2a5f6e2c-2708-4f25-94bd-bc638555dec2\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 14:39:40.268631 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:40.268556 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/2a5f6e2c-2708-4f25-94bd-bc638555dec2-config\") pod \"prometheus-k8s-0\" (UID: \"2a5f6e2c-2708-4f25-94bd-bc638555dec2\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 14:39:40.268631 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:40.268601 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2a5f6e2c-2708-4f25-94bd-bc638555dec2-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"2a5f6e2c-2708-4f25-94bd-bc638555dec2\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 14:39:40.268856 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:40.268641 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2a5f6e2c-2708-4f25-94bd-bc638555dec2-web-config\") pod \"prometheus-k8s-0\" (UID: \"2a5f6e2c-2708-4f25-94bd-bc638555dec2\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 14:39:40.268856 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:40.268664 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a5f6e2c-2708-4f25-94bd-bc638555dec2-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2a5f6e2c-2708-4f25-94bd-bc638555dec2\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 14:39:40.268856 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:40.268684 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2a5f6e2c-2708-4f25-94bd-bc638555dec2-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"2a5f6e2c-2708-4f25-94bd-bc638555dec2\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 14:39:40.268856 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:40.268725 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2a5f6e2c-2708-4f25-94bd-bc638555dec2-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"2a5f6e2c-2708-4f25-94bd-bc638555dec2\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 14:39:40.369291 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:40.369263 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2a5f6e2c-2708-4f25-94bd-bc638555dec2-config-out\") pod \"prometheus-k8s-0\" (UID: \"2a5f6e2c-2708-4f25-94bd-bc638555dec2\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 14:39:40.369291 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:40.369293 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2a5f6e2c-2708-4f25-94bd-bc638555dec2-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"2a5f6e2c-2708-4f25-94bd-bc638555dec2\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 14:39:40.369497 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:40.369339 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2a5f6e2c-2708-4f25-94bd-bc638555dec2-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"2a5f6e2c-2708-4f25-94bd-bc638555dec2\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 14:39:40.369497 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:40.369362 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2a5f6e2c-2708-4f25-94bd-bc638555dec2-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"2a5f6e2c-2708-4f25-94bd-bc638555dec2\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 14:39:40.369497 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:40.369389 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2a5f6e2c-2708-4f25-94bd-bc638555dec2-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"2a5f6e2c-2708-4f25-94bd-bc638555dec2\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 14:39:40.369497 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:40.369416 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2a5f6e2c-2708-4f25-94bd-bc638555dec2-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"2a5f6e2c-2708-4f25-94bd-bc638555dec2\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 14:39:40.369497 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:40.369451 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2a5f6e2c-2708-4f25-94bd-bc638555dec2-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"2a5f6e2c-2708-4f25-94bd-bc638555dec2\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 14:39:40.369497 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:40.369487 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/2a5f6e2c-2708-4f25-94bd-bc638555dec2-config\") pod \"prometheus-k8s-0\" (UID: \"2a5f6e2c-2708-4f25-94bd-bc638555dec2\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 14:39:40.369779 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:40.369518 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2a5f6e2c-2708-4f25-94bd-bc638555dec2-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"2a5f6e2c-2708-4f25-94bd-bc638555dec2\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 14:39:40.369779 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:40.369543 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2a5f6e2c-2708-4f25-94bd-bc638555dec2-web-config\") pod \"prometheus-k8s-0\" (UID: \"2a5f6e2c-2708-4f25-94bd-bc638555dec2\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 14:39:40.369779 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:40.369560 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a5f6e2c-2708-4f25-94bd-bc638555dec2-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2a5f6e2c-2708-4f25-94bd-bc638555dec2\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 14:39:40.369779 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:40.369575 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2a5f6e2c-2708-4f25-94bd-bc638555dec2-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"2a5f6e2c-2708-4f25-94bd-bc638555dec2\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 14:39:40.369779 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:40.369596 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2a5f6e2c-2708-4f25-94bd-bc638555dec2-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"2a5f6e2c-2708-4f25-94bd-bc638555dec2\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 14:39:40.369779 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:40.369626 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a5f6e2c-2708-4f25-94bd-bc638555dec2-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2a5f6e2c-2708-4f25-94bd-bc638555dec2\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 14:39:40.369779 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:40.369652 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2a5f6e2c-2708-4f25-94bd-bc638555dec2-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"2a5f6e2c-2708-4f25-94bd-bc638555dec2\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 14:39:40.369779 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:40.369683 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2a5f6e2c-2708-4f25-94bd-bc638555dec2-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"2a5f6e2c-2708-4f25-94bd-bc638555dec2\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 14:39:40.369779 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:40.369709 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a5f6e2c-2708-4f25-94bd-bc638555dec2-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2a5f6e2c-2708-4f25-94bd-bc638555dec2\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 14:39:40.369779 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:40.369736 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjr5f\" (UniqueName: \"kubernetes.io/projected/2a5f6e2c-2708-4f25-94bd-bc638555dec2-kube-api-access-vjr5f\") pod \"prometheus-k8s-0\" (UID: \"2a5f6e2c-2708-4f25-94bd-bc638555dec2\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 14:39:40.369779 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:40.369764 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2a5f6e2c-2708-4f25-94bd-bc638555dec2-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"2a5f6e2c-2708-4f25-94bd-bc638555dec2\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 14:39:40.370506 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:40.369787 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2a5f6e2c-2708-4f25-94bd-bc638555dec2-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"2a5f6e2c-2708-4f25-94bd-bc638555dec2\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 14:39:40.370506 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:40.369831 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2a5f6e2c-2708-4f25-94bd-bc638555dec2-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"2a5f6e2c-2708-4f25-94bd-bc638555dec2\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 14:39:40.370506 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:40.370065 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2a5f6e2c-2708-4f25-94bd-bc638555dec2-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"2a5f6e2c-2708-4f25-94bd-bc638555dec2\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 14:39:40.370506 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:40.370480 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a5f6e2c-2708-4f25-94bd-bc638555dec2-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2a5f6e2c-2708-4f25-94bd-bc638555dec2\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 14:39:40.370711 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:40.370616 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2a5f6e2c-2708-4f25-94bd-bc638555dec2-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"2a5f6e2c-2708-4f25-94bd-bc638555dec2\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 14:39:40.372046 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:40.371765 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2a5f6e2c-2708-4f25-94bd-bc638555dec2-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"2a5f6e2c-2708-4f25-94bd-bc638555dec2\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 14:39:40.372046 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:40.371947 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a5f6e2c-2708-4f25-94bd-bc638555dec2-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2a5f6e2c-2708-4f25-94bd-bc638555dec2\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 14:39:40.372340 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:40.372292 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/2a5f6e2c-2708-4f25-94bd-bc638555dec2-config\") pod \"prometheus-k8s-0\" (UID: \"2a5f6e2c-2708-4f25-94bd-bc638555dec2\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 14:39:40.372433 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:40.372343 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2a5f6e2c-2708-4f25-94bd-bc638555dec2-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"2a5f6e2c-2708-4f25-94bd-bc638555dec2\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 14:39:40.372433 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:40.372294 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2a5f6e2c-2708-4f25-94bd-bc638555dec2-config-out\") pod \"prometheus-k8s-0\" (UID: \"2a5f6e2c-2708-4f25-94bd-bc638555dec2\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 14:39:40.372806 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:40.372779 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2a5f6e2c-2708-4f25-94bd-bc638555dec2-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"2a5f6e2c-2708-4f25-94bd-bc638555dec2\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 14:39:40.373129 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:40.373088 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2a5f6e2c-2708-4f25-94bd-bc638555dec2-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"2a5f6e2c-2708-4f25-94bd-bc638555dec2\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 14:39:40.373603 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:40.373549 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a5f6e2c-2708-4f25-94bd-bc638555dec2-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2a5f6e2c-2708-4f25-94bd-bc638555dec2\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 14:39:40.374348 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:40.374323 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2a5f6e2c-2708-4f25-94bd-bc638555dec2-web-config\") pod \"prometheus-k8s-0\" (UID: \"2a5f6e2c-2708-4f25-94bd-bc638555dec2\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 14:39:40.374803 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:40.374778 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2a5f6e2c-2708-4f25-94bd-bc638555dec2-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"2a5f6e2c-2708-4f25-94bd-bc638555dec2\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 14:39:40.374934 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:40.374915 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2a5f6e2c-2708-4f25-94bd-bc638555dec2-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"2a5f6e2c-2708-4f25-94bd-bc638555dec2\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 14:39:40.375086 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:40.375070 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2a5f6e2c-2708-4f25-94bd-bc638555dec2-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"2a5f6e2c-2708-4f25-94bd-bc638555dec2\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 14:39:40.375562 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:40.375542 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2a5f6e2c-2708-4f25-94bd-bc638555dec2-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"2a5f6e2c-2708-4f25-94bd-bc638555dec2\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 14:39:40.375621 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:40.375572 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2a5f6e2c-2708-4f25-94bd-bc638555dec2-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"2a5f6e2c-2708-4f25-94bd-bc638555dec2\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 14:39:40.375773 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:40.375757 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2a5f6e2c-2708-4f25-94bd-bc638555dec2-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"2a5f6e2c-2708-4f25-94bd-bc638555dec2\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 14:39:40.381994 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:40.381973 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjr5f\" (UniqueName: \"kubernetes.io/projected/2a5f6e2c-2708-4f25-94bd-bc638555dec2-kube-api-access-vjr5f\") pod \"prometheus-k8s-0\" (UID: \"2a5f6e2c-2708-4f25-94bd-bc638555dec2\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 14:39:40.531067 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:40.531026 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 09 14:39:40.815396 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:40.815370 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 09 14:39:40.817481 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:39:40.817453 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a5f6e2c_2708_4f25_94bd_bc638555dec2.slice/crio-730972ad028f6d0f415f2a67fc7917d1e06b76a75319d3859b7127fa38cca3e0 WatchSource:0}: Error finding container 730972ad028f6d0f415f2a67fc7917d1e06b76a75319d3859b7127fa38cca3e0: Status 404 returned error can't find the container with id 730972ad028f6d0f415f2a67fc7917d1e06b76a75319d3859b7127fa38cca3e0 Sep 09 14:39:40.861325 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:40.861284 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2a5f6e2c-2708-4f25-94bd-bc638555dec2","Type":"ContainerStarted","Data":"730972ad028f6d0f415f2a67fc7917d1e06b76a75319d3859b7127fa38cca3e0"} Sep 09 14:39:41.514860 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:41.514829 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a424647-68c0-4d35-87e8-496494c0b8f0" path="/var/lib/kubelet/pods/5a424647-68c0-4d35-87e8-496494c0b8f0/volumes" Sep 09 14:39:41.866796 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:41.866710 2576 generic.go:356] "Generic (PLEG): container finished" podID="2a5f6e2c-2708-4f25-94bd-bc638555dec2" containerID="5931f391d7727ecb68917379a0e6fe33b8d35fdce02bd9ba6284ba72e579b399" exitCode=0 Sep 09 14:39:41.866796 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:41.866785 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2a5f6e2c-2708-4f25-94bd-bc638555dec2","Type":"ContainerDied","Data":"5931f391d7727ecb68917379a0e6fe33b8d35fdce02bd9ba6284ba72e579b399"} Sep 09 14:39:42.874007 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:42.873974 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2a5f6e2c-2708-4f25-94bd-bc638555dec2","Type":"ContainerStarted","Data":"e2b5d29ac8cda70b9dd8e20cb37a92eaf98fa7923f237c71e773cd2e806000f8"} Sep 09 14:39:42.874450 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:42.874034 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2a5f6e2c-2708-4f25-94bd-bc638555dec2","Type":"ContainerStarted","Data":"b76599a6cf6a78a0a059aa4a0386bbc6e7d82fdd16bb349b4d8e6dedcd1630b2"} Sep 09 14:39:42.874450 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:42.874060 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2a5f6e2c-2708-4f25-94bd-bc638555dec2","Type":"ContainerStarted","Data":"8c9e63b206317effaa850acdd82cf70335a472645af44276d3c9d6ea436af0d6"} Sep 09 14:39:42.874450 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:42.874083 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2a5f6e2c-2708-4f25-94bd-bc638555dec2","Type":"ContainerStarted","Data":"a4b499f46cbfd13bde27f3527c4f8b55955bc59c08e00cfb4c3c508060bd9c65"} Sep 09 14:39:42.874450 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:42.874108 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2a5f6e2c-2708-4f25-94bd-bc638555dec2","Type":"ContainerStarted","Data":"5e5e7db742905024c4e7560ece4cec9340304c08572872fb3c296f94d489aef3"} Sep 09 14:39:42.874450 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:42.874131 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2a5f6e2c-2708-4f25-94bd-bc638555dec2","Type":"ContainerStarted","Data":"d1b859eec3a644c741b4afacaafabd241b263318c3c5ff87d6d2b77ff747c1d4"} Sep 09 14:39:42.911566 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:42.911505 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.8929944770000002 podStartE2EDuration="2.911484591s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.490114ms" isStatefulPod=false podCreationTimestamp="2026-09-09 14:39:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:39:42.909831094 +0000 UTC m=+397.920747364" watchObservedRunningTime="2026-09-09 14:39:42.911484591 +0000 UTC m=+397.922400844" Sep 09 14:39:45.531636 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:45.531606 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 09 14:39:49.331803 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:39:49.331762 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-79977f9844-gpnd5" podUID="2177b959-0d75-41f6-b309-06f66757e45b" Sep 09 14:39:49.901797 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:49.901771 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-gpnd5" Sep 09 14:39:51.457987 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:51.457956 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2177b959-0d75-41f6-b309-06f66757e45b-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-gpnd5\" (UID: \"2177b959-0d75-41f6-b309-06f66757e45b\") " pod="openshift-network-console/networking-console-plugin-79977f9844-gpnd5" Sep 09 14:39:51.460420 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:51.460400 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2177b959-0d75-41f6-b309-06f66757e45b-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-gpnd5\" (UID: \"2177b959-0d75-41f6-b309-06f66757e45b\") " pod="openshift-network-console/networking-console-plugin-79977f9844-gpnd5" Sep 09 14:39:51.705450 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:51.705422 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-2lx2l\"" Sep 09 14:39:51.712955 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:51.712896 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-gpnd5" Sep 09 14:39:51.980949 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:51.980881 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-gpnd5"] Sep 09 14:39:51.989002 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:39:51.988959 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2177b959_0d75_41f6_b309_06f66757e45b.slice/crio-77c0f089f7b01b38251a05ca83814c80c7b3256ab736406a0b769abb91236de5 WatchSource:0}: Error finding container 77c0f089f7b01b38251a05ca83814c80c7b3256ab736406a0b769abb91236de5: Status 404 returned error can't find the container with id 77c0f089f7b01b38251a05ca83814c80c7b3256ab736406a0b769abb91236de5 Sep 09 14:39:52.916286 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:52.916244 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-gpnd5" event={"ID":"2177b959-0d75-41f6-b309-06f66757e45b","Type":"ContainerStarted","Data":"77c0f089f7b01b38251a05ca83814c80c7b3256ab736406a0b769abb91236de5"} Sep 09 14:39:53.921497 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:53.921461 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-gpnd5" event={"ID":"2177b959-0d75-41f6-b309-06f66757e45b","Type":"ContainerStarted","Data":"fcd6e562a9483aec1242f8948aabc8d251b2f86cdc2069787e284d236ddefa95"} Sep 09 14:39:53.940104 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:39:53.940054 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-79977f9844-gpnd5" podStartSLOduration=381.773804444 podStartE2EDuration="6m22.9400393s" totalImagesPullingTime="1.166234856s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 14:33:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:39:53.93825525 +0000 UTC m=+408.949171513" watchObservedRunningTime="2026-09-09 14:39:53.9400393 +0000 UTC m=+408.950955551" Sep 09 14:40:40.531447 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:40:40.531363 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Sep 09 14:40:40.547167 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:40:40.547138 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Sep 09 14:40:41.113921 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:40:41.113893 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 09 14:43:05.401800 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:43:05.401775 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7xcn5_186aab90-266b-4eae-977a-3f053b423ef3/ovn-acl-logging/0.log" Sep 09 14:43:05.402349 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:43:05.402333 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7xcn5_186aab90-266b-4eae-977a-3f053b423ef3/ovn-acl-logging/0.log" Sep 09 14:43:34.752274 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:43:34.752198 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-75cf6f4fdf-vn5ms"] Sep 09 14:43:34.755726 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:43:34.755696 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75cf6f4fdf-vn5ms" Sep 09 14:43:34.758743 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:43:34.758725 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 09 14:43:34.758845 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:43:34.758767 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 09 14:43:34.759649 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:43:34.759625 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 09 14:43:34.759752 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:43:34.759654 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 09 14:43:34.759752 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:43:34.759741 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 09 14:43:34.759880 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:43:34.759863 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 09 14:43:34.759940 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:43:34.759912 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 09 14:43:34.760024 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:43:34.760005 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-wcftc\"" Sep 09 14:43:34.766664 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:43:34.766645 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Sep 09 14:43:34.769467 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:43:34.769446 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-75cf6f4fdf-vn5ms"] Sep 09 14:43:34.840284 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:43:34.840251 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/637a9e8d-3d50-46dd-8781-58b5528732cd-console-serving-cert\") pod \"console-75cf6f4fdf-vn5ms\" (UID: \"637a9e8d-3d50-46dd-8781-58b5528732cd\") " pod="openshift-console/console-75cf6f4fdf-vn5ms" Sep 09 14:43:34.840453 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:43:34.840290 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njwvb\" (UniqueName: \"kubernetes.io/projected/637a9e8d-3d50-46dd-8781-58b5528732cd-kube-api-access-njwvb\") pod \"console-75cf6f4fdf-vn5ms\" (UID: \"637a9e8d-3d50-46dd-8781-58b5528732cd\") " pod="openshift-console/console-75cf6f4fdf-vn5ms" Sep 09 14:43:34.840453 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:43:34.840356 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/637a9e8d-3d50-46dd-8781-58b5528732cd-service-ca\") pod \"console-75cf6f4fdf-vn5ms\" (UID: \"637a9e8d-3d50-46dd-8781-58b5528732cd\") " pod="openshift-console/console-75cf6f4fdf-vn5ms" Sep 09 14:43:34.840453 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:43:34.840382 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/637a9e8d-3d50-46dd-8781-58b5528732cd-console-config\") pod \"console-75cf6f4fdf-vn5ms\" (UID: \"637a9e8d-3d50-46dd-8781-58b5528732cd\") " pod="openshift-console/console-75cf6f4fdf-vn5ms" Sep 09 14:43:34.840453 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:43:34.840397 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/637a9e8d-3d50-46dd-8781-58b5528732cd-trusted-ca-bundle\") pod \"console-75cf6f4fdf-vn5ms\" (UID: \"637a9e8d-3d50-46dd-8781-58b5528732cd\") " pod="openshift-console/console-75cf6f4fdf-vn5ms" Sep 09 14:43:34.840583 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:43:34.840474 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/637a9e8d-3d50-46dd-8781-58b5528732cd-console-oauth-config\") pod \"console-75cf6f4fdf-vn5ms\" (UID: \"637a9e8d-3d50-46dd-8781-58b5528732cd\") " pod="openshift-console/console-75cf6f4fdf-vn5ms" Sep 09 14:43:34.840583 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:43:34.840504 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/637a9e8d-3d50-46dd-8781-58b5528732cd-oauth-serving-cert\") pod \"console-75cf6f4fdf-vn5ms\" (UID: \"637a9e8d-3d50-46dd-8781-58b5528732cd\") " pod="openshift-console/console-75cf6f4fdf-vn5ms" Sep 09 14:43:34.941292 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:43:34.941259 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/637a9e8d-3d50-46dd-8781-58b5528732cd-oauth-serving-cert\") pod \"console-75cf6f4fdf-vn5ms\" (UID: \"637a9e8d-3d50-46dd-8781-58b5528732cd\") " pod="openshift-console/console-75cf6f4fdf-vn5ms" Sep 09 14:43:34.941465 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:43:34.941322 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/637a9e8d-3d50-46dd-8781-58b5528732cd-console-serving-cert\") pod \"console-75cf6f4fdf-vn5ms\" (UID: \"637a9e8d-3d50-46dd-8781-58b5528732cd\") " pod="openshift-console/console-75cf6f4fdf-vn5ms" Sep 09 14:43:34.941465 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:43:34.941353 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njwvb\" (UniqueName: \"kubernetes.io/projected/637a9e8d-3d50-46dd-8781-58b5528732cd-kube-api-access-njwvb\") pod \"console-75cf6f4fdf-vn5ms\" (UID: \"637a9e8d-3d50-46dd-8781-58b5528732cd\") " pod="openshift-console/console-75cf6f4fdf-vn5ms" Sep 09 14:43:34.941465 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:43:34.941376 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/637a9e8d-3d50-46dd-8781-58b5528732cd-service-ca\") pod \"console-75cf6f4fdf-vn5ms\" (UID: \"637a9e8d-3d50-46dd-8781-58b5528732cd\") " pod="openshift-console/console-75cf6f4fdf-vn5ms" Sep 09 14:43:34.941465 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:43:34.941396 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/637a9e8d-3d50-46dd-8781-58b5528732cd-console-config\") pod \"console-75cf6f4fdf-vn5ms\" (UID: \"637a9e8d-3d50-46dd-8781-58b5528732cd\") " pod="openshift-console/console-75cf6f4fdf-vn5ms" Sep 09 14:43:34.941648 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:43:34.941555 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/637a9e8d-3d50-46dd-8781-58b5528732cd-trusted-ca-bundle\") pod \"console-75cf6f4fdf-vn5ms\" (UID: \"637a9e8d-3d50-46dd-8781-58b5528732cd\") " pod="openshift-console/console-75cf6f4fdf-vn5ms" Sep 09 14:43:34.941700 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:43:34.941649 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/637a9e8d-3d50-46dd-8781-58b5528732cd-console-oauth-config\") pod \"console-75cf6f4fdf-vn5ms\" (UID: \"637a9e8d-3d50-46dd-8781-58b5528732cd\") " pod="openshift-console/console-75cf6f4fdf-vn5ms" Sep 09 14:43:34.942114 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:43:34.942080 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/637a9e8d-3d50-46dd-8781-58b5528732cd-oauth-serving-cert\") pod \"console-75cf6f4fdf-vn5ms\" (UID: \"637a9e8d-3d50-46dd-8781-58b5528732cd\") " pod="openshift-console/console-75cf6f4fdf-vn5ms" Sep 09 14:43:34.942347 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:43:34.942295 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/637a9e8d-3d50-46dd-8781-58b5528732cd-service-ca\") pod \"console-75cf6f4fdf-vn5ms\" (UID: \"637a9e8d-3d50-46dd-8781-58b5528732cd\") " pod="openshift-console/console-75cf6f4fdf-vn5ms" Sep 09 14:43:34.942471 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:43:34.942452 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/637a9e8d-3d50-46dd-8781-58b5528732cd-console-config\") pod \"console-75cf6f4fdf-vn5ms\" (UID: \"637a9e8d-3d50-46dd-8781-58b5528732cd\") " pod="openshift-console/console-75cf6f4fdf-vn5ms" Sep 09 14:43:34.942578 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:43:34.942563 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/637a9e8d-3d50-46dd-8781-58b5528732cd-trusted-ca-bundle\") pod \"console-75cf6f4fdf-vn5ms\" (UID: \"637a9e8d-3d50-46dd-8781-58b5528732cd\") " pod="openshift-console/console-75cf6f4fdf-vn5ms" Sep 09 14:43:34.944017 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:43:34.943992 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/637a9e8d-3d50-46dd-8781-58b5528732cd-console-serving-cert\") pod \"console-75cf6f4fdf-vn5ms\" (UID: \"637a9e8d-3d50-46dd-8781-58b5528732cd\") " pod="openshift-console/console-75cf6f4fdf-vn5ms" Sep 09 14:43:34.944117 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:43:34.944100 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/637a9e8d-3d50-46dd-8781-58b5528732cd-console-oauth-config\") pod \"console-75cf6f4fdf-vn5ms\" (UID: \"637a9e8d-3d50-46dd-8781-58b5528732cd\") " pod="openshift-console/console-75cf6f4fdf-vn5ms" Sep 09 14:43:34.950048 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:43:34.950027 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-njwvb\" (UniqueName: \"kubernetes.io/projected/637a9e8d-3d50-46dd-8781-58b5528732cd-kube-api-access-njwvb\") pod \"console-75cf6f4fdf-vn5ms\" (UID: \"637a9e8d-3d50-46dd-8781-58b5528732cd\") " pod="openshift-console/console-75cf6f4fdf-vn5ms" Sep 09 14:43:35.065675 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:43:35.065628 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75cf6f4fdf-vn5ms" Sep 09 14:43:35.350622 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:43:35.350595 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-75cf6f4fdf-vn5ms"] Sep 09 14:43:35.353019 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:43:35.352988 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod637a9e8d_3d50_46dd_8781_58b5528732cd.slice/crio-e81d91065367cef58d4ec697e0d9f8ed231e9f68b5f55925912ac3972b0deec3 WatchSource:0}: Error finding container e81d91065367cef58d4ec697e0d9f8ed231e9f68b5f55925912ac3972b0deec3: Status 404 returned error can't find the container with id e81d91065367cef58d4ec697e0d9f8ed231e9f68b5f55925912ac3972b0deec3 Sep 09 14:43:35.722709 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:43:35.722617 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75cf6f4fdf-vn5ms" event={"ID":"637a9e8d-3d50-46dd-8781-58b5528732cd","Type":"ContainerStarted","Data":"93412c6d66c8a197d41797aac01ca17ba70f5f66b74f31a33d21a9346a7ebbfc"} Sep 09 14:43:35.722709 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:43:35.722669 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75cf6f4fdf-vn5ms" event={"ID":"637a9e8d-3d50-46dd-8781-58b5528732cd","Type":"ContainerStarted","Data":"e81d91065367cef58d4ec697e0d9f8ed231e9f68b5f55925912ac3972b0deec3"} Sep 09 14:43:35.749695 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:43:35.749640 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-75cf6f4fdf-vn5ms" podStartSLOduration=1.749622718 podStartE2EDuration="1.749622718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 14:43:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:43:35.749586336 +0000 UTC m=+630.760502611" watchObservedRunningTime="2026-09-09 14:43:35.749622718 +0000 UTC m=+630.760538969" Sep 09 14:43:45.066571 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:43:45.066518 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-75cf6f4fdf-vn5ms" Sep 09 14:43:45.067061 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:43:45.066597 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-75cf6f4fdf-vn5ms" Sep 09 14:43:45.071097 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:43:45.071078 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-75cf6f4fdf-vn5ms" Sep 09 14:43:45.766488 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:43:45.766461 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-75cf6f4fdf-vn5ms" Sep 09 14:48:05.426422 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:48:05.426346 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7xcn5_186aab90-266b-4eae-977a-3f053b423ef3/ovn-acl-logging/0.log" Sep 09 14:48:05.426422 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:48:05.426346 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7xcn5_186aab90-266b-4eae-977a-3f053b423ef3/ovn-acl-logging/0.log" Sep 09 14:48:17.141603 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:48:17.141562 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/redis-69bbd798b6-kz8lw"] Sep 09 14:48:17.144828 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:48:17.144807 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/redis-69bbd798b6-kz8lw" Sep 09 14:48:17.148576 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:48:17.148551 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\"" Sep 09 14:48:17.148930 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:48:17.148911 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\"" Sep 09 14:48:17.149620 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:48:17.149606 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"default-dockercfg-jfdpb\"" Sep 09 14:48:17.161245 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:48:17.161220 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/redis-69bbd798b6-kz8lw"] Sep 09 14:48:17.322437 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:48:17.322396 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmxjb\" (UniqueName: \"kubernetes.io/projected/9626bf65-25a9-4051-9db1-fb9b90242e76-kube-api-access-jmxjb\") pod \"redis-69bbd798b6-kz8lw\" (UID: \"9626bf65-25a9-4051-9db1-fb9b90242e76\") " pod="test-ns-feast/redis-69bbd798b6-kz8lw" Sep 09 14:48:17.422936 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:48:17.422842 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmxjb\" (UniqueName: \"kubernetes.io/projected/9626bf65-25a9-4051-9db1-fb9b90242e76-kube-api-access-jmxjb\") pod \"redis-69bbd798b6-kz8lw\" (UID: \"9626bf65-25a9-4051-9db1-fb9b90242e76\") " pod="test-ns-feast/redis-69bbd798b6-kz8lw" Sep 09 14:48:17.431627 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:48:17.431594 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmxjb\" (UniqueName: \"kubernetes.io/projected/9626bf65-25a9-4051-9db1-fb9b90242e76-kube-api-access-jmxjb\") pod \"redis-69bbd798b6-kz8lw\" (UID: \"9626bf65-25a9-4051-9db1-fb9b90242e76\") " pod="test-ns-feast/redis-69bbd798b6-kz8lw" Sep 09 14:48:17.473569 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:48:17.473531 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/redis-69bbd798b6-kz8lw" Sep 09 14:48:17.748102 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:48:17.748012 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/redis-69bbd798b6-kz8lw"] Sep 09 14:48:17.751053 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:48:17.751021 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9626bf65_25a9_4051_9db1_fb9b90242e76.slice/crio-236104a4362992beae4c95f4b929d9db13ff828616d20ccddae64608b413ed4d WatchSource:0}: Error finding container 236104a4362992beae4c95f4b929d9db13ff828616d20ccddae64608b413ed4d: Status 404 returned error can't find the container with id 236104a4362992beae4c95f4b929d9db13ff828616d20ccddae64608b413ed4d Sep 09 14:48:17.752931 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:48:17.752912 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 09 14:48:17.800135 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:48:17.800095 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/redis-69bbd798b6-kz8lw" event={"ID":"9626bf65-25a9-4051-9db1-fb9b90242e76","Type":"ContainerStarted","Data":"236104a4362992beae4c95f4b929d9db13ff828616d20ccddae64608b413ed4d"} Sep 09 14:48:19.810153 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:48:19.810113 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/redis-69bbd798b6-kz8lw" event={"ID":"9626bf65-25a9-4051-9db1-fb9b90242e76","Type":"ContainerStarted","Data":"392f990bbe94be1e61ff93c15618f39dcfcc71a1f247dabc814c26fdbab06029"} Sep 09 14:48:19.845388 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:48:19.845334 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/redis-69bbd798b6-kz8lw" podStartSLOduration=1.389657602 podStartE2EDuration="2.845292635s" totalImagesPullingTime="1.455635033s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 14:48:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:48:19.845014664 +0000 UTC m=+914.855930914" watchObservedRunningTime="2026-09-09 14:48:19.845292635 +0000 UTC m=+914.856208885" Sep 09 14:49:57.087369 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:49:57.087333 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/redis-69bbd798b6-kz8lw"] Sep 09 14:49:57.087900 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:49:57.087501 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/redis-69bbd798b6-kz8lw" podUID="9626bf65-25a9-4051-9db1-fb9b90242e76" containerName="redis" containerID="cri-o://392f990bbe94be1e61ff93c15618f39dcfcc71a1f247dabc814c26fdbab06029" gracePeriod=30 Sep 09 14:49:57.331150 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:49:57.331122 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/redis-69bbd798b6-kz8lw" Sep 09 14:49:57.438923 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:49:57.438839 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jmxjb\" (UniqueName: \"kubernetes.io/projected/9626bf65-25a9-4051-9db1-fb9b90242e76-kube-api-access-jmxjb\") pod \"9626bf65-25a9-4051-9db1-fb9b90242e76\" (UID: \"9626bf65-25a9-4051-9db1-fb9b90242e76\") " Sep 09 14:49:57.441174 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:49:57.441140 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9626bf65-25a9-4051-9db1-fb9b90242e76-kube-api-access-jmxjb" (OuterVolumeSpecName: "kube-api-access-jmxjb") pod "9626bf65-25a9-4051-9db1-fb9b90242e76" (UID: "9626bf65-25a9-4051-9db1-fb9b90242e76"). InnerVolumeSpecName "kube-api-access-jmxjb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 14:49:57.539790 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:49:57.539760 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmxjb\" (UniqueName: \"kubernetes.io/projected/9626bf65-25a9-4051-9db1-fb9b90242e76-kube-api-access-jmxjb\") on node \"ip-10-0-133-66.ec2.internal\" DevicePath \"\"" Sep 09 14:49:58.189440 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:49:58.189401 2576 generic.go:356] "Generic (PLEG): container finished" podID="9626bf65-25a9-4051-9db1-fb9b90242e76" containerID="392f990bbe94be1e61ff93c15618f39dcfcc71a1f247dabc814c26fdbab06029" exitCode=0 Sep 09 14:49:58.189440 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:49:58.189440 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/redis-69bbd798b6-kz8lw" Sep 09 14:49:58.189902 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:49:58.189489 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/redis-69bbd798b6-kz8lw" event={"ID":"9626bf65-25a9-4051-9db1-fb9b90242e76","Type":"ContainerDied","Data":"392f990bbe94be1e61ff93c15618f39dcfcc71a1f247dabc814c26fdbab06029"} Sep 09 14:49:58.189902 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:49:58.189527 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/redis-69bbd798b6-kz8lw" event={"ID":"9626bf65-25a9-4051-9db1-fb9b90242e76","Type":"ContainerDied","Data":"236104a4362992beae4c95f4b929d9db13ff828616d20ccddae64608b413ed4d"} Sep 09 14:49:58.189902 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:49:58.189547 2576 scope.go:117] "RemoveContainer" containerID="392f990bbe94be1e61ff93c15618f39dcfcc71a1f247dabc814c26fdbab06029" Sep 09 14:49:58.197709 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:49:58.197690 2576 scope.go:117] "RemoveContainer" containerID="392f990bbe94be1e61ff93c15618f39dcfcc71a1f247dabc814c26fdbab06029" Sep 09 14:49:58.198063 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:49:58.198043 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"392f990bbe94be1e61ff93c15618f39dcfcc71a1f247dabc814c26fdbab06029\": container with ID starting with 392f990bbe94be1e61ff93c15618f39dcfcc71a1f247dabc814c26fdbab06029 not found: ID does not exist" containerID="392f990bbe94be1e61ff93c15618f39dcfcc71a1f247dabc814c26fdbab06029" Sep 09 14:49:58.198132 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:49:58.198072 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"392f990bbe94be1e61ff93c15618f39dcfcc71a1f247dabc814c26fdbab06029"} err="failed to get container status \"392f990bbe94be1e61ff93c15618f39dcfcc71a1f247dabc814c26fdbab06029\": rpc error: code = NotFound desc = could not find container \"392f990bbe94be1e61ff93c15618f39dcfcc71a1f247dabc814c26fdbab06029\": container with ID starting with 392f990bbe94be1e61ff93c15618f39dcfcc71a1f247dabc814c26fdbab06029 not found: ID does not exist" Sep 09 14:49:58.208443 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:49:58.208412 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/redis-69bbd798b6-kz8lw"] Sep 09 14:49:58.212771 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:49:58.212742 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/redis-69bbd798b6-kz8lw"] Sep 09 14:49:59.514538 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:49:59.514505 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9626bf65-25a9-4051-9db1-fb9b90242e76" path="/var/lib/kubelet/pods/9626bf65-25a9-4051-9db1-fb9b90242e76/volumes" Sep 09 14:53:05.455006 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:53:05.454978 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7xcn5_186aab90-266b-4eae-977a-3f053b423ef3/ovn-acl-logging/0.log" Sep 09 14:53:05.456144 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:53:05.456124 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7xcn5_186aab90-266b-4eae-977a-3f053b423ef3/ovn-acl-logging/0.log" Sep 09 14:55:16.885217 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:55:16.885176 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-qpb79"] Sep 09 14:55:16.885843 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:55:16.885626 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9626bf65-25a9-4051-9db1-fb9b90242e76" containerName="redis" Sep 09 14:55:16.885843 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:55:16.885643 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="9626bf65-25a9-4051-9db1-fb9b90242e76" containerName="redis" Sep 09 14:55:16.885843 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:55:16.885730 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="9626bf65-25a9-4051-9db1-fb9b90242e76" containerName="redis" Sep 09 14:55:16.889210 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:55:16.889189 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qpb79" Sep 09 14:55:16.891917 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:55:16.891895 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\"" Sep 09 14:55:16.892068 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:55:16.892048 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-online-tls\"" Sep 09 14:55:16.892196 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:55:16.891955 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\"" Sep 09 14:55:16.892283 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:55:16.891985 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-offline-tls\"" Sep 09 14:55:16.892363 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:55:16.891945 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-registry-tls\"" Sep 09 14:55:16.892363 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:55:16.892010 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-ui-tls\"" Sep 09 14:55:16.892521 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:55:16.892056 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-6q4dm\"" Sep 09 14:55:16.900429 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:55:16.900407 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-qpb79"] Sep 09 14:55:16.952587 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:55:16.952553 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/dc917b09-9650-4792-bba2-d9d8dd383d90-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-qpb79\" (UID: \"dc917b09-9650-4792-bba2-d9d8dd383d90\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qpb79" Sep 09 14:55:16.952779 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:55:16.952601 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/dc917b09-9650-4792-bba2-d9d8dd383d90-feast-data\") pod \"feast-simple-feast-setup-57576b8fcf-qpb79\" (UID: \"dc917b09-9650-4792-bba2-d9d8dd383d90\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qpb79" Sep 09 14:55:16.952779 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:55:16.952655 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bht9b\" (UniqueName: \"kubernetes.io/projected/dc917b09-9650-4792-bba2-d9d8dd383d90-kube-api-access-bht9b\") pod \"feast-simple-feast-setup-57576b8fcf-qpb79\" (UID: \"dc917b09-9650-4792-bba2-d9d8dd383d90\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qpb79" Sep 09 14:55:16.952779 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:55:16.952683 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/dc917b09-9650-4792-bba2-d9d8dd383d90-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-qpb79\" (UID: \"dc917b09-9650-4792-bba2-d9d8dd383d90\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qpb79" Sep 09 14:55:16.952779 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:55:16.952714 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/dc917b09-9650-4792-bba2-d9d8dd383d90-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-qpb79\" (UID: \"dc917b09-9650-4792-bba2-d9d8dd383d90\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qpb79" Sep 09 14:55:16.952779 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:55:16.952754 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/dc917b09-9650-4792-bba2-d9d8dd383d90-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-qpb79\" (UID: \"dc917b09-9650-4792-bba2-d9d8dd383d90\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qpb79" Sep 09 14:55:17.053178 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:55:17.053138 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/dc917b09-9650-4792-bba2-d9d8dd383d90-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-qpb79\" (UID: \"dc917b09-9650-4792-bba2-d9d8dd383d90\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qpb79" Sep 09 14:55:17.053391 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:55:17.053193 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/dc917b09-9650-4792-bba2-d9d8dd383d90-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-qpb79\" (UID: \"dc917b09-9650-4792-bba2-d9d8dd383d90\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qpb79" Sep 09 14:55:17.053391 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:55:17.053225 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/dc917b09-9650-4792-bba2-d9d8dd383d90-feast-data\") pod \"feast-simple-feast-setup-57576b8fcf-qpb79\" (UID: \"dc917b09-9650-4792-bba2-d9d8dd383d90\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qpb79" Sep 09 14:55:17.053391 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:55:17.053279 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bht9b\" (UniqueName: \"kubernetes.io/projected/dc917b09-9650-4792-bba2-d9d8dd383d90-kube-api-access-bht9b\") pod \"feast-simple-feast-setup-57576b8fcf-qpb79\" (UID: \"dc917b09-9650-4792-bba2-d9d8dd383d90\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qpb79" Sep 09 14:55:17.053391 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:55:17.053337 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/dc917b09-9650-4792-bba2-d9d8dd383d90-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-qpb79\" (UID: \"dc917b09-9650-4792-bba2-d9d8dd383d90\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qpb79" Sep 09 14:55:17.053391 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:55:17.053369 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/dc917b09-9650-4792-bba2-d9d8dd383d90-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-qpb79\" (UID: \"dc917b09-9650-4792-bba2-d9d8dd383d90\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qpb79" Sep 09 14:55:17.053668 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:55:17.053469 2576 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-registry-tls: secret "feast-simple-feast-setup-registry-tls" not found Sep 09 14:55:17.053668 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:55:17.053506 2576 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-ui-tls: secret "feast-simple-feast-setup-ui-tls" not found Sep 09 14:55:17.053668 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:55:17.053538 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc917b09-9650-4792-bba2-d9d8dd383d90-registry-tls podName:dc917b09-9650-4792-bba2-d9d8dd383d90 nodeName:}" failed. No retries permitted until 2026-09-09 14:55:17.553518318 +0000 UTC m=+1332.564434553 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/dc917b09-9650-4792-bba2-d9d8dd383d90-registry-tls") pod "feast-simple-feast-setup-57576b8fcf-qpb79" (UID: "dc917b09-9650-4792-bba2-d9d8dd383d90") : secret "feast-simple-feast-setup-registry-tls" not found Sep 09 14:55:17.053668 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:55:17.053574 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc917b09-9650-4792-bba2-d9d8dd383d90-ui-tls podName:dc917b09-9650-4792-bba2-d9d8dd383d90 nodeName:}" failed. No retries permitted until 2026-09-09 14:55:17.553557811 +0000 UTC m=+1332.564474047 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "ui-tls" (UniqueName: "kubernetes.io/secret/dc917b09-9650-4792-bba2-d9d8dd383d90-ui-tls") pod "feast-simple-feast-setup-57576b8fcf-qpb79" (UID: "dc917b09-9650-4792-bba2-d9d8dd383d90") : secret "feast-simple-feast-setup-ui-tls" not found Sep 09 14:55:17.053871 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:55:17.053671 2576 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-online-tls: secret "feast-simple-feast-setup-online-tls" not found Sep 09 14:55:17.053871 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:55:17.053740 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc917b09-9650-4792-bba2-d9d8dd383d90-online-tls podName:dc917b09-9650-4792-bba2-d9d8dd383d90 nodeName:}" failed. No retries permitted until 2026-09-09 14:55:17.55372462 +0000 UTC m=+1332.564640862 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/dc917b09-9650-4792-bba2-d9d8dd383d90-online-tls") pod "feast-simple-feast-setup-57576b8fcf-qpb79" (UID: "dc917b09-9650-4792-bba2-d9d8dd383d90") : secret "feast-simple-feast-setup-online-tls" not found Sep 09 14:55:17.053871 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:55:17.053745 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/dc917b09-9650-4792-bba2-d9d8dd383d90-feast-data\") pod \"feast-simple-feast-setup-57576b8fcf-qpb79\" (UID: \"dc917b09-9650-4792-bba2-d9d8dd383d90\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qpb79" Sep 09 14:55:17.055954 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:55:17.055806 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/dc917b09-9650-4792-bba2-d9d8dd383d90-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-qpb79\" (UID: \"dc917b09-9650-4792-bba2-d9d8dd383d90\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qpb79" Sep 09 14:55:17.068800 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:55:17.068773 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bht9b\" (UniqueName: \"kubernetes.io/projected/dc917b09-9650-4792-bba2-d9d8dd383d90-kube-api-access-bht9b\") pod \"feast-simple-feast-setup-57576b8fcf-qpb79\" (UID: \"dc917b09-9650-4792-bba2-d9d8dd383d90\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qpb79" Sep 09 14:55:17.558276 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:55:17.558244 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/dc917b09-9650-4792-bba2-d9d8dd383d90-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-qpb79\" (UID: \"dc917b09-9650-4792-bba2-d9d8dd383d90\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qpb79" Sep 09 14:55:17.558276 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:55:17.558287 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/dc917b09-9650-4792-bba2-d9d8dd383d90-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-qpb79\" (UID: \"dc917b09-9650-4792-bba2-d9d8dd383d90\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qpb79" Sep 09 14:55:17.558581 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:55:17.558367 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/dc917b09-9650-4792-bba2-d9d8dd383d90-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-qpb79\" (UID: \"dc917b09-9650-4792-bba2-d9d8dd383d90\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qpb79" Sep 09 14:55:17.560950 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:55:17.560916 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/dc917b09-9650-4792-bba2-d9d8dd383d90-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-qpb79\" (UID: \"dc917b09-9650-4792-bba2-d9d8dd383d90\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qpb79" Sep 09 14:55:17.560950 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:55:17.560943 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/dc917b09-9650-4792-bba2-d9d8dd383d90-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-qpb79\" (UID: \"dc917b09-9650-4792-bba2-d9d8dd383d90\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qpb79" Sep 09 14:55:17.561107 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:55:17.561026 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/dc917b09-9650-4792-bba2-d9d8dd383d90-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-qpb79\" (UID: \"dc917b09-9650-4792-bba2-d9d8dd383d90\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qpb79" Sep 09 14:55:17.807493 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:55:17.807447 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qpb79" Sep 09 14:55:18.088610 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:55:18.088577 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-qpb79"] Sep 09 14:55:18.089556 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:55:18.089530 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddc917b09_9650_4792_bba2_d9d8dd383d90.slice/crio-b7f1dc3a6d68deb9892a42cbf0e25f2988388fb95730ec48f6283d93f6c4326c WatchSource:0}: Error finding container b7f1dc3a6d68deb9892a42cbf0e25f2988388fb95730ec48f6283d93f6c4326c: Status 404 returned error can't find the container with id b7f1dc3a6d68deb9892a42cbf0e25f2988388fb95730ec48f6283d93f6c4326c Sep 09 14:55:18.091654 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:55:18.091635 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 09 14:55:18.414933 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:55:18.414843 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qpb79" event={"ID":"dc917b09-9650-4792-bba2-d9d8dd383d90","Type":"ContainerStarted","Data":"b7f1dc3a6d68deb9892a42cbf0e25f2988388fb95730ec48f6283d93f6c4326c"} Sep 09 14:55:31.480980 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:55:31.480897 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qpb79" event={"ID":"dc917b09-9650-4792-bba2-d9d8dd383d90","Type":"ContainerStarted","Data":"df2e7aa91a451de44bdc7f15c4c73e5c70983bac1a96bd30a75ae0c534201219"} Sep 09 14:55:36.501444 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:55:36.501357 2576 generic.go:356] "Generic (PLEG): container finished" podID="dc917b09-9650-4792-bba2-d9d8dd383d90" containerID="df2e7aa91a451de44bdc7f15c4c73e5c70983bac1a96bd30a75ae0c534201219" exitCode=0 Sep 09 14:55:36.501444 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:55:36.501434 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qpb79" event={"ID":"dc917b09-9650-4792-bba2-d9d8dd383d90","Type":"ContainerDied","Data":"df2e7aa91a451de44bdc7f15c4c73e5c70983bac1a96bd30a75ae0c534201219"} Sep 09 14:55:37.507944 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:55:37.507906 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qpb79" event={"ID":"dc917b09-9650-4792-bba2-d9d8dd383d90","Type":"ContainerStarted","Data":"c018f725dd73e58a2f31654b68587dca6ccee50c7976ca89e5d19a2c5b00cbde"} Sep 09 14:55:42.530560 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:55:42.530525 2576 generic.go:356] "Generic (PLEG): container finished" podID="dc917b09-9650-4792-bba2-d9d8dd383d90" containerID="c018f725dd73e58a2f31654b68587dca6ccee50c7976ca89e5d19a2c5b00cbde" exitCode=0 Sep 09 14:55:42.530919 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:55:42.530602 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qpb79" event={"ID":"dc917b09-9650-4792-bba2-d9d8dd383d90","Type":"ContainerDied","Data":"c018f725dd73e58a2f31654b68587dca6ccee50c7976ca89e5d19a2c5b00cbde"} Sep 09 14:55:43.540004 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:55:43.539966 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qpb79" event={"ID":"dc917b09-9650-4792-bba2-d9d8dd383d90","Type":"ContainerStarted","Data":"fed01c6fc3d3a0e0161882f28ecc3cd1a9ba13f3629ada9165a7821fc50505a2"} Sep 09 14:55:43.540569 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:55:43.540032 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qpb79" event={"ID":"dc917b09-9650-4792-bba2-d9d8dd383d90","Type":"ContainerStarted","Data":"0457ddaae41225e6b38c8becfd1bf3bad80e45ddf7a666a8f52dd4df6546e20c"} Sep 09 14:55:43.540569 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:55:43.540053 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qpb79" event={"ID":"dc917b09-9650-4792-bba2-d9d8dd383d90","Type":"ContainerStarted","Data":"a7f695bce56ef9257e88f90f57d345af7ff206d38c76a4cd823f99ee51705710"} Sep 09 14:55:43.540569 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:55:43.540072 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qpb79" event={"ID":"dc917b09-9650-4792-bba2-d9d8dd383d90","Type":"ContainerStarted","Data":"5f61835ddeaa08a5cfaa5b531eddde9355d8ef844a527a3655c5e7fbeaf28510"} Sep 09 14:55:43.565125 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:55:43.565066 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qpb79" podStartSLOduration=14.522073625 podStartE2EDuration="27.565042655s" totalImagesPullingTime="13.04296903s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 14:55:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:55:43.563379425 +0000 UTC m=+1358.574295677" watchObservedRunningTime="2026-09-09 14:55:43.565042655 +0000 UTC m=+1358.575958907" Sep 09 14:55:44.809205 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:55:44.808475 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qpb79" Sep 09 14:55:44.809205 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:55:44.808695 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qpb79" Sep 09 14:55:44.809205 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:55:44.808741 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qpb79" Sep 09 14:55:44.809205 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:55:44.808761 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qpb79" Sep 09 14:55:44.810985 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:55:44.810811 2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qpb79" podUID="dc917b09-9650-4792-bba2-d9d8dd383d90" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.25:8443: connect: connection refused" Sep 09 14:55:44.810985 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:55:44.810849 2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qpb79" podUID="dc917b09-9650-4792-bba2-d9d8dd383d90" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.25:8816: connect: connection refused" Sep 09 14:55:44.810985 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:55:44.810869 2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qpb79" podUID="dc917b09-9650-4792-bba2-d9d8dd383d90" containerName="online" probeResult="failure" output="Get \"https://10.132.0.25:6567/health\": dial tcp 10.132.0.25:6567: connect: connection refused" Sep 09 14:55:44.810985 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:55:44.810936 2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qpb79" podUID="dc917b09-9650-4792-bba2-d9d8dd383d90" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.25:6571: connect: connection refused" Sep 09 14:55:47.809352 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:55:47.808433 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qpb79" Sep 09 14:55:47.811426 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:55:47.810529 2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qpb79" podUID="dc917b09-9650-4792-bba2-d9d8dd383d90" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.25:6571: connect: connection refused" Sep 09 14:55:47.811426 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:55:47.811153 2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qpb79" podUID="dc917b09-9650-4792-bba2-d9d8dd383d90" containerName="online" probeResult="failure" output="Get \"https://10.132.0.25:6567/health\": dial tcp 10.132.0.25:6567: connect: connection refused" Sep 09 14:55:47.811426 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:55:47.811350 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qpb79" Sep 09 14:55:47.811426 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:55:47.811370 2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qpb79" podUID="dc917b09-9650-4792-bba2-d9d8dd383d90" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.25:8443: connect: connection refused" Sep 09 14:55:47.811426 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:55:47.811412 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qpb79" Sep 09 14:55:47.811426 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:55:47.811433 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qpb79" Sep 09 14:55:47.811826 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:55:47.811526 2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qpb79" podUID="dc917b09-9650-4792-bba2-d9d8dd383d90" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.25:8816: connect: connection refused" Sep 09 14:55:50.808608 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:55:50.808503 2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qpb79" podUID="dc917b09-9650-4792-bba2-d9d8dd383d90" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.25:8443: connect: connection refused" Sep 09 14:55:50.809428 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:55:50.809404 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qpb79" Sep 09 14:55:50.809577 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:55:50.809505 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qpb79" Sep 09 14:55:50.810136 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:55:50.810115 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qpb79" Sep 09 14:55:50.810441 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:55:50.810420 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qpb79" Sep 09 14:55:50.815041 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:55:50.815023 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qpb79" Sep 09 14:55:51.599056 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:55:51.599025 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qpb79" Sep 09 14:55:53.808552 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:55:53.808515 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qpb79" Sep 09 14:55:54.606803 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:55:54.606772 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qpb79" Sep 09 14:56:57.303743 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:56:57.303658 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-qpb79"] Sep 09 14:56:57.304243 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:56:57.303896 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qpb79" podUID="dc917b09-9650-4792-bba2-d9d8dd383d90" containerName="registry" containerID="cri-o://5f61835ddeaa08a5cfaa5b531eddde9355d8ef844a527a3655c5e7fbeaf28510" gracePeriod=30 Sep 09 14:56:57.304243 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:56:57.303994 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qpb79" podUID="dc917b09-9650-4792-bba2-d9d8dd383d90" containerName="online" containerID="cri-o://a7f695bce56ef9257e88f90f57d345af7ff206d38c76a4cd823f99ee51705710" gracePeriod=30 Sep 09 14:56:57.304243 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:56:57.304000 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qpb79" podUID="dc917b09-9650-4792-bba2-d9d8dd383d90" containerName="offline" containerID="cri-o://0457ddaae41225e6b38c8becfd1bf3bad80e45ddf7a666a8f52dd4df6546e20c" gracePeriod=30 Sep 09 14:56:57.304243 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:56:57.304000 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qpb79" podUID="dc917b09-9650-4792-bba2-d9d8dd383d90" containerName="ui" containerID="cri-o://fed01c6fc3d3a0e0161882f28ecc3cd1a9ba13f3629ada9165a7821fc50505a2" gracePeriod=30 Sep 09 14:56:57.720056 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:56:57.719953 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-zggqz"] Sep 09 14:56:57.724241 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:56:57.724217 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zggqz" Sep 09 14:56:57.727112 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:56:57.727090 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-l9cgc\"" Sep 09 14:56:57.736959 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:56:57.736889 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-zggqz"] Sep 09 14:56:57.800867 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:56:57.800814 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/6bfe4cb5-2f10-42aa-9141-7d128b9c6e91-feast-data\") pod \"feast-simple-feast-setup-57576b8fcf-zggqz\" (UID: \"6bfe4cb5-2f10-42aa-9141-7d128b9c6e91\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zggqz" Sep 09 14:56:57.801072 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:56:57.800880 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/6bfe4cb5-2f10-42aa-9141-7d128b9c6e91-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-zggqz\" (UID: \"6bfe4cb5-2f10-42aa-9141-7d128b9c6e91\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zggqz" Sep 09 14:56:57.801072 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:56:57.800920 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/6bfe4cb5-2f10-42aa-9141-7d128b9c6e91-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-zggqz\" (UID: \"6bfe4cb5-2f10-42aa-9141-7d128b9c6e91\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zggqz" Sep 09 14:56:57.801072 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:56:57.800968 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/6bfe4cb5-2f10-42aa-9141-7d128b9c6e91-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-zggqz\" (UID: \"6bfe4cb5-2f10-42aa-9141-7d128b9c6e91\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zggqz" Sep 09 14:56:57.801072 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:56:57.800993 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpd8j\" (UniqueName: \"kubernetes.io/projected/6bfe4cb5-2f10-42aa-9141-7d128b9c6e91-kube-api-access-gpd8j\") pod \"feast-simple-feast-setup-57576b8fcf-zggqz\" (UID: \"6bfe4cb5-2f10-42aa-9141-7d128b9c6e91\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zggqz" Sep 09 14:56:57.801072 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:56:57.801029 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/6bfe4cb5-2f10-42aa-9141-7d128b9c6e91-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-zggqz\" (UID: \"6bfe4cb5-2f10-42aa-9141-7d128b9c6e91\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zggqz" Sep 09 14:56:57.901729 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:56:57.901690 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/6bfe4cb5-2f10-42aa-9141-7d128b9c6e91-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-zggqz\" (UID: \"6bfe4cb5-2f10-42aa-9141-7d128b9c6e91\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zggqz" Sep 09 14:56:57.901943 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:56:57.901772 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/6bfe4cb5-2f10-42aa-9141-7d128b9c6e91-feast-data\") pod \"feast-simple-feast-setup-57576b8fcf-zggqz\" (UID: \"6bfe4cb5-2f10-42aa-9141-7d128b9c6e91\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zggqz" Sep 09 14:56:57.901943 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:56:57.901811 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/6bfe4cb5-2f10-42aa-9141-7d128b9c6e91-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-zggqz\" (UID: \"6bfe4cb5-2f10-42aa-9141-7d128b9c6e91\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zggqz" Sep 09 14:56:57.902732 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:56:57.902287 2576 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-offline-tls: secret "feast-simple-feast-setup-offline-tls" not found Sep 09 14:56:57.902732 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:56:57.902290 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/6bfe4cb5-2f10-42aa-9141-7d128b9c6e91-feast-data\") pod \"feast-simple-feast-setup-57576b8fcf-zggqz\" (UID: \"6bfe4cb5-2f10-42aa-9141-7d128b9c6e91\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zggqz" Sep 09 14:56:57.902732 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:56:57.902381 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6bfe4cb5-2f10-42aa-9141-7d128b9c6e91-offline-tls podName:6bfe4cb5-2f10-42aa-9141-7d128b9c6e91 nodeName:}" failed. No retries permitted until 2026-09-09 14:56:58.402357669 +0000 UTC m=+1433.413273901 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/6bfe4cb5-2f10-42aa-9141-7d128b9c6e91-offline-tls") pod "feast-simple-feast-setup-57576b8fcf-zggqz" (UID: "6bfe4cb5-2f10-42aa-9141-7d128b9c6e91") : secret "feast-simple-feast-setup-offline-tls" not found Sep 09 14:56:57.902732 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:56:57.902379 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/6bfe4cb5-2f10-42aa-9141-7d128b9c6e91-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-zggqz\" (UID: \"6bfe4cb5-2f10-42aa-9141-7d128b9c6e91\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zggqz" Sep 09 14:56:57.902732 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:56:57.902469 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/6bfe4cb5-2f10-42aa-9141-7d128b9c6e91-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-zggqz\" (UID: \"6bfe4cb5-2f10-42aa-9141-7d128b9c6e91\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zggqz" Sep 09 14:56:57.902732 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:56:57.902500 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpd8j\" (UniqueName: \"kubernetes.io/projected/6bfe4cb5-2f10-42aa-9141-7d128b9c6e91-kube-api-access-gpd8j\") pod \"feast-simple-feast-setup-57576b8fcf-zggqz\" (UID: \"6bfe4cb5-2f10-42aa-9141-7d128b9c6e91\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zggqz" Sep 09 14:56:57.902732 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:56:57.902669 2576 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-online-tls: secret "feast-simple-feast-setup-online-tls" not found Sep 09 14:56:57.902732 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:56:57.902711 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6bfe4cb5-2f10-42aa-9141-7d128b9c6e91-online-tls podName:6bfe4cb5-2f10-42aa-9141-7d128b9c6e91 nodeName:}" failed. No retries permitted until 2026-09-09 14:56:58.402697872 +0000 UTC m=+1433.413614122 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/6bfe4cb5-2f10-42aa-9141-7d128b9c6e91-online-tls") pod "feast-simple-feast-setup-57576b8fcf-zggqz" (UID: "6bfe4cb5-2f10-42aa-9141-7d128b9c6e91") : secret "feast-simple-feast-setup-online-tls" not found Sep 09 14:56:57.905011 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:56:57.904986 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/6bfe4cb5-2f10-42aa-9141-7d128b9c6e91-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-zggqz\" (UID: \"6bfe4cb5-2f10-42aa-9141-7d128b9c6e91\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zggqz" Sep 09 14:56:57.906690 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:56:57.906657 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/6bfe4cb5-2f10-42aa-9141-7d128b9c6e91-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-zggqz\" (UID: \"6bfe4cb5-2f10-42aa-9141-7d128b9c6e91\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zggqz" Sep 09 14:56:57.914544 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:56:57.914518 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpd8j\" (UniqueName: \"kubernetes.io/projected/6bfe4cb5-2f10-42aa-9141-7d128b9c6e91-kube-api-access-gpd8j\") pod \"feast-simple-feast-setup-57576b8fcf-zggqz\" (UID: \"6bfe4cb5-2f10-42aa-9141-7d128b9c6e91\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zggqz" Sep 09 14:56:58.408107 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:56:58.408066 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/6bfe4cb5-2f10-42aa-9141-7d128b9c6e91-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-zggqz\" (UID: \"6bfe4cb5-2f10-42aa-9141-7d128b9c6e91\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zggqz" Sep 09 14:56:58.408611 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:56:58.408120 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/6bfe4cb5-2f10-42aa-9141-7d128b9c6e91-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-zggqz\" (UID: \"6bfe4cb5-2f10-42aa-9141-7d128b9c6e91\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zggqz" Sep 09 14:56:58.410667 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:56:58.410645 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/6bfe4cb5-2f10-42aa-9141-7d128b9c6e91-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-zggqz\" (UID: \"6bfe4cb5-2f10-42aa-9141-7d128b9c6e91\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zggqz" Sep 09 14:56:58.410728 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:56:58.410686 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/6bfe4cb5-2f10-42aa-9141-7d128b9c6e91-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-zggqz\" (UID: \"6bfe4cb5-2f10-42aa-9141-7d128b9c6e91\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zggqz" Sep 09 14:56:58.637674 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:56:58.637641 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zggqz" Sep 09 14:56:58.852358 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:56:58.852317 2576 generic.go:356] "Generic (PLEG): container finished" podID="dc917b09-9650-4792-bba2-d9d8dd383d90" containerID="fed01c6fc3d3a0e0161882f28ecc3cd1a9ba13f3629ada9165a7821fc50505a2" exitCode=0 Sep 09 14:56:58.852358 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:56:58.852348 2576 generic.go:356] "Generic (PLEG): container finished" podID="dc917b09-9650-4792-bba2-d9d8dd383d90" containerID="a7f695bce56ef9257e88f90f57d345af7ff206d38c76a4cd823f99ee51705710" exitCode=0 Sep 09 14:56:58.852555 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:56:58.852339 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qpb79" event={"ID":"dc917b09-9650-4792-bba2-d9d8dd383d90","Type":"ContainerDied","Data":"fed01c6fc3d3a0e0161882f28ecc3cd1a9ba13f3629ada9165a7821fc50505a2"} Sep 09 14:56:58.852555 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:56:58.852439 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qpb79" event={"ID":"dc917b09-9650-4792-bba2-d9d8dd383d90","Type":"ContainerDied","Data":"a7f695bce56ef9257e88f90f57d345af7ff206d38c76a4cd823f99ee51705710"} Sep 09 14:56:58.922950 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:56:58.922918 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-zggqz"] Sep 09 14:56:58.923785 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:56:58.923747 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6bfe4cb5_2f10_42aa_9141_7d128b9c6e91.slice/crio-7b57660aa3531411922cb2bec2ed145b332ca9cc58abca1f32662da61123cf4b WatchSource:0}: Error finding container 7b57660aa3531411922cb2bec2ed145b332ca9cc58abca1f32662da61123cf4b: Status 404 returned error can't find the container with id 7b57660aa3531411922cb2bec2ed145b332ca9cc58abca1f32662da61123cf4b Sep 09 14:56:59.858315 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:56:59.858276 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zggqz" event={"ID":"6bfe4cb5-2f10-42aa-9141-7d128b9c6e91","Type":"ContainerStarted","Data":"3c49a305cc19526431f56b71191839c150c85bc078e1b79d56d7a41ccf07313b"} Sep 09 14:56:59.858697 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:56:59.858345 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zggqz" event={"ID":"6bfe4cb5-2f10-42aa-9141-7d128b9c6e91","Type":"ContainerStarted","Data":"7b57660aa3531411922cb2bec2ed145b332ca9cc58abca1f32662da61123cf4b"} Sep 09 14:57:01.597025 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:57:01.596980 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qpb79" podUID="dc917b09-9650-4792-bba2-d9d8dd383d90" containerName="online" probeResult="failure" output="Get \"https://10.132.0.25:6567/health\": dial tcp 10.132.0.25:6567: connect: connection refused" Sep 09 14:57:03.877166 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:57:03.877134 2576 generic.go:356] "Generic (PLEG): container finished" podID="6bfe4cb5-2f10-42aa-9141-7d128b9c6e91" containerID="3c49a305cc19526431f56b71191839c150c85bc078e1b79d56d7a41ccf07313b" exitCode=0 Sep 09 14:57:03.877612 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:57:03.877181 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zggqz" event={"ID":"6bfe4cb5-2f10-42aa-9141-7d128b9c6e91","Type":"ContainerDied","Data":"3c49a305cc19526431f56b71191839c150c85bc078e1b79d56d7a41ccf07313b"} Sep 09 14:57:04.607115 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:57:04.607066 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qpb79" podUID="dc917b09-9650-4792-bba2-d9d8dd383d90" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.25:8443: connect: connection refused" Sep 09 14:57:04.883390 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:57:04.883286 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zggqz" event={"ID":"6bfe4cb5-2f10-42aa-9141-7d128b9c6e91","Type":"ContainerStarted","Data":"b73910956d643dea0ef518d0db5b114844de8c3fb1e0bd4ecd1021bd7566a2ab"} Sep 09 14:57:09.913275 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:57:09.913238 2576 generic.go:356] "Generic (PLEG): container finished" podID="6bfe4cb5-2f10-42aa-9141-7d128b9c6e91" containerID="b73910956d643dea0ef518d0db5b114844de8c3fb1e0bd4ecd1021bd7566a2ab" exitCode=0 Sep 09 14:57:09.913657 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:57:09.913324 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zggqz" event={"ID":"6bfe4cb5-2f10-42aa-9141-7d128b9c6e91","Type":"ContainerDied","Data":"b73910956d643dea0ef518d0db5b114844de8c3fb1e0bd4ecd1021bd7566a2ab"} Sep 09 14:57:10.923783 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:57:10.923739 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zggqz" event={"ID":"6bfe4cb5-2f10-42aa-9141-7d128b9c6e91","Type":"ContainerStarted","Data":"d4f0f616273c23dee12d55da52dac30a96838a033d3853d72d3d6189fbc06a3c"} Sep 09 14:57:10.924327 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:57:10.923803 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zggqz" event={"ID":"6bfe4cb5-2f10-42aa-9141-7d128b9c6e91","Type":"ContainerStarted","Data":"914536ff7cedd943bd070f48dbe944ad53329b25d427c687e6aab4cdf3d0ac5a"} Sep 09 14:57:10.924327 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:57:10.923823 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zggqz" event={"ID":"6bfe4cb5-2f10-42aa-9141-7d128b9c6e91","Type":"ContainerStarted","Data":"fe803cc52e67f746b7ab2c3c7225aaec5f301747116bbd90ae86dbef66580e3d"} Sep 09 14:57:10.924327 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:57:10.923842 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zggqz" event={"ID":"6bfe4cb5-2f10-42aa-9141-7d128b9c6e91","Type":"ContainerStarted","Data":"db9c30bdde2d6c55b15ae4f3b8da6b9e38c2db3e8481e3470b661db21b05c3fe"} Sep 09 14:57:10.951829 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:57:10.951764 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zggqz" podStartSLOduration=13.951741405 podStartE2EDuration="13.951741405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 14:56:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:57:10.950031591 +0000 UTC m=+1445.960947867" watchObservedRunningTime="2026-09-09 14:57:10.951741405 +0000 UTC m=+1445.962657656" Sep 09 14:57:11.597204 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:57:11.597153 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qpb79" podUID="dc917b09-9650-4792-bba2-d9d8dd383d90" containerName="online" probeResult="failure" output="Get \"https://10.132.0.25:6567/health\": dial tcp 10.132.0.25:6567: connect: connection refused" Sep 09 14:57:13.638482 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:57:13.638444 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zggqz" Sep 09 14:57:13.639071 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:57:13.639047 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zggqz" Sep 09 14:57:13.639071 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:57:13.639081 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zggqz" Sep 09 14:57:13.639270 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:57:13.639100 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zggqz" Sep 09 14:57:13.640569 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:57:13.640529 2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zggqz" podUID="6bfe4cb5-2f10-42aa-9141-7d128b9c6e91" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.26:8443: connect: connection refused" Sep 09 14:57:13.640569 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:57:13.640553 2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zggqz" podUID="6bfe4cb5-2f10-42aa-9141-7d128b9c6e91" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.26:6571: connect: connection refused" Sep 09 14:57:13.640741 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:57:13.640585 2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zggqz" podUID="6bfe4cb5-2f10-42aa-9141-7d128b9c6e91" containerName="online" probeResult="failure" output="Get \"https://10.132.0.26:6567/health\": dial tcp 10.132.0.26:6567: connect: connection refused" Sep 09 14:57:13.640741 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:57:13.640674 2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zggqz" podUID="6bfe4cb5-2f10-42aa-9141-7d128b9c6e91" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.26:8816: connect: connection refused" Sep 09 14:57:14.606470 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:57:14.606419 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qpb79" podUID="dc917b09-9650-4792-bba2-d9d8dd383d90" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.25:8443: connect: connection refused" Sep 09 14:57:16.638840 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:57:16.638682 2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zggqz" podUID="6bfe4cb5-2f10-42aa-9141-7d128b9c6e91" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.26:8816: connect: connection refused" Sep 09 14:57:16.638840 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:57:16.638747 2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zggqz" podUID="6bfe4cb5-2f10-42aa-9141-7d128b9c6e91" containerName="online" probeResult="failure" output="Get \"https://10.132.0.26:6567/health\": dial tcp 10.132.0.26:6567: connect: connection refused" Sep 09 14:57:16.638840 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:57:16.638756 2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zggqz" podUID="6bfe4cb5-2f10-42aa-9141-7d128b9c6e91" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.26:6571: connect: connection refused" Sep 09 14:57:16.638840 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:57:16.638687 2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zggqz" podUID="6bfe4cb5-2f10-42aa-9141-7d128b9c6e91" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.26:8443: connect: connection refused" Sep 09 14:57:18.638057 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:57:18.638017 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zggqz" Sep 09 14:57:18.638490 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:57:18.638101 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zggqz" Sep 09 14:57:18.638490 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:57:18.638143 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zggqz" Sep 09 14:57:18.638490 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:57:18.638163 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zggqz" Sep 09 14:57:19.638801 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:57:19.638765 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zggqz" Sep 09 14:57:19.639258 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:57:19.639074 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zggqz" Sep 09 14:57:19.639258 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:57:19.639191 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zggqz" Sep 09 14:57:19.644076 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:57:19.644054 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zggqz" Sep 09 14:57:19.984278 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:57:19.984187 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zggqz" Sep 09 14:57:19.984278 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:57:19.984272 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zggqz" Sep 09 14:57:19.984489 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:57:19.984360 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zggqz" Sep 09 14:57:19.987038 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:57:19.987021 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zggqz" Sep 09 14:57:21.596272 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:57:21.596230 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qpb79" podUID="dc917b09-9650-4792-bba2-d9d8dd383d90" containerName="online" probeResult="failure" output="Get \"https://10.132.0.25:6567/health\": dial tcp 10.132.0.25:6567: connect: connection refused" Sep 09 14:57:21.596691 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:57:21.596385 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qpb79" Sep 09 14:57:24.606370 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:57:24.606287 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qpb79" podUID="dc917b09-9650-4792-bba2-d9d8dd383d90" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.25:8443: connect: connection refused" Sep 09 14:57:24.606791 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:57:24.606468 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qpb79" Sep 09 14:57:27.959349 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:57:27.959323 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qpb79" Sep 09 14:57:28.019969 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:57:28.019935 2576 generic.go:356] "Generic (PLEG): container finished" podID="dc917b09-9650-4792-bba2-d9d8dd383d90" containerID="0457ddaae41225e6b38c8becfd1bf3bad80e45ddf7a666a8f52dd4df6546e20c" exitCode=137 Sep 09 14:57:28.019969 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:57:28.019964 2576 generic.go:356] "Generic (PLEG): container finished" podID="dc917b09-9650-4792-bba2-d9d8dd383d90" containerID="5f61835ddeaa08a5cfaa5b531eddde9355d8ef844a527a3655c5e7fbeaf28510" exitCode=137 Sep 09 14:57:28.020186 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:57:28.020013 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qpb79" Sep 09 14:57:28.020186 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:57:28.020011 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qpb79" event={"ID":"dc917b09-9650-4792-bba2-d9d8dd383d90","Type":"ContainerDied","Data":"0457ddaae41225e6b38c8becfd1bf3bad80e45ddf7a666a8f52dd4df6546e20c"} Sep 09 14:57:28.020186 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:57:28.020137 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qpb79" event={"ID":"dc917b09-9650-4792-bba2-d9d8dd383d90","Type":"ContainerDied","Data":"5f61835ddeaa08a5cfaa5b531eddde9355d8ef844a527a3655c5e7fbeaf28510"} Sep 09 14:57:28.020186 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:57:28.020154 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qpb79" event={"ID":"dc917b09-9650-4792-bba2-d9d8dd383d90","Type":"ContainerDied","Data":"b7f1dc3a6d68deb9892a42cbf0e25f2988388fb95730ec48f6283d93f6c4326c"} Sep 09 14:57:28.020186 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:57:28.020172 2576 scope.go:117] "RemoveContainer" containerID="fed01c6fc3d3a0e0161882f28ecc3cd1a9ba13f3629ada9165a7821fc50505a2" Sep 09 14:57:28.028454 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:57:28.028436 2576 scope.go:117] "RemoveContainer" containerID="0457ddaae41225e6b38c8becfd1bf3bad80e45ddf7a666a8f52dd4df6546e20c" Sep 09 14:57:28.036701 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:57:28.036682 2576 scope.go:117] "RemoveContainer" containerID="a7f695bce56ef9257e88f90f57d345af7ff206d38c76a4cd823f99ee51705710" Sep 09 14:57:28.044743 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:57:28.044721 2576 scope.go:117] "RemoveContainer" containerID="5f61835ddeaa08a5cfaa5b531eddde9355d8ef844a527a3655c5e7fbeaf28510" Sep 09 14:57:28.052957 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:57:28.052936 2576 scope.go:117] "RemoveContainer" containerID="c018f725dd73e58a2f31654b68587dca6ccee50c7976ca89e5d19a2c5b00cbde" Sep 09 14:57:28.060726 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:57:28.060706 2576 scope.go:117] "RemoveContainer" containerID="df2e7aa91a451de44bdc7f15c4c73e5c70983bac1a96bd30a75ae0c534201219" Sep 09 14:57:28.068433 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:57:28.068412 2576 scope.go:117] "RemoveContainer" containerID="fed01c6fc3d3a0e0161882f28ecc3cd1a9ba13f3629ada9165a7821fc50505a2" Sep 09 14:57:28.068735 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:57:28.068718 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fed01c6fc3d3a0e0161882f28ecc3cd1a9ba13f3629ada9165a7821fc50505a2\": container with ID starting with fed01c6fc3d3a0e0161882f28ecc3cd1a9ba13f3629ada9165a7821fc50505a2 not found: ID does not exist" containerID="fed01c6fc3d3a0e0161882f28ecc3cd1a9ba13f3629ada9165a7821fc50505a2" Sep 09 14:57:28.068782 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:57:28.068744 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fed01c6fc3d3a0e0161882f28ecc3cd1a9ba13f3629ada9165a7821fc50505a2"} err="failed to get container status \"fed01c6fc3d3a0e0161882f28ecc3cd1a9ba13f3629ada9165a7821fc50505a2\": rpc error: code = NotFound desc = could not find container \"fed01c6fc3d3a0e0161882f28ecc3cd1a9ba13f3629ada9165a7821fc50505a2\": container with ID starting with fed01c6fc3d3a0e0161882f28ecc3cd1a9ba13f3629ada9165a7821fc50505a2 not found: ID does not exist" Sep 09 14:57:28.068782 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:57:28.068765 2576 scope.go:117] "RemoveContainer" containerID="0457ddaae41225e6b38c8becfd1bf3bad80e45ddf7a666a8f52dd4df6546e20c" Sep 09 14:57:28.069039 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:57:28.069020 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0457ddaae41225e6b38c8becfd1bf3bad80e45ddf7a666a8f52dd4df6546e20c\": container with ID starting with 0457ddaae41225e6b38c8becfd1bf3bad80e45ddf7a666a8f52dd4df6546e20c not found: ID does not exist" containerID="0457ddaae41225e6b38c8becfd1bf3bad80e45ddf7a666a8f52dd4df6546e20c" Sep 09 14:57:28.069100 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:57:28.069045 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0457ddaae41225e6b38c8becfd1bf3bad80e45ddf7a666a8f52dd4df6546e20c"} err="failed to get container status \"0457ddaae41225e6b38c8becfd1bf3bad80e45ddf7a666a8f52dd4df6546e20c\": rpc error: code = NotFound desc = could not find container \"0457ddaae41225e6b38c8becfd1bf3bad80e45ddf7a666a8f52dd4df6546e20c\": container with ID starting with 0457ddaae41225e6b38c8becfd1bf3bad80e45ddf7a666a8f52dd4df6546e20c not found: ID does not exist" Sep 09 14:57:28.069100 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:57:28.069059 2576 scope.go:117] "RemoveContainer" containerID="a7f695bce56ef9257e88f90f57d345af7ff206d38c76a4cd823f99ee51705710" Sep 09 14:57:28.069313 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:57:28.069282 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7f695bce56ef9257e88f90f57d345af7ff206d38c76a4cd823f99ee51705710\": container with ID starting with a7f695bce56ef9257e88f90f57d345af7ff206d38c76a4cd823f99ee51705710 not found: ID does not exist" containerID="a7f695bce56ef9257e88f90f57d345af7ff206d38c76a4cd823f99ee51705710" Sep 09 14:57:28.069378 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:57:28.069318 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7f695bce56ef9257e88f90f57d345af7ff206d38c76a4cd823f99ee51705710"} err="failed to get container status \"a7f695bce56ef9257e88f90f57d345af7ff206d38c76a4cd823f99ee51705710\": rpc error: code = NotFound desc = could not find container \"a7f695bce56ef9257e88f90f57d345af7ff206d38c76a4cd823f99ee51705710\": container with ID starting with a7f695bce56ef9257e88f90f57d345af7ff206d38c76a4cd823f99ee51705710 not found: ID does not exist" Sep 09 14:57:28.069378 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:57:28.069332 2576 scope.go:117] "RemoveContainer" containerID="5f61835ddeaa08a5cfaa5b531eddde9355d8ef844a527a3655c5e7fbeaf28510" Sep 09 14:57:28.069593 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:57:28.069574 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f61835ddeaa08a5cfaa5b531eddde9355d8ef844a527a3655c5e7fbeaf28510\": container with ID starting with 5f61835ddeaa08a5cfaa5b531eddde9355d8ef844a527a3655c5e7fbeaf28510 not found: ID does not exist" containerID="5f61835ddeaa08a5cfaa5b531eddde9355d8ef844a527a3655c5e7fbeaf28510" Sep 09 14:57:28.069647 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:57:28.069599 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f61835ddeaa08a5cfaa5b531eddde9355d8ef844a527a3655c5e7fbeaf28510"} err="failed to get container status \"5f61835ddeaa08a5cfaa5b531eddde9355d8ef844a527a3655c5e7fbeaf28510\": rpc error: code = NotFound desc = could not find container \"5f61835ddeaa08a5cfaa5b531eddde9355d8ef844a527a3655c5e7fbeaf28510\": container with ID starting with 5f61835ddeaa08a5cfaa5b531eddde9355d8ef844a527a3655c5e7fbeaf28510 not found: ID does not exist" Sep 09 14:57:28.069647 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:57:28.069614 2576 scope.go:117] "RemoveContainer" containerID="c018f725dd73e58a2f31654b68587dca6ccee50c7976ca89e5d19a2c5b00cbde" Sep 09 14:57:28.069923 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:57:28.069884 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c018f725dd73e58a2f31654b68587dca6ccee50c7976ca89e5d19a2c5b00cbde\": container with ID starting with c018f725dd73e58a2f31654b68587dca6ccee50c7976ca89e5d19a2c5b00cbde not found: ID does not exist" containerID="c018f725dd73e58a2f31654b68587dca6ccee50c7976ca89e5d19a2c5b00cbde" Sep 09 14:57:28.069923 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:57:28.069906 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c018f725dd73e58a2f31654b68587dca6ccee50c7976ca89e5d19a2c5b00cbde"} err="failed to get container status \"c018f725dd73e58a2f31654b68587dca6ccee50c7976ca89e5d19a2c5b00cbde\": rpc error: code = NotFound desc = could not find container \"c018f725dd73e58a2f31654b68587dca6ccee50c7976ca89e5d19a2c5b00cbde\": container with ID starting with c018f725dd73e58a2f31654b68587dca6ccee50c7976ca89e5d19a2c5b00cbde not found: ID does not exist" Sep 09 14:57:28.069923 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:57:28.069922 2576 scope.go:117] "RemoveContainer" containerID="df2e7aa91a451de44bdc7f15c4c73e5c70983bac1a96bd30a75ae0c534201219" Sep 09 14:57:28.070149 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:57:28.070132 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df2e7aa91a451de44bdc7f15c4c73e5c70983bac1a96bd30a75ae0c534201219\": container with ID starting with df2e7aa91a451de44bdc7f15c4c73e5c70983bac1a96bd30a75ae0c534201219 not found: ID does not exist" containerID="df2e7aa91a451de44bdc7f15c4c73e5c70983bac1a96bd30a75ae0c534201219" Sep 09 14:57:28.070203 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:57:28.070152 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df2e7aa91a451de44bdc7f15c4c73e5c70983bac1a96bd30a75ae0c534201219"} err="failed to get container status \"df2e7aa91a451de44bdc7f15c4c73e5c70983bac1a96bd30a75ae0c534201219\": rpc error: code = NotFound desc = could not find container \"df2e7aa91a451de44bdc7f15c4c73e5c70983bac1a96bd30a75ae0c534201219\": container with ID starting with df2e7aa91a451de44bdc7f15c4c73e5c70983bac1a96bd30a75ae0c534201219 not found: ID does not exist" Sep 09 14:57:28.070203 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:57:28.070177 2576 scope.go:117] "RemoveContainer" containerID="fed01c6fc3d3a0e0161882f28ecc3cd1a9ba13f3629ada9165a7821fc50505a2" Sep 09 14:57:28.070438 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:57:28.070420 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fed01c6fc3d3a0e0161882f28ecc3cd1a9ba13f3629ada9165a7821fc50505a2"} err="failed to get container status \"fed01c6fc3d3a0e0161882f28ecc3cd1a9ba13f3629ada9165a7821fc50505a2\": rpc error: code = NotFound desc = could not find container \"fed01c6fc3d3a0e0161882f28ecc3cd1a9ba13f3629ada9165a7821fc50505a2\": container with ID starting with fed01c6fc3d3a0e0161882f28ecc3cd1a9ba13f3629ada9165a7821fc50505a2 not found: ID does not exist" Sep 09 14:57:28.070495 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:57:28.070438 2576 scope.go:117] "RemoveContainer" containerID="0457ddaae41225e6b38c8becfd1bf3bad80e45ddf7a666a8f52dd4df6546e20c" Sep 09 14:57:28.070690 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:57:28.070672 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0457ddaae41225e6b38c8becfd1bf3bad80e45ddf7a666a8f52dd4df6546e20c"} err="failed to get container status \"0457ddaae41225e6b38c8becfd1bf3bad80e45ddf7a666a8f52dd4df6546e20c\": rpc error: code = NotFound desc = could not find container \"0457ddaae41225e6b38c8becfd1bf3bad80e45ddf7a666a8f52dd4df6546e20c\": container with ID starting with 0457ddaae41225e6b38c8becfd1bf3bad80e45ddf7a666a8f52dd4df6546e20c not found: ID does not exist" Sep 09 14:57:28.070744 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:57:28.070691 2576 scope.go:117] "RemoveContainer" containerID="a7f695bce56ef9257e88f90f57d345af7ff206d38c76a4cd823f99ee51705710" Sep 09 14:57:28.070899 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:57:28.070883 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7f695bce56ef9257e88f90f57d345af7ff206d38c76a4cd823f99ee51705710"} err="failed to get container status \"a7f695bce56ef9257e88f90f57d345af7ff206d38c76a4cd823f99ee51705710\": rpc error: code = NotFound desc = could not find container \"a7f695bce56ef9257e88f90f57d345af7ff206d38c76a4cd823f99ee51705710\": container with ID starting with a7f695bce56ef9257e88f90f57d345af7ff206d38c76a4cd823f99ee51705710 not found: ID does not exist" Sep 09 14:57:28.070937 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:57:28.070900 2576 scope.go:117] "RemoveContainer" containerID="5f61835ddeaa08a5cfaa5b531eddde9355d8ef844a527a3655c5e7fbeaf28510" Sep 09 14:57:28.071107 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:57:28.071090 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f61835ddeaa08a5cfaa5b531eddde9355d8ef844a527a3655c5e7fbeaf28510"} err="failed to get container status \"5f61835ddeaa08a5cfaa5b531eddde9355d8ef844a527a3655c5e7fbeaf28510\": rpc error: code = NotFound desc = could not find container \"5f61835ddeaa08a5cfaa5b531eddde9355d8ef844a527a3655c5e7fbeaf28510\": container with ID starting with 5f61835ddeaa08a5cfaa5b531eddde9355d8ef844a527a3655c5e7fbeaf28510 not found: ID does not exist" Sep 09 14:57:28.071107 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:57:28.071106 2576 scope.go:117] "RemoveContainer" containerID="c018f725dd73e58a2f31654b68587dca6ccee50c7976ca89e5d19a2c5b00cbde" Sep 09 14:57:28.071313 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:57:28.071285 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c018f725dd73e58a2f31654b68587dca6ccee50c7976ca89e5d19a2c5b00cbde"} err="failed to get container status \"c018f725dd73e58a2f31654b68587dca6ccee50c7976ca89e5d19a2c5b00cbde\": rpc error: code = NotFound desc = could not find container \"c018f725dd73e58a2f31654b68587dca6ccee50c7976ca89e5d19a2c5b00cbde\": container with ID starting with c018f725dd73e58a2f31654b68587dca6ccee50c7976ca89e5d19a2c5b00cbde not found: ID does not exist" Sep 09 14:57:28.071353 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:57:28.071313 2576 scope.go:117] "RemoveContainer" containerID="df2e7aa91a451de44bdc7f15c4c73e5c70983bac1a96bd30a75ae0c534201219" Sep 09 14:57:28.071517 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:57:28.071503 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df2e7aa91a451de44bdc7f15c4c73e5c70983bac1a96bd30a75ae0c534201219"} err="failed to get container status \"df2e7aa91a451de44bdc7f15c4c73e5c70983bac1a96bd30a75ae0c534201219\": rpc error: code = NotFound desc = could not find container \"df2e7aa91a451de44bdc7f15c4c73e5c70983bac1a96bd30a75ae0c534201219\": container with ID starting with df2e7aa91a451de44bdc7f15c4c73e5c70983bac1a96bd30a75ae0c534201219 not found: ID does not exist" Sep 09 14:57:28.077765 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:57:28.077742 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/dc917b09-9650-4792-bba2-d9d8dd383d90-feast-data\") pod \"dc917b09-9650-4792-bba2-d9d8dd383d90\" (UID: \"dc917b09-9650-4792-bba2-d9d8dd383d90\") " Sep 09 14:57:28.077835 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:57:28.077778 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bht9b\" (UniqueName: \"kubernetes.io/projected/dc917b09-9650-4792-bba2-d9d8dd383d90-kube-api-access-bht9b\") pod \"dc917b09-9650-4792-bba2-d9d8dd383d90\" (UID: \"dc917b09-9650-4792-bba2-d9d8dd383d90\") " Sep 09 14:57:28.077835 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:57:28.077805 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/dc917b09-9650-4792-bba2-d9d8dd383d90-registry-tls\") pod \"dc917b09-9650-4792-bba2-d9d8dd383d90\" (UID: \"dc917b09-9650-4792-bba2-d9d8dd383d90\") " Sep 09 14:57:28.077902 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:57:28.077869 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/dc917b09-9650-4792-bba2-d9d8dd383d90-online-tls\") pod \"dc917b09-9650-4792-bba2-d9d8dd383d90\" (UID: \"dc917b09-9650-4792-bba2-d9d8dd383d90\") " Sep 09 14:57:28.077902 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:57:28.077889 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/dc917b09-9650-4792-bba2-d9d8dd383d90-offline-tls\") pod \"dc917b09-9650-4792-bba2-d9d8dd383d90\" (UID: \"dc917b09-9650-4792-bba2-d9d8dd383d90\") " Sep 09 14:57:28.077995 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:57:28.077937 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/dc917b09-9650-4792-bba2-d9d8dd383d90-ui-tls\") pod \"dc917b09-9650-4792-bba2-d9d8dd383d90\" (UID: \"dc917b09-9650-4792-bba2-d9d8dd383d90\") " Sep 09 14:57:28.078374 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:57:28.078338 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc917b09-9650-4792-bba2-d9d8dd383d90-feast-data" (OuterVolumeSpecName: "feast-data") pod "dc917b09-9650-4792-bba2-d9d8dd383d90" (UID: "dc917b09-9650-4792-bba2-d9d8dd383d90"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 14:57:28.080349 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:57:28.080288 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc917b09-9650-4792-bba2-d9d8dd383d90-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "dc917b09-9650-4792-bba2-d9d8dd383d90" (UID: "dc917b09-9650-4792-bba2-d9d8dd383d90"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 14:57:28.080349 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:57:28.080338 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc917b09-9650-4792-bba2-d9d8dd383d90-kube-api-access-bht9b" (OuterVolumeSpecName: "kube-api-access-bht9b") pod "dc917b09-9650-4792-bba2-d9d8dd383d90" (UID: "dc917b09-9650-4792-bba2-d9d8dd383d90"). InnerVolumeSpecName "kube-api-access-bht9b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 14:57:28.080482 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:57:28.080375 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc917b09-9650-4792-bba2-d9d8dd383d90-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "dc917b09-9650-4792-bba2-d9d8dd383d90" (UID: "dc917b09-9650-4792-bba2-d9d8dd383d90"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 14:57:28.080583 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:57:28.080563 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc917b09-9650-4792-bba2-d9d8dd383d90-online-tls" (OuterVolumeSpecName: "online-tls") pod "dc917b09-9650-4792-bba2-d9d8dd383d90" (UID: "dc917b09-9650-4792-bba2-d9d8dd383d90"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 14:57:28.080638 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:57:28.080592 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc917b09-9650-4792-bba2-d9d8dd383d90-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "dc917b09-9650-4792-bba2-d9d8dd383d90" (UID: "dc917b09-9650-4792-bba2-d9d8dd383d90"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 14:57:28.179138 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:57:28.179101 2576 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/dc917b09-9650-4792-bba2-d9d8dd383d90-feast-data\") on node \"ip-10-0-133-66.ec2.internal\" DevicePath \"\"" Sep 09 14:57:28.179138 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:57:28.179132 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bht9b\" (UniqueName: \"kubernetes.io/projected/dc917b09-9650-4792-bba2-d9d8dd383d90-kube-api-access-bht9b\") on node \"ip-10-0-133-66.ec2.internal\" DevicePath \"\"" Sep 09 14:57:28.179138 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:57:28.179143 2576 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/dc917b09-9650-4792-bba2-d9d8dd383d90-registry-tls\") on node \"ip-10-0-133-66.ec2.internal\" DevicePath \"\"" Sep 09 14:57:28.179420 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:57:28.179173 2576 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/dc917b09-9650-4792-bba2-d9d8dd383d90-online-tls\") on node \"ip-10-0-133-66.ec2.internal\" DevicePath \"\"" Sep 09 14:57:28.179420 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:57:28.179181 2576 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/dc917b09-9650-4792-bba2-d9d8dd383d90-offline-tls\") on node \"ip-10-0-133-66.ec2.internal\" DevicePath \"\"" Sep 09 14:57:28.179420 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:57:28.179188 2576 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/dc917b09-9650-4792-bba2-d9d8dd383d90-ui-tls\") on node \"ip-10-0-133-66.ec2.internal\" DevicePath \"\"" Sep 09 14:57:28.361102 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:57:28.361066 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-qpb79"] Sep 09 14:57:28.369279 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:57:28.369250 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-qpb79"] Sep 09 14:57:29.514614 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:57:29.514577 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc917b09-9650-4792-bba2-d9d8dd383d90" path="/var/lib/kubelet/pods/dc917b09-9650-4792-bba2-d9d8dd383d90/volumes" Sep 09 14:58:05.480702 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:58:05.480671 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7xcn5_186aab90-266b-4eae-977a-3f053b423ef3/ovn-acl-logging/0.log" Sep 09 14:58:05.482462 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:58:05.482443 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7xcn5_186aab90-266b-4eae-977a-3f053b423ef3/ovn-acl-logging/0.log" Sep 09 14:58:22.721181 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:58:22.721148 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-l5mkg"] Sep 09 14:58:22.721686 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:58:22.721512 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc917b09-9650-4792-bba2-d9d8dd383d90" containerName="feast-init" Sep 09 14:58:22.721686 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:58:22.721525 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc917b09-9650-4792-bba2-d9d8dd383d90" containerName="feast-init" Sep 09 14:58:22.721686 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:58:22.721540 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc917b09-9650-4792-bba2-d9d8dd383d90" containerName="online" Sep 09 14:58:22.721686 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:58:22.721546 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc917b09-9650-4792-bba2-d9d8dd383d90" containerName="online" Sep 09 14:58:22.721686 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:58:22.721553 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc917b09-9650-4792-bba2-d9d8dd383d90" containerName="offline" Sep 09 14:58:22.721686 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:58:22.721558 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc917b09-9650-4792-bba2-d9d8dd383d90" containerName="offline" Sep 09 14:58:22.721686 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:58:22.721568 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc917b09-9650-4792-bba2-d9d8dd383d90" containerName="feast-apply" Sep 09 14:58:22.721686 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:58:22.721575 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc917b09-9650-4792-bba2-d9d8dd383d90" containerName="feast-apply" Sep 09 14:58:22.721686 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:58:22.721587 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc917b09-9650-4792-bba2-d9d8dd383d90" containerName="registry" Sep 09 14:58:22.721686 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:58:22.721594 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc917b09-9650-4792-bba2-d9d8dd383d90" containerName="registry" Sep 09 14:58:22.721686 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:58:22.721607 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc917b09-9650-4792-bba2-d9d8dd383d90" containerName="ui" Sep 09 14:58:22.721686 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:58:22.721612 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc917b09-9650-4792-bba2-d9d8dd383d90" containerName="ui" Sep 09 14:58:22.721686 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:58:22.721665 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc917b09-9650-4792-bba2-d9d8dd383d90" containerName="registry" Sep 09 14:58:22.721686 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:58:22.721674 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc917b09-9650-4792-bba2-d9d8dd383d90" containerName="ui" Sep 09 14:58:22.721686 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:58:22.721679 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc917b09-9650-4792-bba2-d9d8dd383d90" containerName="offline" Sep 09 14:58:22.721686 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:58:22.721685 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc917b09-9650-4792-bba2-d9d8dd383d90" containerName="online" Sep 09 14:58:22.726606 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:58:22.726586 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-l5mkg" Sep 09 14:58:22.729283 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:58:22.729253 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-client-ca\"" Sep 09 14:58:22.729429 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:58:22.729293 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-online-tls\"" Sep 09 14:58:22.729429 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:58:22.729347 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-dockercfg-ckj2x\"" Sep 09 14:58:22.729429 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:58:22.729255 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"kube-root-ca.crt\"" Sep 09 14:58:22.729429 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:58:22.729416 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"openshift-service-ca.crt\"" Sep 09 14:58:22.729668 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:58:22.729531 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-ui-tls\"" Sep 09 14:58:22.729823 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:58:22.729807 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-offline-tls\"" Sep 09 14:58:22.738342 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:58:22.738317 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-l5mkg"] Sep 09 14:58:22.813611 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:58:22.813578 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/eb9c3a35-6a4b-452c-89ca-0ce13611a7a8-offline-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-l5mkg\" (UID: \"eb9c3a35-6a4b-452c-89ca-0ce13611a7a8\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-l5mkg" Sep 09 14:58:22.813844 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:58:22.813619 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/eb9c3a35-6a4b-452c-89ca-0ce13611a7a8-feast-data\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-l5mkg\" (UID: \"eb9c3a35-6a4b-452c-89ca-0ce13611a7a8\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-l5mkg" Sep 09 14:58:22.813844 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:58:22.813656 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/eb9c3a35-6a4b-452c-89ca-0ce13611a7a8-ui-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-l5mkg\" (UID: \"eb9c3a35-6a4b-452c-89ca-0ce13611a7a8\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-l5mkg" Sep 09 14:58:22.813844 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:58:22.813741 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/eb9c3a35-6a4b-452c-89ca-0ce13611a7a8-online-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-l5mkg\" (UID: \"eb9c3a35-6a4b-452c-89ca-0ce13611a7a8\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-l5mkg" Sep 09 14:58:22.813844 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:58:22.813777 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/eb9c3a35-6a4b-452c-89ca-0ce13611a7a8-registry-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-l5mkg\" (UID: \"eb9c3a35-6a4b-452c-89ca-0ce13611a7a8\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-l5mkg" Sep 09 14:58:22.814019 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:58:22.813853 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqzql\" (UniqueName: \"kubernetes.io/projected/eb9c3a35-6a4b-452c-89ca-0ce13611a7a8-kube-api-access-bqzql\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-l5mkg\" (UID: \"eb9c3a35-6a4b-452c-89ca-0ce13611a7a8\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-l5mkg" Sep 09 14:58:22.914479 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:58:22.914442 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/eb9c3a35-6a4b-452c-89ca-0ce13611a7a8-online-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-l5mkg\" (UID: \"eb9c3a35-6a4b-452c-89ca-0ce13611a7a8\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-l5mkg" Sep 09 14:58:22.914479 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:58:22.914483 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/eb9c3a35-6a4b-452c-89ca-0ce13611a7a8-registry-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-l5mkg\" (UID: \"eb9c3a35-6a4b-452c-89ca-0ce13611a7a8\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-l5mkg" Sep 09 14:58:22.914684 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:58:22.914525 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqzql\" (UniqueName: \"kubernetes.io/projected/eb9c3a35-6a4b-452c-89ca-0ce13611a7a8-kube-api-access-bqzql\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-l5mkg\" (UID: \"eb9c3a35-6a4b-452c-89ca-0ce13611a7a8\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-l5mkg" Sep 09 14:58:22.914684 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:58:22.914574 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/eb9c3a35-6a4b-452c-89ca-0ce13611a7a8-offline-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-l5mkg\" (UID: \"eb9c3a35-6a4b-452c-89ca-0ce13611a7a8\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-l5mkg" Sep 09 14:58:22.914684 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:58:22.914632 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/eb9c3a35-6a4b-452c-89ca-0ce13611a7a8-feast-data\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-l5mkg\" (UID: \"eb9c3a35-6a4b-452c-89ca-0ce13611a7a8\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-l5mkg" Sep 09 14:58:22.914684 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:58:22.914677 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/eb9c3a35-6a4b-452c-89ca-0ce13611a7a8-ui-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-l5mkg\" (UID: \"eb9c3a35-6a4b-452c-89ca-0ce13611a7a8\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-l5mkg" Sep 09 14:58:22.914894 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:58:22.914765 2576 secret.go:189] Couldn't get secret test-ns-remote-registry/feast-simple-feast-remote-setup-ui-tls: secret "feast-simple-feast-remote-setup-ui-tls" not found Sep 09 14:58:22.914894 ip-10-0-133-66 kubenswrapper[2576]: E0909 14:58:22.914825 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb9c3a35-6a4b-452c-89ca-0ce13611a7a8-ui-tls podName:eb9c3a35-6a4b-452c-89ca-0ce13611a7a8 nodeName:}" failed. No retries permitted until 2026-09-09 14:58:23.414804824 +0000 UTC m=+1518.425721066 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "ui-tls" (UniqueName: "kubernetes.io/secret/eb9c3a35-6a4b-452c-89ca-0ce13611a7a8-ui-tls") pod "feast-simple-feast-remote-setup-c5d49d8c6-l5mkg" (UID: "eb9c3a35-6a4b-452c-89ca-0ce13611a7a8") : secret "feast-simple-feast-remote-setup-ui-tls" not found Sep 09 14:58:22.915050 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:58:22.915028 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/eb9c3a35-6a4b-452c-89ca-0ce13611a7a8-feast-data\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-l5mkg\" (UID: \"eb9c3a35-6a4b-452c-89ca-0ce13611a7a8\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-l5mkg" Sep 09 14:58:22.915638 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:58:22.915612 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/eb9c3a35-6a4b-452c-89ca-0ce13611a7a8-registry-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-l5mkg\" (UID: \"eb9c3a35-6a4b-452c-89ca-0ce13611a7a8\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-l5mkg" Sep 09 14:58:22.917059 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:58:22.917036 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/eb9c3a35-6a4b-452c-89ca-0ce13611a7a8-offline-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-l5mkg\" (UID: \"eb9c3a35-6a4b-452c-89ca-0ce13611a7a8\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-l5mkg" Sep 09 14:58:22.917143 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:58:22.917078 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/eb9c3a35-6a4b-452c-89ca-0ce13611a7a8-online-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-l5mkg\" (UID: \"eb9c3a35-6a4b-452c-89ca-0ce13611a7a8\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-l5mkg" Sep 09 14:58:22.923696 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:58:22.923678 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqzql\" (UniqueName: \"kubernetes.io/projected/eb9c3a35-6a4b-452c-89ca-0ce13611a7a8-kube-api-access-bqzql\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-l5mkg\" (UID: \"eb9c3a35-6a4b-452c-89ca-0ce13611a7a8\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-l5mkg" Sep 09 14:58:23.418744 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:58:23.418702 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/eb9c3a35-6a4b-452c-89ca-0ce13611a7a8-ui-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-l5mkg\" (UID: \"eb9c3a35-6a4b-452c-89ca-0ce13611a7a8\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-l5mkg" Sep 09 14:58:23.421147 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:58:23.421121 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/eb9c3a35-6a4b-452c-89ca-0ce13611a7a8-ui-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-l5mkg\" (UID: \"eb9c3a35-6a4b-452c-89ca-0ce13611a7a8\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-l5mkg" Sep 09 14:58:23.637804 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:58:23.637757 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-l5mkg" Sep 09 14:58:23.913862 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:58:23.913836 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-l5mkg"] Sep 09 14:58:23.915206 ip-10-0-133-66 kubenswrapper[2576]: W0909 14:58:23.915176 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeb9c3a35_6a4b_452c_89ca_0ce13611a7a8.slice/crio-6820c76600c12751a4535bf5b591c6c33961deb8c347e4f9410c562225411a60 WatchSource:0}: Error finding container 6820c76600c12751a4535bf5b591c6c33961deb8c347e4f9410c562225411a60: Status 404 returned error can't find the container with id 6820c76600c12751a4535bf5b591c6c33961deb8c347e4f9410c562225411a60 Sep 09 14:58:24.250511 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:58:24.250415 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-l5mkg" event={"ID":"eb9c3a35-6a4b-452c-89ca-0ce13611a7a8","Type":"ContainerStarted","Data":"94c722e1f34813a5e5b727e0255bd85ef1643162f2b869212551d54dbf8be80b"} Sep 09 14:58:24.250511 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:58:24.250471 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-l5mkg" event={"ID":"eb9c3a35-6a4b-452c-89ca-0ce13611a7a8","Type":"ContainerStarted","Data":"6820c76600c12751a4535bf5b591c6c33961deb8c347e4f9410c562225411a60"} Sep 09 14:58:29.278863 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:58:29.278828 2576 generic.go:356] "Generic (PLEG): container finished" podID="eb9c3a35-6a4b-452c-89ca-0ce13611a7a8" containerID="94c722e1f34813a5e5b727e0255bd85ef1643162f2b869212551d54dbf8be80b" exitCode=0 Sep 09 14:58:29.279376 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:58:29.278890 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-l5mkg" event={"ID":"eb9c3a35-6a4b-452c-89ca-0ce13611a7a8","Type":"ContainerDied","Data":"94c722e1f34813a5e5b727e0255bd85ef1643162f2b869212551d54dbf8be80b"} Sep 09 14:58:30.284668 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:58:30.284631 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-l5mkg" event={"ID":"eb9c3a35-6a4b-452c-89ca-0ce13611a7a8","Type":"ContainerStarted","Data":"b75445db55a127ce6e860c143d9d403061e2e52c82920225fb442117cd8b1d85"} Sep 09 14:58:51.368530 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:58:51.368496 2576 generic.go:356] "Generic (PLEG): container finished" podID="eb9c3a35-6a4b-452c-89ca-0ce13611a7a8" containerID="b75445db55a127ce6e860c143d9d403061e2e52c82920225fb442117cd8b1d85" exitCode=0 Sep 09 14:58:51.369059 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:58:51.368574 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-l5mkg" event={"ID":"eb9c3a35-6a4b-452c-89ca-0ce13611a7a8","Type":"ContainerDied","Data":"b75445db55a127ce6e860c143d9d403061e2e52c82920225fb442117cd8b1d85"} Sep 09 14:58:52.376928 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:58:52.376890 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-l5mkg" event={"ID":"eb9c3a35-6a4b-452c-89ca-0ce13611a7a8","Type":"ContainerStarted","Data":"a7b62de3e74f58adb054b51b7910f93568416a1dc4ce9f930ed2b5a436ee796b"} Sep 09 14:58:52.377341 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:58:52.376953 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-l5mkg" event={"ID":"eb9c3a35-6a4b-452c-89ca-0ce13611a7a8","Type":"ContainerStarted","Data":"94b203e0256c877674be9314b3e109f425a516926bb486ec210ceba42933e860"} Sep 09 14:58:52.377341 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:58:52.376974 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-l5mkg" event={"ID":"eb9c3a35-6a4b-452c-89ca-0ce13611a7a8","Type":"ContainerStarted","Data":"bbf5a549900c601d76201479a105c25ff173d8ee99a8c181c1e0a99e4bd7e771"} Sep 09 14:58:52.402435 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:58:52.402378 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-l5mkg" podStartSLOduration=30.402358094 podStartE2EDuration="30.402358094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 14:58:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:58:52.400546659 +0000 UTC m=+1547.411462923" watchObservedRunningTime="2026-09-09 14:58:52.402358094 +0000 UTC m=+1547.413274345" Sep 09 14:58:53.638418 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:58:53.638384 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-l5mkg" Sep 09 14:58:53.638842 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:58:53.638509 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-l5mkg" Sep 09 14:58:53.638842 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:58:53.638532 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-l5mkg" Sep 09 14:58:53.638842 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:58:53.638553 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-l5mkg" Sep 09 14:58:53.638842 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:58:53.638571 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-l5mkg" Sep 09 14:58:53.638842 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:58:53.638587 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-l5mkg" Sep 09 14:58:53.640437 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:58:53.640402 2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-l5mkg" podUID="eb9c3a35-6a4b-452c-89ca-0ce13611a7a8" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.27:8816: connect: connection refused" Sep 09 14:58:53.640577 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:58:53.640461 2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-l5mkg" podUID="eb9c3a35-6a4b-452c-89ca-0ce13611a7a8" containerName="online" probeResult="failure" output="Get \"https://10.132.0.27:6567/health\": dial tcp 10.132.0.27:6567: connect: connection refused" Sep 09 14:58:53.640577 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:58:53.640413 2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-l5mkg" podUID="eb9c3a35-6a4b-452c-89ca-0ce13611a7a8" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.27:8443: connect: connection refused" Sep 09 14:58:56.638638 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:58:56.638580 2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-l5mkg" podUID="eb9c3a35-6a4b-452c-89ca-0ce13611a7a8" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.27:8816: connect: connection refused" Sep 09 14:58:56.639094 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:58:56.638580 2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-l5mkg" podUID="eb9c3a35-6a4b-452c-89ca-0ce13611a7a8" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.27:8443: connect: connection refused" Sep 09 14:58:56.639094 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:58:56.638589 2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-l5mkg" podUID="eb9c3a35-6a4b-452c-89ca-0ce13611a7a8" containerName="online" probeResult="failure" output="Get \"https://10.132.0.27:6567/health\": dial tcp 10.132.0.27:6567: connect: connection refused" Sep 09 14:58:59.638693 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:58:59.638648 2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-l5mkg" podUID="eb9c3a35-6a4b-452c-89ca-0ce13611a7a8" containerName="online" probeResult="failure" output="Get \"https://10.132.0.27:6567/health\": dial tcp 10.132.0.27:6567: connect: connection refused" Sep 09 14:58:59.639494 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:58:59.639471 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-l5mkg" Sep 09 14:58:59.639617 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:58:59.639581 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-l5mkg" Sep 09 14:58:59.640201 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:58:59.640173 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-l5mkg" Sep 09 14:58:59.640410 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:58:59.640393 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-l5mkg" Sep 09 14:59:03.271180 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:59:03.271144 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-l5mkg" Sep 09 14:59:03.893762 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:59:03.893729 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-l5mkg" Sep 09 14:59:44.642462 ip-10-0-133-66 kubenswrapper[2576]: I0909 14:59:44.642416 2576 prober.go:120] "Probe failed" probeType="Liveness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-l5mkg" podUID="eb9c3a35-6a4b-452c-89ca-0ce13611a7a8" containerName="online" probeResult="failure" output="Get \"https://10.132.0.27:6567/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Sep 09 15:00:05.256167 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:00:05.256073 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-l5mkg"] Sep 09 15:00:05.256794 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:00:05.256419 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-l5mkg" podUID="eb9c3a35-6a4b-452c-89ca-0ce13611a7a8" containerName="online" containerID="cri-o://bbf5a549900c601d76201479a105c25ff173d8ee99a8c181c1e0a99e4bd7e771" gracePeriod=30 Sep 09 15:00:05.257249 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:00:05.256547 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-l5mkg" podUID="eb9c3a35-6a4b-452c-89ca-0ce13611a7a8" containerName="ui" containerID="cri-o://a7b62de3e74f58adb054b51b7910f93568416a1dc4ce9f930ed2b5a436ee796b" gracePeriod=30 Sep 09 15:00:05.257249 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:00:05.256577 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-l5mkg" podUID="eb9c3a35-6a4b-452c-89ca-0ce13611a7a8" containerName="offline" containerID="cri-o://94b203e0256c877674be9314b3e109f425a516926bb486ec210ceba42933e860" gracePeriod=30 Sep 09 15:00:05.398481 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:00:05.398448 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-zggqz"] Sep 09 15:00:05.398775 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:00:05.398739 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zggqz" podUID="6bfe4cb5-2f10-42aa-9141-7d128b9c6e91" containerName="registry" containerID="cri-o://db9c30bdde2d6c55b15ae4f3b8da6b9e38c2db3e8481e3470b661db21b05c3fe" gracePeriod=30 Sep 09 15:00:05.398928 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:00:05.398842 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zggqz" podUID="6bfe4cb5-2f10-42aa-9141-7d128b9c6e91" containerName="offline" containerID="cri-o://914536ff7cedd943bd070f48dbe944ad53329b25d427c687e6aab4cdf3d0ac5a" gracePeriod=30 Sep 09 15:00:05.398928 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:00:05.398851 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zggqz" podUID="6bfe4cb5-2f10-42aa-9141-7d128b9c6e91" containerName="online" containerID="cri-o://fe803cc52e67f746b7ab2c3c7225aaec5f301747116bbd90ae86dbef66580e3d" gracePeriod=30 Sep 09 15:00:05.399059 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:00:05.398982 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zggqz" podUID="6bfe4cb5-2f10-42aa-9141-7d128b9c6e91" containerName="ui" containerID="cri-o://d4f0f616273c23dee12d55da52dac30a96838a033d3853d72d3d6189fbc06a3c" gracePeriod=30 Sep 09 15:00:06.685123 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:00:06.685092 2576 generic.go:356] "Generic (PLEG): container finished" podID="6bfe4cb5-2f10-42aa-9141-7d128b9c6e91" containerID="d4f0f616273c23dee12d55da52dac30a96838a033d3853d72d3d6189fbc06a3c" exitCode=0 Sep 09 15:00:06.685123 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:00:06.685116 2576 generic.go:356] "Generic (PLEG): container finished" podID="6bfe4cb5-2f10-42aa-9141-7d128b9c6e91" containerID="fe803cc52e67f746b7ab2c3c7225aaec5f301747116bbd90ae86dbef66580e3d" exitCode=0 Sep 09 15:00:06.685620 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:00:06.685170 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zggqz" event={"ID":"6bfe4cb5-2f10-42aa-9141-7d128b9c6e91","Type":"ContainerDied","Data":"d4f0f616273c23dee12d55da52dac30a96838a033d3853d72d3d6189fbc06a3c"} Sep 09 15:00:06.685620 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:00:06.685218 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zggqz" event={"ID":"6bfe4cb5-2f10-42aa-9141-7d128b9c6e91","Type":"ContainerDied","Data":"fe803cc52e67f746b7ab2c3c7225aaec5f301747116bbd90ae86dbef66580e3d"} Sep 09 15:00:06.687877 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:00:06.687855 2576 generic.go:356] "Generic (PLEG): container finished" podID="eb9c3a35-6a4b-452c-89ca-0ce13611a7a8" containerID="a7b62de3e74f58adb054b51b7910f93568416a1dc4ce9f930ed2b5a436ee796b" exitCode=0 Sep 09 15:00:06.687877 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:00:06.687875 2576 generic.go:356] "Generic (PLEG): container finished" podID="eb9c3a35-6a4b-452c-89ca-0ce13611a7a8" containerID="bbf5a549900c601d76201479a105c25ff173d8ee99a8c181c1e0a99e4bd7e771" exitCode=0 Sep 09 15:00:06.688043 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:00:06.687929 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-l5mkg" event={"ID":"eb9c3a35-6a4b-452c-89ca-0ce13611a7a8","Type":"ContainerDied","Data":"a7b62de3e74f58adb054b51b7910f93568416a1dc4ce9f930ed2b5a436ee796b"} Sep 09 15:00:06.688043 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:00:06.687972 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-l5mkg" event={"ID":"eb9c3a35-6a4b-452c-89ca-0ce13611a7a8","Type":"ContainerDied","Data":"bbf5a549900c601d76201479a105c25ff173d8ee99a8c181c1e0a99e4bd7e771"} Sep 09 15:00:09.640843 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:00:09.640791 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-l5mkg" podUID="eb9c3a35-6a4b-452c-89ca-0ce13611a7a8" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.27:8443: connect: connection refused" Sep 09 15:00:09.984760 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:00:09.984654 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zggqz" podUID="6bfe4cb5-2f10-42aa-9141-7d128b9c6e91" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.26:8443: connect: connection refused" Sep 09 15:00:09.984760 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:00:09.984699 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zggqz" podUID="6bfe4cb5-2f10-42aa-9141-7d128b9c6e91" containerName="online" probeResult="failure" output="Get \"https://10.132.0.26:6567/health\": dial tcp 10.132.0.26:6567: connect: connection refused" Sep 09 15:00:13.272077 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:00:13.272028 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-l5mkg" podUID="eb9c3a35-6a4b-452c-89ca-0ce13611a7a8" containerName="online" probeResult="failure" output="Get \"https://10.132.0.27:6567/health\": dial tcp 10.132.0.27:6567: connect: connection refused" Sep 09 15:00:19.640216 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:00:19.640157 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-l5mkg" podUID="eb9c3a35-6a4b-452c-89ca-0ce13611a7a8" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.27:8443: connect: connection refused" Sep 09 15:00:19.984631 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:00:19.984518 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zggqz" podUID="6bfe4cb5-2f10-42aa-9141-7d128b9c6e91" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.26:8443: connect: connection refused" Sep 09 15:00:19.984631 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:00:19.984566 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zggqz" podUID="6bfe4cb5-2f10-42aa-9141-7d128b9c6e91" containerName="online" probeResult="failure" output="Get \"https://10.132.0.26:6567/health\": dial tcp 10.132.0.26:6567: connect: connection refused" Sep 09 15:00:23.272494 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:00:23.272444 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-l5mkg" podUID="eb9c3a35-6a4b-452c-89ca-0ce13611a7a8" containerName="online" probeResult="failure" output="Get \"https://10.132.0.27:6567/health\": dial tcp 10.132.0.27:6567: connect: connection refused" Sep 09 15:00:29.640318 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:00:29.640263 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-l5mkg" podUID="eb9c3a35-6a4b-452c-89ca-0ce13611a7a8" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.27:8443: connect: connection refused" Sep 09 15:00:29.640725 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:00:29.640438 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-l5mkg" Sep 09 15:00:29.984889 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:00:29.984772 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zggqz" podUID="6bfe4cb5-2f10-42aa-9141-7d128b9c6e91" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.26:8443: connect: connection refused" Sep 09 15:00:29.984889 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:00:29.984801 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zggqz" podUID="6bfe4cb5-2f10-42aa-9141-7d128b9c6e91" containerName="online" probeResult="failure" output="Get \"https://10.132.0.26:6567/health\": dial tcp 10.132.0.26:6567: connect: connection refused" Sep 09 15:00:29.985129 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:00:29.984964 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zggqz" Sep 09 15:00:29.985129 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:00:29.985031 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zggqz" Sep 09 15:00:33.272760 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:00:33.272713 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-l5mkg" podUID="eb9c3a35-6a4b-452c-89ca-0ce13611a7a8" containerName="online" probeResult="failure" output="Get \"https://10.132.0.27:6567/health\": dial tcp 10.132.0.27:6567: connect: connection refused" Sep 09 15:00:33.273170 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:00:33.272861 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-l5mkg" Sep 09 15:00:35.809924 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:00:35.809888 2576 generic.go:356] "Generic (PLEG): container finished" podID="eb9c3a35-6a4b-452c-89ca-0ce13611a7a8" containerID="94b203e0256c877674be9314b3e109f425a516926bb486ec210ceba42933e860" exitCode=137 Sep 09 15:00:35.810479 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:00:35.809962 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-l5mkg" event={"ID":"eb9c3a35-6a4b-452c-89ca-0ce13611a7a8","Type":"ContainerDied","Data":"94b203e0256c877674be9314b3e109f425a516926bb486ec210ceba42933e860"} Sep 09 15:00:35.813569 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:00:35.813537 2576 generic.go:356] "Generic (PLEG): container finished" podID="6bfe4cb5-2f10-42aa-9141-7d128b9c6e91" containerID="914536ff7cedd943bd070f48dbe944ad53329b25d427c687e6aab4cdf3d0ac5a" exitCode=137 Sep 09 15:00:35.813569 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:00:35.813564 2576 generic.go:356] "Generic (PLEG): container finished" podID="6bfe4cb5-2f10-42aa-9141-7d128b9c6e91" containerID="db9c30bdde2d6c55b15ae4f3b8da6b9e38c2db3e8481e3470b661db21b05c3fe" exitCode=137 Sep 09 15:00:35.813756 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:00:35.813606 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zggqz" event={"ID":"6bfe4cb5-2f10-42aa-9141-7d128b9c6e91","Type":"ContainerDied","Data":"914536ff7cedd943bd070f48dbe944ad53329b25d427c687e6aab4cdf3d0ac5a"} Sep 09 15:00:35.813756 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:00:35.813648 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zggqz" event={"ID":"6bfe4cb5-2f10-42aa-9141-7d128b9c6e91","Type":"ContainerDied","Data":"db9c30bdde2d6c55b15ae4f3b8da6b9e38c2db3e8481e3470b661db21b05c3fe"} Sep 09 15:00:35.914048 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:00:35.914020 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-l5mkg" Sep 09 15:00:36.030824 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:00:36.030717 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/eb9c3a35-6a4b-452c-89ca-0ce13611a7a8-feast-data\") pod \"eb9c3a35-6a4b-452c-89ca-0ce13611a7a8\" (UID: \"eb9c3a35-6a4b-452c-89ca-0ce13611a7a8\") " Sep 09 15:00:36.030824 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:00:36.030807 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bqzql\" (UniqueName: \"kubernetes.io/projected/eb9c3a35-6a4b-452c-89ca-0ce13611a7a8-kube-api-access-bqzql\") pod \"eb9c3a35-6a4b-452c-89ca-0ce13611a7a8\" (UID: \"eb9c3a35-6a4b-452c-89ca-0ce13611a7a8\") " Sep 09 15:00:36.031154 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:00:36.030845 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/eb9c3a35-6a4b-452c-89ca-0ce13611a7a8-registry-tls\") pod \"eb9c3a35-6a4b-452c-89ca-0ce13611a7a8\" (UID: \"eb9c3a35-6a4b-452c-89ca-0ce13611a7a8\") " Sep 09 15:00:36.031154 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:00:36.030911 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/eb9c3a35-6a4b-452c-89ca-0ce13611a7a8-online-tls\") pod \"eb9c3a35-6a4b-452c-89ca-0ce13611a7a8\" (UID: \"eb9c3a35-6a4b-452c-89ca-0ce13611a7a8\") " Sep 09 15:00:36.031154 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:00:36.030939 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/eb9c3a35-6a4b-452c-89ca-0ce13611a7a8-ui-tls\") pod \"eb9c3a35-6a4b-452c-89ca-0ce13611a7a8\" (UID: \"eb9c3a35-6a4b-452c-89ca-0ce13611a7a8\") " Sep 09 15:00:36.031154 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:00:36.030962 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/eb9c3a35-6a4b-452c-89ca-0ce13611a7a8-offline-tls\") pod \"eb9c3a35-6a4b-452c-89ca-0ce13611a7a8\" (UID: \"eb9c3a35-6a4b-452c-89ca-0ce13611a7a8\") " Sep 09 15:00:36.031420 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:00:36.031283 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb9c3a35-6a4b-452c-89ca-0ce13611a7a8-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "eb9c3a35-6a4b-452c-89ca-0ce13611a7a8" (UID: "eb9c3a35-6a4b-452c-89ca-0ce13611a7a8"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 15:00:36.031420 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:00:36.031404 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb9c3a35-6a4b-452c-89ca-0ce13611a7a8-feast-data" (OuterVolumeSpecName: "feast-data") pod "eb9c3a35-6a4b-452c-89ca-0ce13611a7a8" (UID: "eb9c3a35-6a4b-452c-89ca-0ce13611a7a8"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 15:00:36.033544 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:00:36.033511 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb9c3a35-6a4b-452c-89ca-0ce13611a7a8-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "eb9c3a35-6a4b-452c-89ca-0ce13611a7a8" (UID: "eb9c3a35-6a4b-452c-89ca-0ce13611a7a8"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 15:00:36.033544 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:00:36.033521 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb9c3a35-6a4b-452c-89ca-0ce13611a7a8-online-tls" (OuterVolumeSpecName: "online-tls") pod "eb9c3a35-6a4b-452c-89ca-0ce13611a7a8" (UID: "eb9c3a35-6a4b-452c-89ca-0ce13611a7a8"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 15:00:36.033747 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:00:36.033678 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb9c3a35-6a4b-452c-89ca-0ce13611a7a8-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "eb9c3a35-6a4b-452c-89ca-0ce13611a7a8" (UID: "eb9c3a35-6a4b-452c-89ca-0ce13611a7a8"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 15:00:36.033747 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:00:36.033720 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb9c3a35-6a4b-452c-89ca-0ce13611a7a8-kube-api-access-bqzql" (OuterVolumeSpecName: "kube-api-access-bqzql") pod "eb9c3a35-6a4b-452c-89ca-0ce13611a7a8" (UID: "eb9c3a35-6a4b-452c-89ca-0ce13611a7a8"). InnerVolumeSpecName "kube-api-access-bqzql". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 15:00:36.051061 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:00:36.051031 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zggqz" Sep 09 15:00:36.132259 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:00:36.132221 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/6bfe4cb5-2f10-42aa-9141-7d128b9c6e91-feast-data\") pod \"6bfe4cb5-2f10-42aa-9141-7d128b9c6e91\" (UID: \"6bfe4cb5-2f10-42aa-9141-7d128b9c6e91\") " Sep 09 15:00:36.132259 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:00:36.132261 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/6bfe4cb5-2f10-42aa-9141-7d128b9c6e91-offline-tls\") pod \"6bfe4cb5-2f10-42aa-9141-7d128b9c6e91\" (UID: \"6bfe4cb5-2f10-42aa-9141-7d128b9c6e91\") " Sep 09 15:00:36.132531 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:00:36.132287 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/6bfe4cb5-2f10-42aa-9141-7d128b9c6e91-ui-tls\") pod \"6bfe4cb5-2f10-42aa-9141-7d128b9c6e91\" (UID: \"6bfe4cb5-2f10-42aa-9141-7d128b9c6e91\") " Sep 09 15:00:36.132531 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:00:36.132389 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/6bfe4cb5-2f10-42aa-9141-7d128b9c6e91-registry-tls\") pod \"6bfe4cb5-2f10-42aa-9141-7d128b9c6e91\" (UID: \"6bfe4cb5-2f10-42aa-9141-7d128b9c6e91\") " Sep 09 15:00:36.132531 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:00:36.132451 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/6bfe4cb5-2f10-42aa-9141-7d128b9c6e91-online-tls\") pod \"6bfe4cb5-2f10-42aa-9141-7d128b9c6e91\" (UID: \"6bfe4cb5-2f10-42aa-9141-7d128b9c6e91\") " Sep 09 15:00:36.132531 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:00:36.132486 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gpd8j\" (UniqueName: \"kubernetes.io/projected/6bfe4cb5-2f10-42aa-9141-7d128b9c6e91-kube-api-access-gpd8j\") pod \"6bfe4cb5-2f10-42aa-9141-7d128b9c6e91\" (UID: \"6bfe4cb5-2f10-42aa-9141-7d128b9c6e91\") " Sep 09 15:00:36.132765 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:00:36.132716 2576 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/eb9c3a35-6a4b-452c-89ca-0ce13611a7a8-online-tls\") on node \"ip-10-0-133-66.ec2.internal\" DevicePath \"\"" Sep 09 15:00:36.132765 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:00:36.132734 2576 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/eb9c3a35-6a4b-452c-89ca-0ce13611a7a8-ui-tls\") on node \"ip-10-0-133-66.ec2.internal\" DevicePath \"\"" Sep 09 15:00:36.132765 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:00:36.132747 2576 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/eb9c3a35-6a4b-452c-89ca-0ce13611a7a8-offline-tls\") on node \"ip-10-0-133-66.ec2.internal\" DevicePath \"\"" Sep 09 15:00:36.132765 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:00:36.132758 2576 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/eb9c3a35-6a4b-452c-89ca-0ce13611a7a8-feast-data\") on node \"ip-10-0-133-66.ec2.internal\" DevicePath \"\"" Sep 09 15:00:36.132952 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:00:36.132773 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqzql\" (UniqueName: \"kubernetes.io/projected/eb9c3a35-6a4b-452c-89ca-0ce13611a7a8-kube-api-access-bqzql\") on node \"ip-10-0-133-66.ec2.internal\" DevicePath \"\"" Sep 09 15:00:36.132952 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:00:36.132787 2576 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/eb9c3a35-6a4b-452c-89ca-0ce13611a7a8-registry-tls\") on node \"ip-10-0-133-66.ec2.internal\" DevicePath \"\"" Sep 09 15:00:36.132952 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:00:36.132880 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bfe4cb5-2f10-42aa-9141-7d128b9c6e91-feast-data" (OuterVolumeSpecName: "feast-data") pod "6bfe4cb5-2f10-42aa-9141-7d128b9c6e91" (UID: "6bfe4cb5-2f10-42aa-9141-7d128b9c6e91"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 15:00:36.134884 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:00:36.134842 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6bfe4cb5-2f10-42aa-9141-7d128b9c6e91-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "6bfe4cb5-2f10-42aa-9141-7d128b9c6e91" (UID: "6bfe4cb5-2f10-42aa-9141-7d128b9c6e91"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 15:00:36.134884 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:00:36.134877 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bfe4cb5-2f10-42aa-9141-7d128b9c6e91-kube-api-access-gpd8j" (OuterVolumeSpecName: "kube-api-access-gpd8j") pod "6bfe4cb5-2f10-42aa-9141-7d128b9c6e91" (UID: "6bfe4cb5-2f10-42aa-9141-7d128b9c6e91"). InnerVolumeSpecName "kube-api-access-gpd8j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 15:00:36.135103 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:00:36.134894 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6bfe4cb5-2f10-42aa-9141-7d128b9c6e91-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "6bfe4cb5-2f10-42aa-9141-7d128b9c6e91" (UID: "6bfe4cb5-2f10-42aa-9141-7d128b9c6e91"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 15:00:36.135103 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:00:36.135080 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6bfe4cb5-2f10-42aa-9141-7d128b9c6e91-online-tls" (OuterVolumeSpecName: "online-tls") pod "6bfe4cb5-2f10-42aa-9141-7d128b9c6e91" (UID: "6bfe4cb5-2f10-42aa-9141-7d128b9c6e91"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 15:00:36.135185 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:00:36.135161 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6bfe4cb5-2f10-42aa-9141-7d128b9c6e91-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "6bfe4cb5-2f10-42aa-9141-7d128b9c6e91" (UID: "6bfe4cb5-2f10-42aa-9141-7d128b9c6e91"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 15:00:36.233776 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:00:36.233739 2576 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/6bfe4cb5-2f10-42aa-9141-7d128b9c6e91-online-tls\") on node \"ip-10-0-133-66.ec2.internal\" DevicePath \"\"" Sep 09 15:00:36.233776 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:00:36.233768 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpd8j\" (UniqueName: \"kubernetes.io/projected/6bfe4cb5-2f10-42aa-9141-7d128b9c6e91-kube-api-access-gpd8j\") on node \"ip-10-0-133-66.ec2.internal\" DevicePath \"\"" Sep 09 15:00:36.233776 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:00:36.233778 2576 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/6bfe4cb5-2f10-42aa-9141-7d128b9c6e91-feast-data\") on node \"ip-10-0-133-66.ec2.internal\" DevicePath \"\"" Sep 09 15:00:36.233776 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:00:36.233786 2576 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/6bfe4cb5-2f10-42aa-9141-7d128b9c6e91-offline-tls\") on node \"ip-10-0-133-66.ec2.internal\" DevicePath \"\"" Sep 09 15:00:36.234050 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:00:36.233796 2576 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/6bfe4cb5-2f10-42aa-9141-7d128b9c6e91-ui-tls\") on node \"ip-10-0-133-66.ec2.internal\" DevicePath \"\"" Sep 09 15:00:36.234050 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:00:36.233803 2576 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/6bfe4cb5-2f10-42aa-9141-7d128b9c6e91-registry-tls\") on node \"ip-10-0-133-66.ec2.internal\" DevicePath \"\"" Sep 09 15:00:36.820487 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:00:36.820451 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-l5mkg" Sep 09 15:00:36.820960 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:00:36.820442 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-l5mkg" event={"ID":"eb9c3a35-6a4b-452c-89ca-0ce13611a7a8","Type":"ContainerDied","Data":"6820c76600c12751a4535bf5b591c6c33961deb8c347e4f9410c562225411a60"} Sep 09 15:00:36.820960 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:00:36.820607 2576 scope.go:117] "RemoveContainer" containerID="a7b62de3e74f58adb054b51b7910f93568416a1dc4ce9f930ed2b5a436ee796b" Sep 09 15:00:36.823966 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:00:36.823936 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zggqz" event={"ID":"6bfe4cb5-2f10-42aa-9141-7d128b9c6e91","Type":"ContainerDied","Data":"7b57660aa3531411922cb2bec2ed145b332ca9cc58abca1f32662da61123cf4b"} Sep 09 15:00:36.824088 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:00:36.823990 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zggqz" Sep 09 15:00:36.831597 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:00:36.831576 2576 scope.go:117] "RemoveContainer" containerID="94b203e0256c877674be9314b3e109f425a516926bb486ec210ceba42933e860" Sep 09 15:00:36.840614 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:00:36.840592 2576 scope.go:117] "RemoveContainer" containerID="bbf5a549900c601d76201479a105c25ff173d8ee99a8c181c1e0a99e4bd7e771" Sep 09 15:00:36.844703 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:00:36.844678 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-l5mkg"] Sep 09 15:00:36.848595 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:00:36.848569 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-l5mkg"] Sep 09 15:00:36.850222 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:00:36.850205 2576 scope.go:117] "RemoveContainer" containerID="b75445db55a127ce6e860c143d9d403061e2e52c82920225fb442117cd8b1d85" Sep 09 15:00:36.858353 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:00:36.858333 2576 scope.go:117] "RemoveContainer" containerID="94c722e1f34813a5e5b727e0255bd85ef1643162f2b869212551d54dbf8be80b" Sep 09 15:00:36.865714 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:00:36.865677 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-zggqz"] Sep 09 15:00:36.867579 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:00:36.867553 2576 scope.go:117] "RemoveContainer" containerID="d4f0f616273c23dee12d55da52dac30a96838a033d3853d72d3d6189fbc06a3c" Sep 09 15:00:36.869447 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:00:36.869427 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-zggqz"] Sep 09 15:00:36.875966 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:00:36.875946 2576 scope.go:117] "RemoveContainer" containerID="914536ff7cedd943bd070f48dbe944ad53329b25d427c687e6aab4cdf3d0ac5a" Sep 09 15:00:36.884065 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:00:36.884044 2576 scope.go:117] "RemoveContainer" containerID="fe803cc52e67f746b7ab2c3c7225aaec5f301747116bbd90ae86dbef66580e3d" Sep 09 15:00:36.892098 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:00:36.892076 2576 scope.go:117] "RemoveContainer" containerID="db9c30bdde2d6c55b15ae4f3b8da6b9e38c2db3e8481e3470b661db21b05c3fe" Sep 09 15:00:36.900042 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:00:36.900022 2576 scope.go:117] "RemoveContainer" containerID="b73910956d643dea0ef518d0db5b114844de8c3fb1e0bd4ecd1021bd7566a2ab" Sep 09 15:00:36.908023 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:00:36.908005 2576 scope.go:117] "RemoveContainer" containerID="3c49a305cc19526431f56b71191839c150c85bc078e1b79d56d7a41ccf07313b" Sep 09 15:00:37.514799 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:00:37.514760 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6bfe4cb5-2f10-42aa-9141-7d128b9c6e91" path="/var/lib/kubelet/pods/6bfe4cb5-2f10-42aa-9141-7d128b9c6e91/volumes" Sep 09 15:00:37.515396 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:00:37.515381 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb9c3a35-6a4b-452c-89ca-0ce13611a7a8" path="/var/lib/kubelet/pods/eb9c3a35-6a4b-452c-89ca-0ce13611a7a8/volumes" Sep 09 15:01:17.400965 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:01:17.400930 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-89fjc"] Sep 09 15:01:17.401577 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:01:17.401414 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6bfe4cb5-2f10-42aa-9141-7d128b9c6e91" containerName="feast-init" Sep 09 15:01:17.401577 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:01:17.401431 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="6bfe4cb5-2f10-42aa-9141-7d128b9c6e91" containerName="feast-init" Sep 09 15:01:17.401577 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:01:17.401453 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6bfe4cb5-2f10-42aa-9141-7d128b9c6e91" containerName="online" Sep 09 15:01:17.401577 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:01:17.401463 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="6bfe4cb5-2f10-42aa-9141-7d128b9c6e91" containerName="online" Sep 09 15:01:17.401577 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:01:17.401474 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6bfe4cb5-2f10-42aa-9141-7d128b9c6e91" containerName="registry" Sep 09 15:01:17.401577 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:01:17.401483 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="6bfe4cb5-2f10-42aa-9141-7d128b9c6e91" containerName="registry" Sep 09 15:01:17.401577 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:01:17.401502 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6bfe4cb5-2f10-42aa-9141-7d128b9c6e91" containerName="feast-apply" Sep 09 15:01:17.401577 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:01:17.401511 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="6bfe4cb5-2f10-42aa-9141-7d128b9c6e91" containerName="feast-apply" Sep 09 15:01:17.401577 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:01:17.401520 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb9c3a35-6a4b-452c-89ca-0ce13611a7a8" containerName="feast-apply" Sep 09 15:01:17.401577 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:01:17.401529 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb9c3a35-6a4b-452c-89ca-0ce13611a7a8" containerName="feast-apply" Sep 09 15:01:17.401577 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:01:17.401539 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb9c3a35-6a4b-452c-89ca-0ce13611a7a8" containerName="online" Sep 09 15:01:17.401577 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:01:17.401575 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb9c3a35-6a4b-452c-89ca-0ce13611a7a8" containerName="online" Sep 09 15:01:17.401577 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:01:17.401584 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6bfe4cb5-2f10-42aa-9141-7d128b9c6e91" containerName="offline" Sep 09 15:01:17.402264 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:01:17.401593 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="6bfe4cb5-2f10-42aa-9141-7d128b9c6e91" containerName="offline" Sep 09 15:01:17.402264 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:01:17.401602 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb9c3a35-6a4b-452c-89ca-0ce13611a7a8" containerName="ui" Sep 09 15:01:17.402264 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:01:17.401611 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb9c3a35-6a4b-452c-89ca-0ce13611a7a8" containerName="ui" Sep 09 15:01:17.402264 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:01:17.401623 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb9c3a35-6a4b-452c-89ca-0ce13611a7a8" containerName="feast-init" Sep 09 15:01:17.402264 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:01:17.401630 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb9c3a35-6a4b-452c-89ca-0ce13611a7a8" containerName="feast-init" Sep 09 15:01:17.402264 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:01:17.401640 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6bfe4cb5-2f10-42aa-9141-7d128b9c6e91" containerName="ui" Sep 09 15:01:17.402264 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:01:17.401648 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="6bfe4cb5-2f10-42aa-9141-7d128b9c6e91" containerName="ui" Sep 09 15:01:17.402264 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:01:17.401660 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb9c3a35-6a4b-452c-89ca-0ce13611a7a8" containerName="offline" Sep 09 15:01:17.402264 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:01:17.401668 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb9c3a35-6a4b-452c-89ca-0ce13611a7a8" containerName="offline" Sep 09 15:01:17.402264 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:01:17.401746 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb9c3a35-6a4b-452c-89ca-0ce13611a7a8" containerName="online" Sep 09 15:01:17.402264 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:01:17.401759 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="6bfe4cb5-2f10-42aa-9141-7d128b9c6e91" containerName="online" Sep 09 15:01:17.402264 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:01:17.401767 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="6bfe4cb5-2f10-42aa-9141-7d128b9c6e91" containerName="ui" Sep 09 15:01:17.402264 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:01:17.401776 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb9c3a35-6a4b-452c-89ca-0ce13611a7a8" containerName="ui" Sep 09 15:01:17.402264 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:01:17.401784 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="6bfe4cb5-2f10-42aa-9141-7d128b9c6e91" containerName="offline" Sep 09 15:01:17.402264 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:01:17.401793 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb9c3a35-6a4b-452c-89ca-0ce13611a7a8" containerName="offline" Sep 09 15:01:17.402264 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:01:17.401801 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="6bfe4cb5-2f10-42aa-9141-7d128b9c6e91" containerName="registry" Sep 09 15:01:17.405684 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:01:17.405664 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-89fjc" Sep 09 15:01:17.408880 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:01:17.408860 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-registry-tls\"" Sep 09 15:01:17.409450 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:01:17.409433 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-hbsck\"" Sep 09 15:01:17.409689 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:01:17.409337 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\"" Sep 09 15:01:17.409806 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:01:17.409385 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-ui-tls\"" Sep 09 15:01:17.409958 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:01:17.409679 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\"" Sep 09 15:01:17.410054 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:01:17.409198 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-offline-tls\"" Sep 09 15:01:17.410368 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:01:17.410351 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-online-tls\"" Sep 09 15:01:17.417768 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:01:17.417679 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-89fjc"] Sep 09 15:01:17.466018 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:01:17.465992 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/101e4323-4efa-4b07-9684-91709f613e98-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-89fjc\" (UID: \"101e4323-4efa-4b07-9684-91709f613e98\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-89fjc" Sep 09 15:01:17.466184 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:01:17.466053 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqx8x\" (UniqueName: \"kubernetes.io/projected/101e4323-4efa-4b07-9684-91709f613e98-kube-api-access-fqx8x\") pod \"feast-simple-feast-setup-57576b8fcf-89fjc\" (UID: \"101e4323-4efa-4b07-9684-91709f613e98\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-89fjc" Sep 09 15:01:17.466184 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:01:17.466118 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/101e4323-4efa-4b07-9684-91709f613e98-feast-data\") pod \"feast-simple-feast-setup-57576b8fcf-89fjc\" (UID: \"101e4323-4efa-4b07-9684-91709f613e98\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-89fjc" Sep 09 15:01:17.466351 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:01:17.466198 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/101e4323-4efa-4b07-9684-91709f613e98-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-89fjc\" (UID: \"101e4323-4efa-4b07-9684-91709f613e98\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-89fjc" Sep 09 15:01:17.466351 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:01:17.466247 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/101e4323-4efa-4b07-9684-91709f613e98-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-89fjc\" (UID: \"101e4323-4efa-4b07-9684-91709f613e98\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-89fjc" Sep 09 15:01:17.466351 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:01:17.466283 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/101e4323-4efa-4b07-9684-91709f613e98-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-89fjc\" (UID: \"101e4323-4efa-4b07-9684-91709f613e98\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-89fjc" Sep 09 15:01:17.566802 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:01:17.566772 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/101e4323-4efa-4b07-9684-91709f613e98-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-89fjc\" (UID: \"101e4323-4efa-4b07-9684-91709f613e98\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-89fjc" Sep 09 15:01:17.566979 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:01:17.566820 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/101e4323-4efa-4b07-9684-91709f613e98-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-89fjc\" (UID: \"101e4323-4efa-4b07-9684-91709f613e98\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-89fjc" Sep 09 15:01:17.566979 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:01:17.566890 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqx8x\" (UniqueName: \"kubernetes.io/projected/101e4323-4efa-4b07-9684-91709f613e98-kube-api-access-fqx8x\") pod \"feast-simple-feast-setup-57576b8fcf-89fjc\" (UID: \"101e4323-4efa-4b07-9684-91709f613e98\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-89fjc" Sep 09 15:01:17.566979 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:01:17.566916 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/101e4323-4efa-4b07-9684-91709f613e98-feast-data\") pod \"feast-simple-feast-setup-57576b8fcf-89fjc\" (UID: \"101e4323-4efa-4b07-9684-91709f613e98\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-89fjc" Sep 09 15:01:17.566979 ip-10-0-133-66 kubenswrapper[2576]: E0909 15:01:17.566939 2576 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-offline-tls: secret "feast-simple-feast-setup-offline-tls" not found Sep 09 15:01:17.567207 ip-10-0-133-66 kubenswrapper[2576]: E0909 15:01:17.567004 2576 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-online-tls: secret "feast-simple-feast-setup-online-tls" not found Sep 09 15:01:17.567207 ip-10-0-133-66 kubenswrapper[2576]: E0909 15:01:17.567016 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/101e4323-4efa-4b07-9684-91709f613e98-offline-tls podName:101e4323-4efa-4b07-9684-91709f613e98 nodeName:}" failed. No retries permitted until 2026-09-09 15:01:18.066997362 +0000 UTC m=+1693.077913606 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/101e4323-4efa-4b07-9684-91709f613e98-offline-tls") pod "feast-simple-feast-setup-57576b8fcf-89fjc" (UID: "101e4323-4efa-4b07-9684-91709f613e98") : secret "feast-simple-feast-setup-offline-tls" not found Sep 09 15:01:17.567207 ip-10-0-133-66 kubenswrapper[2576]: E0909 15:01:17.567062 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/101e4323-4efa-4b07-9684-91709f613e98-online-tls podName:101e4323-4efa-4b07-9684-91709f613e98 nodeName:}" failed. No retries permitted until 2026-09-09 15:01:18.067045711 +0000 UTC m=+1693.077961964 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/101e4323-4efa-4b07-9684-91709f613e98-online-tls") pod "feast-simple-feast-setup-57576b8fcf-89fjc" (UID: "101e4323-4efa-4b07-9684-91709f613e98") : secret "feast-simple-feast-setup-online-tls" not found Sep 09 15:01:17.567207 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:01:17.566947 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/101e4323-4efa-4b07-9684-91709f613e98-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-89fjc\" (UID: \"101e4323-4efa-4b07-9684-91709f613e98\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-89fjc" Sep 09 15:01:17.567207 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:01:17.567151 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/101e4323-4efa-4b07-9684-91709f613e98-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-89fjc\" (UID: \"101e4323-4efa-4b07-9684-91709f613e98\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-89fjc" Sep 09 15:01:17.567512 ip-10-0-133-66 kubenswrapper[2576]: E0909 15:01:17.566939 2576 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-ui-tls: secret "feast-simple-feast-setup-ui-tls" not found Sep 09 15:01:17.567512 ip-10-0-133-66 kubenswrapper[2576]: E0909 15:01:17.567263 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/101e4323-4efa-4b07-9684-91709f613e98-ui-tls podName:101e4323-4efa-4b07-9684-91709f613e98 nodeName:}" failed. No retries permitted until 2026-09-09 15:01:18.067249704 +0000 UTC m=+1693.078165933 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "ui-tls" (UniqueName: "kubernetes.io/secret/101e4323-4efa-4b07-9684-91709f613e98-ui-tls") pod "feast-simple-feast-setup-57576b8fcf-89fjc" (UID: "101e4323-4efa-4b07-9684-91709f613e98") : secret "feast-simple-feast-setup-ui-tls" not found Sep 09 15:01:17.567512 ip-10-0-133-66 kubenswrapper[2576]: E0909 15:01:17.567326 2576 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-registry-tls: secret "feast-simple-feast-setup-registry-tls" not found Sep 09 15:01:17.567512 ip-10-0-133-66 kubenswrapper[2576]: E0909 15:01:17.567374 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/101e4323-4efa-4b07-9684-91709f613e98-registry-tls podName:101e4323-4efa-4b07-9684-91709f613e98 nodeName:}" failed. No retries permitted until 2026-09-09 15:01:18.067363347 +0000 UTC m=+1693.078279584 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/101e4323-4efa-4b07-9684-91709f613e98-registry-tls") pod "feast-simple-feast-setup-57576b8fcf-89fjc" (UID: "101e4323-4efa-4b07-9684-91709f613e98") : secret "feast-simple-feast-setup-registry-tls" not found Sep 09 15:01:17.567512 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:01:17.567432 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/101e4323-4efa-4b07-9684-91709f613e98-feast-data\") pod \"feast-simple-feast-setup-57576b8fcf-89fjc\" (UID: \"101e4323-4efa-4b07-9684-91709f613e98\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-89fjc" Sep 09 15:01:17.578452 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:01:17.578430 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqx8x\" (UniqueName: \"kubernetes.io/projected/101e4323-4efa-4b07-9684-91709f613e98-kube-api-access-fqx8x\") pod \"feast-simple-feast-setup-57576b8fcf-89fjc\" (UID: \"101e4323-4efa-4b07-9684-91709f613e98\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-89fjc" Sep 09 15:01:18.072235 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:01:18.072196 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/101e4323-4efa-4b07-9684-91709f613e98-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-89fjc\" (UID: \"101e4323-4efa-4b07-9684-91709f613e98\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-89fjc" Sep 09 15:01:18.072447 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:01:18.072245 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/101e4323-4efa-4b07-9684-91709f613e98-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-89fjc\" (UID: \"101e4323-4efa-4b07-9684-91709f613e98\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-89fjc" Sep 09 15:01:18.072447 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:01:18.072277 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/101e4323-4efa-4b07-9684-91709f613e98-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-89fjc\" (UID: \"101e4323-4efa-4b07-9684-91709f613e98\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-89fjc" Sep 09 15:01:18.072447 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:01:18.072342 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/101e4323-4efa-4b07-9684-91709f613e98-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-89fjc\" (UID: \"101e4323-4efa-4b07-9684-91709f613e98\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-89fjc" Sep 09 15:01:18.072636 ip-10-0-133-66 kubenswrapper[2576]: E0909 15:01:18.072472 2576 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-offline-tls: secret "feast-simple-feast-setup-offline-tls" not found Sep 09 15:01:18.072636 ip-10-0-133-66 kubenswrapper[2576]: E0909 15:01:18.072549 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/101e4323-4efa-4b07-9684-91709f613e98-offline-tls podName:101e4323-4efa-4b07-9684-91709f613e98 nodeName:}" failed. No retries permitted until 2026-09-09 15:01:19.072529688 +0000 UTC m=+1694.083445931 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/101e4323-4efa-4b07-9684-91709f613e98-offline-tls") pod "feast-simple-feast-setup-57576b8fcf-89fjc" (UID: "101e4323-4efa-4b07-9684-91709f613e98") : secret "feast-simple-feast-setup-offline-tls" not found Sep 09 15:01:18.074958 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:01:18.074925 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/101e4323-4efa-4b07-9684-91709f613e98-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-89fjc\" (UID: \"101e4323-4efa-4b07-9684-91709f613e98\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-89fjc" Sep 09 15:01:18.075072 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:01:18.074932 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/101e4323-4efa-4b07-9684-91709f613e98-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-89fjc\" (UID: \"101e4323-4efa-4b07-9684-91709f613e98\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-89fjc" Sep 09 15:01:18.075072 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:01:18.075060 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/101e4323-4efa-4b07-9684-91709f613e98-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-89fjc\" (UID: \"101e4323-4efa-4b07-9684-91709f613e98\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-89fjc" Sep 09 15:01:19.080459 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:01:19.080415 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/101e4323-4efa-4b07-9684-91709f613e98-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-89fjc\" (UID: \"101e4323-4efa-4b07-9684-91709f613e98\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-89fjc" Sep 09 15:01:19.082973 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:01:19.082953 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/101e4323-4efa-4b07-9684-91709f613e98-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-89fjc\" (UID: \"101e4323-4efa-4b07-9684-91709f613e98\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-89fjc" Sep 09 15:01:19.224233 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:01:19.224201 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-89fjc" Sep 09 15:01:19.502672 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:01:19.502646 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-89fjc"] Sep 09 15:01:19.505034 ip-10-0-133-66 kubenswrapper[2576]: W0909 15:01:19.505005 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod101e4323_4efa_4b07_9684_91709f613e98.slice/crio-64a0413ef8c162dea18a7295abce18f2dd36a2a3049de4ea60e8689f7b4e31d7 WatchSource:0}: Error finding container 64a0413ef8c162dea18a7295abce18f2dd36a2a3049de4ea60e8689f7b4e31d7: Status 404 returned error can't find the container with id 64a0413ef8c162dea18a7295abce18f2dd36a2a3049de4ea60e8689f7b4e31d7 Sep 09 15:01:19.998840 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:01:19.998794 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-89fjc" event={"ID":"101e4323-4efa-4b07-9684-91709f613e98","Type":"ContainerStarted","Data":"4cf9223c2c9b565271b310f3f65424b432cb2924c935decddfc598b461659958"} Sep 09 15:01:19.998840 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:01:19.998838 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-89fjc" event={"ID":"101e4323-4efa-4b07-9684-91709f613e98","Type":"ContainerStarted","Data":"64a0413ef8c162dea18a7295abce18f2dd36a2a3049de4ea60e8689f7b4e31d7"} Sep 09 15:01:25.021634 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:01:25.021595 2576 generic.go:356] "Generic (PLEG): container finished" podID="101e4323-4efa-4b07-9684-91709f613e98" containerID="4cf9223c2c9b565271b310f3f65424b432cb2924c935decddfc598b461659958" exitCode=0 Sep 09 15:01:25.022151 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:01:25.021679 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-89fjc" event={"ID":"101e4323-4efa-4b07-9684-91709f613e98","Type":"ContainerDied","Data":"4cf9223c2c9b565271b310f3f65424b432cb2924c935decddfc598b461659958"} Sep 09 15:01:26.028036 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:01:26.027999 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-89fjc" event={"ID":"101e4323-4efa-4b07-9684-91709f613e98","Type":"ContainerStarted","Data":"9e3495fb350aef9c102c025c81f31792537147ad964c3babae5134bc8fd2a5cf"} Sep 09 15:01:32.055791 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:01:32.055751 2576 generic.go:356] "Generic (PLEG): container finished" podID="101e4323-4efa-4b07-9684-91709f613e98" containerID="9e3495fb350aef9c102c025c81f31792537147ad964c3babae5134bc8fd2a5cf" exitCode=0 Sep 09 15:01:32.056196 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:01:32.055817 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-89fjc" event={"ID":"101e4323-4efa-4b07-9684-91709f613e98","Type":"ContainerDied","Data":"9e3495fb350aef9c102c025c81f31792537147ad964c3babae5134bc8fd2a5cf"} Sep 09 15:01:33.066136 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:01:33.066093 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-89fjc" event={"ID":"101e4323-4efa-4b07-9684-91709f613e98","Type":"ContainerStarted","Data":"12e4bb385645f61b3b82c1133bc427a62f831ada981936cffa3d47f0549b3d86"} Sep 09 15:01:33.066639 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:01:33.066160 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-89fjc" event={"ID":"101e4323-4efa-4b07-9684-91709f613e98","Type":"ContainerStarted","Data":"6fcc61e76d3a9de2b2a13abc560657be46f061cd2223699beb92d4df59bc1eec"} Sep 09 15:01:33.066639 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:01:33.066180 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-89fjc" event={"ID":"101e4323-4efa-4b07-9684-91709f613e98","Type":"ContainerStarted","Data":"65dd9b26c02b38ce76b040dff4504d15b4c48df8758f6c9262fb1bc4ae2b0416"} Sep 09 15:01:33.066639 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:01:33.066199 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-89fjc" event={"ID":"101e4323-4efa-4b07-9684-91709f613e98","Type":"ContainerStarted","Data":"89f7ceda83f30a1245f67fb796e9e87195eb0bae1a88b14bbeeeab3ff67da551"} Sep 09 15:01:33.091007 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:01:33.090941 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-89fjc" podStartSLOduration=16.090918041 podStartE2EDuration="16.090918041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:01:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:01:33.090690565 +0000 UTC m=+1708.101606817" watchObservedRunningTime="2026-09-09 15:01:33.090918041 +0000 UTC m=+1708.101834294" Sep 09 15:01:34.224644 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:01:34.224600 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-89fjc" Sep 09 15:01:34.225231 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:01:34.225210 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-89fjc" Sep 09 15:01:34.225356 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:01:34.225242 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-89fjc" Sep 09 15:01:34.225356 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:01:34.225260 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-89fjc" Sep 09 15:01:34.226908 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:01:34.226869 2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-89fjc" podUID="101e4323-4efa-4b07-9684-91709f613e98" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.28:8443: connect: connection refused" Sep 09 15:01:34.226908 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:01:34.226884 2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-89fjc" podUID="101e4323-4efa-4b07-9684-91709f613e98" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.28:6571: connect: connection refused" Sep 09 15:01:34.227123 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:01:34.226992 2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-89fjc" podUID="101e4323-4efa-4b07-9684-91709f613e98" containerName="online" probeResult="failure" output="Get \"https://10.132.0.28:6567/health\": dial tcp 10.132.0.28:6567: connect: connection refused" Sep 09 15:01:34.227123 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:01:34.227009 2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-89fjc" podUID="101e4323-4efa-4b07-9684-91709f613e98" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.28:8816: connect: connection refused" Sep 09 15:01:37.225464 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:01:37.225415 2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-89fjc" podUID="101e4323-4efa-4b07-9684-91709f613e98" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.28:8816: connect: connection refused" Sep 09 15:01:37.226093 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:01:37.225415 2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-89fjc" podUID="101e4323-4efa-4b07-9684-91709f613e98" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.28:8443: connect: connection refused" Sep 09 15:01:37.226230 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:01:37.225453 2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-89fjc" podUID="101e4323-4efa-4b07-9684-91709f613e98" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.28:6571: connect: connection refused" Sep 09 15:01:37.226230 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:01:37.225503 2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-89fjc" podUID="101e4323-4efa-4b07-9684-91709f613e98" containerName="online" probeResult="failure" output="Get \"https://10.132.0.28:6567/health\": dial tcp 10.132.0.28:6567: connect: connection refused" Sep 09 15:01:39.224947 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:01:39.224893 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-89fjc" Sep 09 15:01:39.225503 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:01:39.224979 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-89fjc" Sep 09 15:01:39.225503 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:01:39.225000 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-89fjc" Sep 09 15:01:39.225503 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:01:39.225020 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-89fjc" Sep 09 15:01:40.225278 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:01:40.225228 2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-89fjc" podUID="101e4323-4efa-4b07-9684-91709f613e98" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.28:6571: connect: connection refused" Sep 09 15:01:40.225810 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:01:40.225220 2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-89fjc" podUID="101e4323-4efa-4b07-9684-91709f613e98" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.28:8816: connect: connection refused" Sep 09 15:01:40.225810 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:01:40.225221 2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-89fjc" podUID="101e4323-4efa-4b07-9684-91709f613e98" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.28:8443: connect: connection refused" Sep 09 15:01:40.225810 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:01:40.225238 2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-89fjc" podUID="101e4323-4efa-4b07-9684-91709f613e98" containerName="online" probeResult="failure" output="Get \"https://10.132.0.28:6567/health\": dial tcp 10.132.0.28:6567: connect: connection refused" Sep 09 15:01:43.225548 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:01:43.225515 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-89fjc" Sep 09 15:01:43.226048 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:01:43.225664 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-89fjc" Sep 09 15:01:43.226048 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:01:43.225767 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-89fjc" Sep 09 15:01:43.226403 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:01:43.226377 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-89fjc" Sep 09 15:01:43.226535 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:01:43.226448 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-89fjc" Sep 09 15:01:43.226535 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:01:43.226481 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-89fjc" Sep 09 15:01:43.230983 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:01:43.230961 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-89fjc" Sep 09 15:01:44.127627 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:01:44.127600 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-89fjc" Sep 09 15:02:46.630963 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:02:46.630924 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-89fjc"] Sep 09 15:02:46.631562 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:02:46.631154 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-89fjc" podUID="101e4323-4efa-4b07-9684-91709f613e98" containerName="registry" containerID="cri-o://89f7ceda83f30a1245f67fb796e9e87195eb0bae1a88b14bbeeeab3ff67da551" gracePeriod=30 Sep 09 15:02:46.631562 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:02:46.631231 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-89fjc" podUID="101e4323-4efa-4b07-9684-91709f613e98" containerName="offline" containerID="cri-o://6fcc61e76d3a9de2b2a13abc560657be46f061cd2223699beb92d4df59bc1eec" gracePeriod=30 Sep 09 15:02:46.631562 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:02:46.631236 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-89fjc" podUID="101e4323-4efa-4b07-9684-91709f613e98" containerName="online" containerID="cri-o://65dd9b26c02b38ce76b040dff4504d15b4c48df8758f6c9262fb1bc4ae2b0416" gracePeriod=30 Sep 09 15:02:46.631562 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:02:46.631229 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-89fjc" podUID="101e4323-4efa-4b07-9684-91709f613e98" containerName="ui" containerID="cri-o://12e4bb385645f61b3b82c1133bc427a62f831ada981936cffa3d47f0549b3d86" gracePeriod=30 Sep 09 15:02:47.131242 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:02:47.131195 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-l2v8b"] Sep 09 15:02:47.135788 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:02:47.135764 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l2v8b" Sep 09 15:02:47.138971 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:02:47.138770 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-gpqkq\"" Sep 09 15:02:47.147748 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:02:47.147724 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/27a978af-ee9e-4dfc-bca6-d877ddd92cc1-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-l2v8b\" (UID: \"27a978af-ee9e-4dfc-bca6-d877ddd92cc1\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l2v8b" Sep 09 15:02:47.147894 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:02:47.147763 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/27a978af-ee9e-4dfc-bca6-d877ddd92cc1-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-l2v8b\" (UID: \"27a978af-ee9e-4dfc-bca6-d877ddd92cc1\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l2v8b" Sep 09 15:02:47.147894 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:02:47.147829 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/27a978af-ee9e-4dfc-bca6-d877ddd92cc1-feast-data\") pod \"feast-simple-feast-setup-57576b8fcf-l2v8b\" (UID: \"27a978af-ee9e-4dfc-bca6-d877ddd92cc1\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l2v8b" Sep 09 15:02:47.147894 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:02:47.147856 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkkcg\" (UniqueName: \"kubernetes.io/projected/27a978af-ee9e-4dfc-bca6-d877ddd92cc1-kube-api-access-nkkcg\") pod \"feast-simple-feast-setup-57576b8fcf-l2v8b\" (UID: \"27a978af-ee9e-4dfc-bca6-d877ddd92cc1\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l2v8b" Sep 09 15:02:47.148062 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:02:47.147915 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/27a978af-ee9e-4dfc-bca6-d877ddd92cc1-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-l2v8b\" (UID: \"27a978af-ee9e-4dfc-bca6-d877ddd92cc1\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l2v8b" Sep 09 15:02:47.148062 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:02:47.147957 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/27a978af-ee9e-4dfc-bca6-d877ddd92cc1-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-l2v8b\" (UID: \"27a978af-ee9e-4dfc-bca6-d877ddd92cc1\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l2v8b" Sep 09 15:02:47.152544 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:02:47.152487 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-l2v8b"] Sep 09 15:02:47.248486 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:02:47.248433 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/27a978af-ee9e-4dfc-bca6-d877ddd92cc1-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-l2v8b\" (UID: \"27a978af-ee9e-4dfc-bca6-d877ddd92cc1\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l2v8b" Sep 09 15:02:47.248694 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:02:47.248502 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/27a978af-ee9e-4dfc-bca6-d877ddd92cc1-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-l2v8b\" (UID: \"27a978af-ee9e-4dfc-bca6-d877ddd92cc1\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l2v8b" Sep 09 15:02:47.248694 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:02:47.248536 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/27a978af-ee9e-4dfc-bca6-d877ddd92cc1-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-l2v8b\" (UID: \"27a978af-ee9e-4dfc-bca6-d877ddd92cc1\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l2v8b" Sep 09 15:02:47.248694 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:02:47.248568 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/27a978af-ee9e-4dfc-bca6-d877ddd92cc1-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-l2v8b\" (UID: \"27a978af-ee9e-4dfc-bca6-d877ddd92cc1\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l2v8b" Sep 09 15:02:47.248694 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:02:47.248619 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/27a978af-ee9e-4dfc-bca6-d877ddd92cc1-feast-data\") pod \"feast-simple-feast-setup-57576b8fcf-l2v8b\" (UID: \"27a978af-ee9e-4dfc-bca6-d877ddd92cc1\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l2v8b" Sep 09 15:02:47.248694 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:02:47.248646 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkkcg\" (UniqueName: \"kubernetes.io/projected/27a978af-ee9e-4dfc-bca6-d877ddd92cc1-kube-api-access-nkkcg\") pod \"feast-simple-feast-setup-57576b8fcf-l2v8b\" (UID: \"27a978af-ee9e-4dfc-bca6-d877ddd92cc1\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l2v8b" Sep 09 15:02:47.249210 ip-10-0-133-66 kubenswrapper[2576]: E0909 15:02:47.249014 2576 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-offline-tls: secret "feast-simple-feast-setup-offline-tls" not found Sep 09 15:02:47.249210 ip-10-0-133-66 kubenswrapper[2576]: E0909 15:02:47.249091 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27a978af-ee9e-4dfc-bca6-d877ddd92cc1-offline-tls podName:27a978af-ee9e-4dfc-bca6-d877ddd92cc1 nodeName:}" failed. No retries permitted until 2026-09-09 15:02:47.749071276 +0000 UTC m=+1782.759987511 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/27a978af-ee9e-4dfc-bca6-d877ddd92cc1-offline-tls") pod "feast-simple-feast-setup-57576b8fcf-l2v8b" (UID: "27a978af-ee9e-4dfc-bca6-d877ddd92cc1") : secret "feast-simple-feast-setup-offline-tls" not found Sep 09 15:02:47.249210 ip-10-0-133-66 kubenswrapper[2576]: E0909 15:02:47.249091 2576 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-ui-tls: secret "feast-simple-feast-setup-ui-tls" not found Sep 09 15:02:47.249210 ip-10-0-133-66 kubenswrapper[2576]: E0909 15:02:47.249171 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27a978af-ee9e-4dfc-bca6-d877ddd92cc1-ui-tls podName:27a978af-ee9e-4dfc-bca6-d877ddd92cc1 nodeName:}" failed. No retries permitted until 2026-09-09 15:02:47.749152367 +0000 UTC m=+1782.760068601 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "ui-tls" (UniqueName: "kubernetes.io/secret/27a978af-ee9e-4dfc-bca6-d877ddd92cc1-ui-tls") pod "feast-simple-feast-setup-57576b8fcf-l2v8b" (UID: "27a978af-ee9e-4dfc-bca6-d877ddd92cc1") : secret "feast-simple-feast-setup-ui-tls" not found Sep 09 15:02:47.249682 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:02:47.249445 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/27a978af-ee9e-4dfc-bca6-d877ddd92cc1-feast-data\") pod \"feast-simple-feast-setup-57576b8fcf-l2v8b\" (UID: \"27a978af-ee9e-4dfc-bca6-d877ddd92cc1\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l2v8b" Sep 09 15:02:47.252189 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:02:47.252124 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/27a978af-ee9e-4dfc-bca6-d877ddd92cc1-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-l2v8b\" (UID: \"27a978af-ee9e-4dfc-bca6-d877ddd92cc1\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l2v8b" Sep 09 15:02:47.252189 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:02:47.252175 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/27a978af-ee9e-4dfc-bca6-d877ddd92cc1-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-l2v8b\" (UID: \"27a978af-ee9e-4dfc-bca6-d877ddd92cc1\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l2v8b" Sep 09 15:02:47.258166 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:02:47.258126 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkkcg\" (UniqueName: \"kubernetes.io/projected/27a978af-ee9e-4dfc-bca6-d877ddd92cc1-kube-api-access-nkkcg\") pod \"feast-simple-feast-setup-57576b8fcf-l2v8b\" (UID: \"27a978af-ee9e-4dfc-bca6-d877ddd92cc1\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l2v8b" Sep 09 15:02:47.752709 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:02:47.752616 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/27a978af-ee9e-4dfc-bca6-d877ddd92cc1-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-l2v8b\" (UID: \"27a978af-ee9e-4dfc-bca6-d877ddd92cc1\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l2v8b" Sep 09 15:02:47.752709 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:02:47.752661 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/27a978af-ee9e-4dfc-bca6-d877ddd92cc1-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-l2v8b\" (UID: \"27a978af-ee9e-4dfc-bca6-d877ddd92cc1\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l2v8b" Sep 09 15:02:47.755394 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:02:47.755372 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/27a978af-ee9e-4dfc-bca6-d877ddd92cc1-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-l2v8b\" (UID: \"27a978af-ee9e-4dfc-bca6-d877ddd92cc1\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l2v8b" Sep 09 15:02:47.755472 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:02:47.755372 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/27a978af-ee9e-4dfc-bca6-d877ddd92cc1-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-l2v8b\" (UID: \"27a978af-ee9e-4dfc-bca6-d877ddd92cc1\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l2v8b" Sep 09 15:02:48.047065 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:02:48.047038 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l2v8b" Sep 09 15:02:48.324393 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:02:48.324361 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-l2v8b"] Sep 09 15:02:48.325538 ip-10-0-133-66 kubenswrapper[2576]: W0909 15:02:48.325512 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod27a978af_ee9e_4dfc_bca6_d877ddd92cc1.slice/crio-323b3ef507c24f2cce2e14fe3de9ed3fbbb62f84635fff4668daf6379c58d402 WatchSource:0}: Error finding container 323b3ef507c24f2cce2e14fe3de9ed3fbbb62f84635fff4668daf6379c58d402: Status 404 returned error can't find the container with id 323b3ef507c24f2cce2e14fe3de9ed3fbbb62f84635fff4668daf6379c58d402 Sep 09 15:02:48.392089 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:02:48.392056 2576 generic.go:356] "Generic (PLEG): container finished" podID="101e4323-4efa-4b07-9684-91709f613e98" containerID="12e4bb385645f61b3b82c1133bc427a62f831ada981936cffa3d47f0549b3d86" exitCode=0 Sep 09 15:02:48.392089 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:02:48.392088 2576 generic.go:356] "Generic (PLEG): container finished" podID="101e4323-4efa-4b07-9684-91709f613e98" containerID="65dd9b26c02b38ce76b040dff4504d15b4c48df8758f6c9262fb1bc4ae2b0416" exitCode=0 Sep 09 15:02:48.392282 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:02:48.392120 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-89fjc" event={"ID":"101e4323-4efa-4b07-9684-91709f613e98","Type":"ContainerDied","Data":"12e4bb385645f61b3b82c1133bc427a62f831ada981936cffa3d47f0549b3d86"} Sep 09 15:02:48.392282 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:02:48.392168 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-89fjc" event={"ID":"101e4323-4efa-4b07-9684-91709f613e98","Type":"ContainerDied","Data":"65dd9b26c02b38ce76b040dff4504d15b4c48df8758f6c9262fb1bc4ae2b0416"} Sep 09 15:02:48.393652 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:02:48.393631 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l2v8b" event={"ID":"27a978af-ee9e-4dfc-bca6-d877ddd92cc1","Type":"ContainerStarted","Data":"c765e510ac62a829653001e4e64a09a44f8f772a3c905be79823c9c77dbaac40"} Sep 09 15:02:48.393775 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:02:48.393661 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l2v8b" event={"ID":"27a978af-ee9e-4dfc-bca6-d877ddd92cc1","Type":"ContainerStarted","Data":"323b3ef507c24f2cce2e14fe3de9ed3fbbb62f84635fff4668daf6379c58d402"} Sep 09 15:02:53.227248 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:02:53.227195 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-89fjc" podUID="101e4323-4efa-4b07-9684-91709f613e98" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.28:8443: connect: connection refused" Sep 09 15:02:53.416285 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:02:53.416252 2576 generic.go:356] "Generic (PLEG): container finished" podID="27a978af-ee9e-4dfc-bca6-d877ddd92cc1" containerID="c765e510ac62a829653001e4e64a09a44f8f772a3c905be79823c9c77dbaac40" exitCode=0 Sep 09 15:02:53.416472 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:02:53.416333 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l2v8b" event={"ID":"27a978af-ee9e-4dfc-bca6-d877ddd92cc1","Type":"ContainerDied","Data":"c765e510ac62a829653001e4e64a09a44f8f772a3c905be79823c9c77dbaac40"} Sep 09 15:02:54.124560 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:02:54.124516 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-89fjc" podUID="101e4323-4efa-4b07-9684-91709f613e98" containerName="online" probeResult="failure" output="Get \"https://10.132.0.28:6567/health\": dial tcp 10.132.0.28:6567: connect: connection refused" Sep 09 15:02:54.422870 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:02:54.422774 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l2v8b" event={"ID":"27a978af-ee9e-4dfc-bca6-d877ddd92cc1","Type":"ContainerStarted","Data":"c2f1ab7040e10614b73087e0db10de6bc717406e0161e73df2fc528e4cab4109"} Sep 09 15:03:00.456378 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:03:00.456338 2576 generic.go:356] "Generic (PLEG): container finished" podID="27a978af-ee9e-4dfc-bca6-d877ddd92cc1" containerID="c2f1ab7040e10614b73087e0db10de6bc717406e0161e73df2fc528e4cab4109" exitCode=0 Sep 09 15:03:00.456894 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:03:00.456391 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l2v8b" event={"ID":"27a978af-ee9e-4dfc-bca6-d877ddd92cc1","Type":"ContainerDied","Data":"c2f1ab7040e10614b73087e0db10de6bc717406e0161e73df2fc528e4cab4109"} Sep 09 15:03:01.467386 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:03:01.467331 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l2v8b" event={"ID":"27a978af-ee9e-4dfc-bca6-d877ddd92cc1","Type":"ContainerStarted","Data":"83be447974cb836d6d7d3d3dd8633e3d9ed45c30d5a7e311011b07e3019aed4d"} Sep 09 15:03:01.467888 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:03:01.467404 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l2v8b" event={"ID":"27a978af-ee9e-4dfc-bca6-d877ddd92cc1","Type":"ContainerStarted","Data":"1f99ad0b9f0bab53d69f244c17a9367906ed6c7a055dade5ea8de0bef888b257"} Sep 09 15:03:01.467888 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:03:01.467424 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l2v8b" event={"ID":"27a978af-ee9e-4dfc-bca6-d877ddd92cc1","Type":"ContainerStarted","Data":"78e3d67c6fb1dfe2db1c3c510a660aad8efbb36694dbce09a044c1c8fd46e0ab"} Sep 09 15:03:01.467888 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:03:01.467442 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l2v8b" event={"ID":"27a978af-ee9e-4dfc-bca6-d877ddd92cc1","Type":"ContainerStarted","Data":"23b3de2eec5fdcf3642621cad03a554f2dd1ced52e873ec250acc6b19c82323f"} Sep 09 15:03:01.494327 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:03:01.494238 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l2v8b" podStartSLOduration=14.494212073 podStartE2EDuration="14.494212073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:02:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:03:01.493385621 +0000 UTC m=+1796.504301897" watchObservedRunningTime="2026-09-09 15:03:01.494212073 +0000 UTC m=+1796.505128324" Sep 09 15:03:03.047643 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:03:03.047500 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l2v8b" Sep 09 15:03:03.047643 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:03:03.047572 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l2v8b" Sep 09 15:03:03.047643 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:03:03.047594 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l2v8b" Sep 09 15:03:03.047643 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:03:03.047614 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l2v8b" Sep 09 15:03:03.050226 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:03:03.050183 2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l2v8b" podUID="27a978af-ee9e-4dfc-bca6-d877ddd92cc1" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.29:8816: connect: connection refused" Sep 09 15:03:03.050226 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:03:03.050213 2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l2v8b" podUID="27a978af-ee9e-4dfc-bca6-d877ddd92cc1" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.29:6571: connect: connection refused" Sep 09 15:03:03.050500 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:03:03.050219 2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l2v8b" podUID="27a978af-ee9e-4dfc-bca6-d877ddd92cc1" containerName="online" probeResult="failure" output="Get \"https://10.132.0.29:6567/health\": dial tcp 10.132.0.29:6567: connect: connection refused" Sep 09 15:03:03.050500 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:03:03.050268 2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l2v8b" podUID="27a978af-ee9e-4dfc-bca6-d877ddd92cc1" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.29:8443: connect: connection refused" Sep 09 15:03:03.226745 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:03:03.226685 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-89fjc" podUID="101e4323-4efa-4b07-9684-91709f613e98" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.28:8443: connect: connection refused" Sep 09 15:03:04.125145 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:03:04.125092 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-89fjc" podUID="101e4323-4efa-4b07-9684-91709f613e98" containerName="online" probeResult="failure" output="Get \"https://10.132.0.28:6567/health\": dial tcp 10.132.0.28:6567: connect: connection refused" Sep 09 15:03:05.523358 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:03:05.523329 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7xcn5_186aab90-266b-4eae-977a-3f053b423ef3/ovn-acl-logging/0.log" Sep 09 15:03:05.528839 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:03:05.528809 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7xcn5_186aab90-266b-4eae-977a-3f053b423ef3/ovn-acl-logging/0.log" Sep 09 15:03:06.048571 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:03:06.048503 2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l2v8b" podUID="27a978af-ee9e-4dfc-bca6-d877ddd92cc1" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.29:6571: connect: connection refused" Sep 09 15:03:06.048804 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:03:06.048501 2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l2v8b" podUID="27a978af-ee9e-4dfc-bca6-d877ddd92cc1" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.29:8816: connect: connection refused" Sep 09 15:03:06.048804 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:03:06.048524 2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l2v8b" podUID="27a978af-ee9e-4dfc-bca6-d877ddd92cc1" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.29:8443: connect: connection refused" Sep 09 15:03:06.048804 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:03:06.048531 2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l2v8b" podUID="27a978af-ee9e-4dfc-bca6-d877ddd92cc1" containerName="online" probeResult="failure" output="Get \"https://10.132.0.29:6567/health\": dial tcp 10.132.0.29:6567: connect: connection refused" Sep 09 15:03:08.048030 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:03:08.047971 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l2v8b" Sep 09 15:03:08.048687 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:03:08.048651 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l2v8b" Sep 09 15:03:08.048828 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:03:08.048698 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l2v8b" Sep 09 15:03:08.048828 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:03:08.048721 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l2v8b" Sep 09 15:03:09.048360 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:03:09.048294 2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l2v8b" podUID="27a978af-ee9e-4dfc-bca6-d877ddd92cc1" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.29:6571: connect: connection refused" Sep 09 15:03:09.048865 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:03:09.048535 2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l2v8b" podUID="27a978af-ee9e-4dfc-bca6-d877ddd92cc1" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.29:8443: connect: connection refused" Sep 09 15:03:09.048989 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:03:09.048457 2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l2v8b" podUID="27a978af-ee9e-4dfc-bca6-d877ddd92cc1" containerName="online" probeResult="failure" output="Get \"https://10.132.0.29:6567/health\": dial tcp 10.132.0.29:6567: connect: connection refused" Sep 09 15:03:09.049462 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:03:09.049435 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l2v8b" Sep 09 15:03:09.515502 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:03:09.515470 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l2v8b" Sep 09 15:03:12.048383 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:03:12.048351 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l2v8b" Sep 09 15:03:12.048836 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:03:12.048482 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l2v8b" Sep 09 15:03:12.049160 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:03:12.049133 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l2v8b" Sep 09 15:03:12.049470 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:03:12.049438 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l2v8b" Sep 09 15:03:12.054140 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:03:12.054108 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l2v8b" Sep 09 15:03:12.531994 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:03:12.531960 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l2v8b" Sep 09 15:03:13.226924 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:03:13.226874 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-89fjc" podUID="101e4323-4efa-4b07-9684-91709f613e98" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.28:8443: connect: connection refused" Sep 09 15:03:13.227415 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:03:13.227025 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-89fjc" Sep 09 15:03:14.125259 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:03:14.125156 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-89fjc" podUID="101e4323-4efa-4b07-9684-91709f613e98" containerName="online" probeResult="failure" output="Get \"https://10.132.0.28:6567/health\": dial tcp 10.132.0.28:6567: connect: connection refused" Sep 09 15:03:14.131325 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:03:14.125918 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-89fjc" Sep 09 15:03:17.285355 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:03:17.285323 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-89fjc" Sep 09 15:03:17.421976 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:03:17.421877 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/101e4323-4efa-4b07-9684-91709f613e98-online-tls\") pod \"101e4323-4efa-4b07-9684-91709f613e98\" (UID: \"101e4323-4efa-4b07-9684-91709f613e98\") " Sep 09 15:03:17.421976 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:03:17.421916 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/101e4323-4efa-4b07-9684-91709f613e98-feast-data\") pod \"101e4323-4efa-4b07-9684-91709f613e98\" (UID: \"101e4323-4efa-4b07-9684-91709f613e98\") " Sep 09 15:03:17.421976 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:03:17.421955 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/101e4323-4efa-4b07-9684-91709f613e98-offline-tls\") pod \"101e4323-4efa-4b07-9684-91709f613e98\" (UID: \"101e4323-4efa-4b07-9684-91709f613e98\") " Sep 09 15:03:17.422276 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:03:17.422098 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/101e4323-4efa-4b07-9684-91709f613e98-registry-tls\") pod \"101e4323-4efa-4b07-9684-91709f613e98\" (UID: \"101e4323-4efa-4b07-9684-91709f613e98\") " Sep 09 15:03:17.422276 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:03:17.422146 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/101e4323-4efa-4b07-9684-91709f613e98-ui-tls\") pod \"101e4323-4efa-4b07-9684-91709f613e98\" (UID: \"101e4323-4efa-4b07-9684-91709f613e98\") " Sep 09 15:03:17.422276 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:03:17.422216 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fqx8x\" (UniqueName: \"kubernetes.io/projected/101e4323-4efa-4b07-9684-91709f613e98-kube-api-access-fqx8x\") pod \"101e4323-4efa-4b07-9684-91709f613e98\" (UID: \"101e4323-4efa-4b07-9684-91709f613e98\") " Sep 09 15:03:17.422515 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:03:17.422487 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/101e4323-4efa-4b07-9684-91709f613e98-feast-data" (OuterVolumeSpecName: "feast-data") pod "101e4323-4efa-4b07-9684-91709f613e98" (UID: "101e4323-4efa-4b07-9684-91709f613e98"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 15:03:17.424745 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:03:17.424714 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/101e4323-4efa-4b07-9684-91709f613e98-kube-api-access-fqx8x" (OuterVolumeSpecName: "kube-api-access-fqx8x") pod "101e4323-4efa-4b07-9684-91709f613e98" (UID: "101e4323-4efa-4b07-9684-91709f613e98"). InnerVolumeSpecName "kube-api-access-fqx8x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 15:03:17.424922 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:03:17.424890 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/101e4323-4efa-4b07-9684-91709f613e98-online-tls" (OuterVolumeSpecName: "online-tls") pod "101e4323-4efa-4b07-9684-91709f613e98" (UID: "101e4323-4efa-4b07-9684-91709f613e98"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 15:03:17.424922 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:03:17.424912 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/101e4323-4efa-4b07-9684-91709f613e98-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "101e4323-4efa-4b07-9684-91709f613e98" (UID: "101e4323-4efa-4b07-9684-91709f613e98"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 15:03:17.425072 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:03:17.424930 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/101e4323-4efa-4b07-9684-91709f613e98-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "101e4323-4efa-4b07-9684-91709f613e98" (UID: "101e4323-4efa-4b07-9684-91709f613e98"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 15:03:17.425072 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:03:17.424942 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/101e4323-4efa-4b07-9684-91709f613e98-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "101e4323-4efa-4b07-9684-91709f613e98" (UID: "101e4323-4efa-4b07-9684-91709f613e98"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 15:03:17.523711 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:03:17.523687 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqx8x\" (UniqueName: \"kubernetes.io/projected/101e4323-4efa-4b07-9684-91709f613e98-kube-api-access-fqx8x\") on node \"ip-10-0-133-66.ec2.internal\" DevicePath \"\"" Sep 09 15:03:17.523711 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:03:17.523712 2576 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/101e4323-4efa-4b07-9684-91709f613e98-online-tls\") on node \"ip-10-0-133-66.ec2.internal\" DevicePath \"\"" Sep 09 15:03:17.523836 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:03:17.523721 2576 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/101e4323-4efa-4b07-9684-91709f613e98-feast-data\") on node \"ip-10-0-133-66.ec2.internal\" DevicePath \"\"" Sep 09 15:03:17.523836 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:03:17.523730 2576 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/101e4323-4efa-4b07-9684-91709f613e98-offline-tls\") on node \"ip-10-0-133-66.ec2.internal\" DevicePath \"\"" Sep 09 15:03:17.523836 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:03:17.523737 2576 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/101e4323-4efa-4b07-9684-91709f613e98-registry-tls\") on node \"ip-10-0-133-66.ec2.internal\" DevicePath \"\"" Sep 09 15:03:17.523836 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:03:17.523745 2576 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/101e4323-4efa-4b07-9684-91709f613e98-ui-tls\") on node \"ip-10-0-133-66.ec2.internal\" DevicePath \"\"" Sep 09 15:03:17.551005 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:03:17.550960 2576 generic.go:356] "Generic (PLEG): container finished" podID="101e4323-4efa-4b07-9684-91709f613e98" containerID="6fcc61e76d3a9de2b2a13abc560657be46f061cd2223699beb92d4df59bc1eec" exitCode=137 Sep 09 15:03:17.551005 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:03:17.550998 2576 generic.go:356] "Generic (PLEG): container finished" podID="101e4323-4efa-4b07-9684-91709f613e98" containerID="89f7ceda83f30a1245f67fb796e9e87195eb0bae1a88b14bbeeeab3ff67da551" exitCode=137 Sep 09 15:03:17.551271 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:03:17.551041 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-89fjc" Sep 09 15:03:17.551271 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:03:17.551038 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-89fjc" event={"ID":"101e4323-4efa-4b07-9684-91709f613e98","Type":"ContainerDied","Data":"6fcc61e76d3a9de2b2a13abc560657be46f061cd2223699beb92d4df59bc1eec"} Sep 09 15:03:17.551271 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:03:17.551118 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-89fjc" event={"ID":"101e4323-4efa-4b07-9684-91709f613e98","Type":"ContainerDied","Data":"89f7ceda83f30a1245f67fb796e9e87195eb0bae1a88b14bbeeeab3ff67da551"} Sep 09 15:03:17.551271 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:03:17.551139 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-89fjc" event={"ID":"101e4323-4efa-4b07-9684-91709f613e98","Type":"ContainerDied","Data":"64a0413ef8c162dea18a7295abce18f2dd36a2a3049de4ea60e8689f7b4e31d7"} Sep 09 15:03:17.551271 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:03:17.551165 2576 scope.go:117] "RemoveContainer" containerID="12e4bb385645f61b3b82c1133bc427a62f831ada981936cffa3d47f0549b3d86" Sep 09 15:03:17.559868 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:03:17.559846 2576 scope.go:117] "RemoveContainer" containerID="6fcc61e76d3a9de2b2a13abc560657be46f061cd2223699beb92d4df59bc1eec" Sep 09 15:03:17.568222 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:03:17.568195 2576 scope.go:117] "RemoveContainer" containerID="65dd9b26c02b38ce76b040dff4504d15b4c48df8758f6c9262fb1bc4ae2b0416" Sep 09 15:03:17.568936 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:03:17.568913 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-89fjc"] Sep 09 15:03:17.571598 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:03:17.571571 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-89fjc"] Sep 09 15:03:17.576999 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:03:17.576980 2576 scope.go:117] "RemoveContainer" containerID="89f7ceda83f30a1245f67fb796e9e87195eb0bae1a88b14bbeeeab3ff67da551" Sep 09 15:03:17.585192 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:03:17.585170 2576 scope.go:117] "RemoveContainer" containerID="9e3495fb350aef9c102c025c81f31792537147ad964c3babae5134bc8fd2a5cf" Sep 09 15:03:17.592957 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:03:17.592937 2576 scope.go:117] "RemoveContainer" containerID="4cf9223c2c9b565271b310f3f65424b432cb2924c935decddfc598b461659958" Sep 09 15:03:17.600884 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:03:17.600859 2576 scope.go:117] "RemoveContainer" containerID="12e4bb385645f61b3b82c1133bc427a62f831ada981936cffa3d47f0549b3d86" Sep 09 15:03:17.601213 ip-10-0-133-66 kubenswrapper[2576]: E0909 15:03:17.601193 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12e4bb385645f61b3b82c1133bc427a62f831ada981936cffa3d47f0549b3d86\": container with ID starting with 12e4bb385645f61b3b82c1133bc427a62f831ada981936cffa3d47f0549b3d86 not found: ID does not exist" containerID="12e4bb385645f61b3b82c1133bc427a62f831ada981936cffa3d47f0549b3d86" Sep 09 15:03:17.601313 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:03:17.601227 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12e4bb385645f61b3b82c1133bc427a62f831ada981936cffa3d47f0549b3d86"} err="failed to get container status \"12e4bb385645f61b3b82c1133bc427a62f831ada981936cffa3d47f0549b3d86\": rpc error: code = NotFound desc = could not find container \"12e4bb385645f61b3b82c1133bc427a62f831ada981936cffa3d47f0549b3d86\": container with ID starting with 12e4bb385645f61b3b82c1133bc427a62f831ada981936cffa3d47f0549b3d86 not found: ID does not exist" Sep 09 15:03:17.601313 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:03:17.601256 2576 scope.go:117] "RemoveContainer" containerID="6fcc61e76d3a9de2b2a13abc560657be46f061cd2223699beb92d4df59bc1eec" Sep 09 15:03:17.601570 ip-10-0-133-66 kubenswrapper[2576]: E0909 15:03:17.601550 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fcc61e76d3a9de2b2a13abc560657be46f061cd2223699beb92d4df59bc1eec\": container with ID starting with 6fcc61e76d3a9de2b2a13abc560657be46f061cd2223699beb92d4df59bc1eec not found: ID does not exist" containerID="6fcc61e76d3a9de2b2a13abc560657be46f061cd2223699beb92d4df59bc1eec" Sep 09 15:03:17.601618 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:03:17.601578 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fcc61e76d3a9de2b2a13abc560657be46f061cd2223699beb92d4df59bc1eec"} err="failed to get container status \"6fcc61e76d3a9de2b2a13abc560657be46f061cd2223699beb92d4df59bc1eec\": rpc error: code = NotFound desc = could not find container \"6fcc61e76d3a9de2b2a13abc560657be46f061cd2223699beb92d4df59bc1eec\": container with ID starting with 6fcc61e76d3a9de2b2a13abc560657be46f061cd2223699beb92d4df59bc1eec not found: ID does not exist" Sep 09 15:03:17.601618 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:03:17.601597 2576 scope.go:117] "RemoveContainer" containerID="65dd9b26c02b38ce76b040dff4504d15b4c48df8758f6c9262fb1bc4ae2b0416" Sep 09 15:03:17.601853 ip-10-0-133-66 kubenswrapper[2576]: E0909 15:03:17.601837 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65dd9b26c02b38ce76b040dff4504d15b4c48df8758f6c9262fb1bc4ae2b0416\": container with ID starting with 65dd9b26c02b38ce76b040dff4504d15b4c48df8758f6c9262fb1bc4ae2b0416 not found: ID does not exist" containerID="65dd9b26c02b38ce76b040dff4504d15b4c48df8758f6c9262fb1bc4ae2b0416" Sep 09 15:03:17.601894 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:03:17.601855 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65dd9b26c02b38ce76b040dff4504d15b4c48df8758f6c9262fb1bc4ae2b0416"} err="failed to get container status \"65dd9b26c02b38ce76b040dff4504d15b4c48df8758f6c9262fb1bc4ae2b0416\": rpc error: code = NotFound desc = could not find container \"65dd9b26c02b38ce76b040dff4504d15b4c48df8758f6c9262fb1bc4ae2b0416\": container with ID starting with 65dd9b26c02b38ce76b040dff4504d15b4c48df8758f6c9262fb1bc4ae2b0416 not found: ID does not exist" Sep 09 15:03:17.601894 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:03:17.601867 2576 scope.go:117] "RemoveContainer" containerID="89f7ceda83f30a1245f67fb796e9e87195eb0bae1a88b14bbeeeab3ff67da551" Sep 09 15:03:17.602075 ip-10-0-133-66 kubenswrapper[2576]: E0909 15:03:17.602057 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89f7ceda83f30a1245f67fb796e9e87195eb0bae1a88b14bbeeeab3ff67da551\": container with ID starting with 89f7ceda83f30a1245f67fb796e9e87195eb0bae1a88b14bbeeeab3ff67da551 not found: ID does not exist" containerID="89f7ceda83f30a1245f67fb796e9e87195eb0bae1a88b14bbeeeab3ff67da551" Sep 09 15:03:17.602140 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:03:17.602093 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89f7ceda83f30a1245f67fb796e9e87195eb0bae1a88b14bbeeeab3ff67da551"} err="failed to get container status \"89f7ceda83f30a1245f67fb796e9e87195eb0bae1a88b14bbeeeab3ff67da551\": rpc error: code = NotFound desc = could not find container \"89f7ceda83f30a1245f67fb796e9e87195eb0bae1a88b14bbeeeab3ff67da551\": container with ID starting with 89f7ceda83f30a1245f67fb796e9e87195eb0bae1a88b14bbeeeab3ff67da551 not found: ID does not exist" Sep 09 15:03:17.602140 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:03:17.602115 2576 scope.go:117] "RemoveContainer" containerID="9e3495fb350aef9c102c025c81f31792537147ad964c3babae5134bc8fd2a5cf" Sep 09 15:03:17.602454 ip-10-0-133-66 kubenswrapper[2576]: E0909 15:03:17.602438 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e3495fb350aef9c102c025c81f31792537147ad964c3babae5134bc8fd2a5cf\": container with ID starting with 9e3495fb350aef9c102c025c81f31792537147ad964c3babae5134bc8fd2a5cf not found: ID does not exist" containerID="9e3495fb350aef9c102c025c81f31792537147ad964c3babae5134bc8fd2a5cf" Sep 09 15:03:17.602517 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:03:17.602458 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e3495fb350aef9c102c025c81f31792537147ad964c3babae5134bc8fd2a5cf"} err="failed to get container status \"9e3495fb350aef9c102c025c81f31792537147ad964c3babae5134bc8fd2a5cf\": rpc error: code = NotFound desc = could not find container \"9e3495fb350aef9c102c025c81f31792537147ad964c3babae5134bc8fd2a5cf\": container with ID starting with 9e3495fb350aef9c102c025c81f31792537147ad964c3babae5134bc8fd2a5cf not found: ID does not exist" Sep 09 15:03:17.602517 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:03:17.602470 2576 scope.go:117] "RemoveContainer" containerID="4cf9223c2c9b565271b310f3f65424b432cb2924c935decddfc598b461659958" Sep 09 15:03:17.602734 ip-10-0-133-66 kubenswrapper[2576]: E0909 15:03:17.602715 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4cf9223c2c9b565271b310f3f65424b432cb2924c935decddfc598b461659958\": container with ID starting with 4cf9223c2c9b565271b310f3f65424b432cb2924c935decddfc598b461659958 not found: ID does not exist" containerID="4cf9223c2c9b565271b310f3f65424b432cb2924c935decddfc598b461659958" Sep 09 15:03:17.602787 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:03:17.602738 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4cf9223c2c9b565271b310f3f65424b432cb2924c935decddfc598b461659958"} err="failed to get container status \"4cf9223c2c9b565271b310f3f65424b432cb2924c935decddfc598b461659958\": rpc error: code = NotFound desc = could not find container \"4cf9223c2c9b565271b310f3f65424b432cb2924c935decddfc598b461659958\": container with ID starting with 4cf9223c2c9b565271b310f3f65424b432cb2924c935decddfc598b461659958 not found: ID does not exist" Sep 09 15:03:17.602787 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:03:17.602754 2576 scope.go:117] "RemoveContainer" containerID="12e4bb385645f61b3b82c1133bc427a62f831ada981936cffa3d47f0549b3d86" Sep 09 15:03:17.602994 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:03:17.602976 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12e4bb385645f61b3b82c1133bc427a62f831ada981936cffa3d47f0549b3d86"} err="failed to get container status \"12e4bb385645f61b3b82c1133bc427a62f831ada981936cffa3d47f0549b3d86\": rpc error: code = NotFound desc = could not find container \"12e4bb385645f61b3b82c1133bc427a62f831ada981936cffa3d47f0549b3d86\": container with ID starting with 12e4bb385645f61b3b82c1133bc427a62f831ada981936cffa3d47f0549b3d86 not found: ID does not exist" Sep 09 15:03:17.603043 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:03:17.602995 2576 scope.go:117] "RemoveContainer" containerID="6fcc61e76d3a9de2b2a13abc560657be46f061cd2223699beb92d4df59bc1eec" Sep 09 15:03:17.603229 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:03:17.603211 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fcc61e76d3a9de2b2a13abc560657be46f061cd2223699beb92d4df59bc1eec"} err="failed to get container status \"6fcc61e76d3a9de2b2a13abc560657be46f061cd2223699beb92d4df59bc1eec\": rpc error: code = NotFound desc = could not find container \"6fcc61e76d3a9de2b2a13abc560657be46f061cd2223699beb92d4df59bc1eec\": container with ID starting with 6fcc61e76d3a9de2b2a13abc560657be46f061cd2223699beb92d4df59bc1eec not found: ID does not exist" Sep 09 15:03:17.603318 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:03:17.603242 2576 scope.go:117] "RemoveContainer" containerID="65dd9b26c02b38ce76b040dff4504d15b4c48df8758f6c9262fb1bc4ae2b0416" Sep 09 15:03:17.603486 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:03:17.603467 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65dd9b26c02b38ce76b040dff4504d15b4c48df8758f6c9262fb1bc4ae2b0416"} err="failed to get container status \"65dd9b26c02b38ce76b040dff4504d15b4c48df8758f6c9262fb1bc4ae2b0416\": rpc error: code = NotFound desc = could not find container \"65dd9b26c02b38ce76b040dff4504d15b4c48df8758f6c9262fb1bc4ae2b0416\": container with ID starting with 65dd9b26c02b38ce76b040dff4504d15b4c48df8758f6c9262fb1bc4ae2b0416 not found: ID does not exist" Sep 09 15:03:17.603552 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:03:17.603487 2576 scope.go:117] "RemoveContainer" containerID="89f7ceda83f30a1245f67fb796e9e87195eb0bae1a88b14bbeeeab3ff67da551" Sep 09 15:03:17.603682 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:03:17.603665 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89f7ceda83f30a1245f67fb796e9e87195eb0bae1a88b14bbeeeab3ff67da551"} err="failed to get container status \"89f7ceda83f30a1245f67fb796e9e87195eb0bae1a88b14bbeeeab3ff67da551\": rpc error: code = NotFound desc = could not find container \"89f7ceda83f30a1245f67fb796e9e87195eb0bae1a88b14bbeeeab3ff67da551\": container with ID starting with 89f7ceda83f30a1245f67fb796e9e87195eb0bae1a88b14bbeeeab3ff67da551 not found: ID does not exist" Sep 09 15:03:17.603754 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:03:17.603683 2576 scope.go:117] "RemoveContainer" containerID="9e3495fb350aef9c102c025c81f31792537147ad964c3babae5134bc8fd2a5cf" Sep 09 15:03:17.603916 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:03:17.603898 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e3495fb350aef9c102c025c81f31792537147ad964c3babae5134bc8fd2a5cf"} err="failed to get container status \"9e3495fb350aef9c102c025c81f31792537147ad964c3babae5134bc8fd2a5cf\": rpc error: code = NotFound desc = could not find container \"9e3495fb350aef9c102c025c81f31792537147ad964c3babae5134bc8fd2a5cf\": container with ID starting with 9e3495fb350aef9c102c025c81f31792537147ad964c3babae5134bc8fd2a5cf not found: ID does not exist" Sep 09 15:03:17.603956 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:03:17.603917 2576 scope.go:117] "RemoveContainer" containerID="4cf9223c2c9b565271b310f3f65424b432cb2924c935decddfc598b461659958" Sep 09 15:03:17.604125 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:03:17.604110 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4cf9223c2c9b565271b310f3f65424b432cb2924c935decddfc598b461659958"} err="failed to get container status \"4cf9223c2c9b565271b310f3f65424b432cb2924c935decddfc598b461659958\": rpc error: code = NotFound desc = could not find container \"4cf9223c2c9b565271b310f3f65424b432cb2924c935decddfc598b461659958\": container with ID starting with 4cf9223c2c9b565271b310f3f65424b432cb2924c935decddfc598b461659958 not found: ID does not exist" Sep 09 15:03:19.514587 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:03:19.514550 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="101e4323-4efa-4b07-9684-91709f613e98" path="/var/lib/kubelet/pods/101e4323-4efa-4b07-9684-91709f613e98/volumes" Sep 09 15:04:16.372389 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:04:16.372342 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7cdwp"] Sep 09 15:04:16.372954 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:04:16.372794 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="101e4323-4efa-4b07-9684-91709f613e98" containerName="registry" Sep 09 15:04:16.372954 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:04:16.372813 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="101e4323-4efa-4b07-9684-91709f613e98" containerName="registry" Sep 09 15:04:16.372954 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:04:16.372829 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="101e4323-4efa-4b07-9684-91709f613e98" containerName="online" Sep 09 15:04:16.372954 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:04:16.372837 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="101e4323-4efa-4b07-9684-91709f613e98" containerName="online" Sep 09 15:04:16.372954 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:04:16.372851 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="101e4323-4efa-4b07-9684-91709f613e98" containerName="offline" Sep 09 15:04:16.372954 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:04:16.372859 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="101e4323-4efa-4b07-9684-91709f613e98" containerName="offline" Sep 09 15:04:16.372954 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:04:16.372870 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="101e4323-4efa-4b07-9684-91709f613e98" containerName="ui" Sep 09 15:04:16.372954 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:04:16.372877 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="101e4323-4efa-4b07-9684-91709f613e98" containerName="ui" Sep 09 15:04:16.372954 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:04:16.372898 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="101e4323-4efa-4b07-9684-91709f613e98" containerName="feast-apply" Sep 09 15:04:16.372954 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:04:16.372906 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="101e4323-4efa-4b07-9684-91709f613e98" containerName="feast-apply" Sep 09 15:04:16.372954 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:04:16.372919 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="101e4323-4efa-4b07-9684-91709f613e98" containerName="feast-init" Sep 09 15:04:16.372954 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:04:16.372927 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="101e4323-4efa-4b07-9684-91709f613e98" containerName="feast-init" Sep 09 15:04:16.373649 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:04:16.373020 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="101e4323-4efa-4b07-9684-91709f613e98" containerName="registry" Sep 09 15:04:16.373649 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:04:16.373034 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="101e4323-4efa-4b07-9684-91709f613e98" containerName="offline" Sep 09 15:04:16.373649 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:04:16.373044 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="101e4323-4efa-4b07-9684-91709f613e98" containerName="ui" Sep 09 15:04:16.373649 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:04:16.373053 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="101e4323-4efa-4b07-9684-91709f613e98" containerName="online" Sep 09 15:04:16.376149 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:04:16.376121 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7cdwp" Sep 09 15:04:16.378837 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:04:16.378806 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-online-tls\"" Sep 09 15:04:16.378837 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:04:16.378824 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-client-ca\"" Sep 09 15:04:16.379059 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:04:16.378848 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-offline-tls\"" Sep 09 15:04:16.379059 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:04:16.378806 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-ui-tls\"" Sep 09 15:04:16.379407 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:04:16.379382 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"kube-root-ca.crt\"" Sep 09 15:04:16.379571 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:04:16.379428 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-dockercfg-plskc\"" Sep 09 15:04:16.379571 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:04:16.379527 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"openshift-service-ca.crt\"" Sep 09 15:04:16.397142 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:04:16.397110 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7cdwp"] Sep 09 15:04:16.534213 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:04:16.534147 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/64c697b1-4214-4bb1-b578-cff4bc1deba8-registry-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-7cdwp\" (UID: \"64c697b1-4214-4bb1-b578-cff4bc1deba8\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7cdwp" Sep 09 15:04:16.534213 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:04:16.534205 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9q2c\" (UniqueName: \"kubernetes.io/projected/64c697b1-4214-4bb1-b578-cff4bc1deba8-kube-api-access-p9q2c\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-7cdwp\" (UID: \"64c697b1-4214-4bb1-b578-cff4bc1deba8\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7cdwp" Sep 09 15:04:16.534477 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:04:16.534332 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/64c697b1-4214-4bb1-b578-cff4bc1deba8-online-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-7cdwp\" (UID: \"64c697b1-4214-4bb1-b578-cff4bc1deba8\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7cdwp" Sep 09 15:04:16.534477 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:04:16.534387 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/64c697b1-4214-4bb1-b578-cff4bc1deba8-offline-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-7cdwp\" (UID: \"64c697b1-4214-4bb1-b578-cff4bc1deba8\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7cdwp" Sep 09 15:04:16.534477 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:04:16.534464 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/64c697b1-4214-4bb1-b578-cff4bc1deba8-ui-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-7cdwp\" (UID: \"64c697b1-4214-4bb1-b578-cff4bc1deba8\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7cdwp" Sep 09 15:04:16.534584 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:04:16.534495 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/64c697b1-4214-4bb1-b578-cff4bc1deba8-feast-data\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-7cdwp\" (UID: \"64c697b1-4214-4bb1-b578-cff4bc1deba8\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7cdwp" Sep 09 15:04:16.635380 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:04:16.635262 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/64c697b1-4214-4bb1-b578-cff4bc1deba8-offline-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-7cdwp\" (UID: \"64c697b1-4214-4bb1-b578-cff4bc1deba8\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7cdwp" Sep 09 15:04:16.635380 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:04:16.635364 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/64c697b1-4214-4bb1-b578-cff4bc1deba8-ui-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-7cdwp\" (UID: \"64c697b1-4214-4bb1-b578-cff4bc1deba8\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7cdwp" Sep 09 15:04:16.635641 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:04:16.635394 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/64c697b1-4214-4bb1-b578-cff4bc1deba8-feast-data\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-7cdwp\" (UID: \"64c697b1-4214-4bb1-b578-cff4bc1deba8\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7cdwp" Sep 09 15:04:16.635641 ip-10-0-133-66 kubenswrapper[2576]: E0909 15:04:16.635411 2576 secret.go:189] Couldn't get secret test-ns-remote-registry/feast-simple-feast-remote-setup-offline-tls: secret "feast-simple-feast-remote-setup-offline-tls" not found Sep 09 15:04:16.635641 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:04:16.635429 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/64c697b1-4214-4bb1-b578-cff4bc1deba8-registry-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-7cdwp\" (UID: \"64c697b1-4214-4bb1-b578-cff4bc1deba8\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7cdwp" Sep 09 15:04:16.635641 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:04:16.635459 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9q2c\" (UniqueName: \"kubernetes.io/projected/64c697b1-4214-4bb1-b578-cff4bc1deba8-kube-api-access-p9q2c\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-7cdwp\" (UID: \"64c697b1-4214-4bb1-b578-cff4bc1deba8\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7cdwp" Sep 09 15:04:16.635641 ip-10-0-133-66 kubenswrapper[2576]: E0909 15:04:16.635493 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64c697b1-4214-4bb1-b578-cff4bc1deba8-offline-tls podName:64c697b1-4214-4bb1-b578-cff4bc1deba8 nodeName:}" failed. No retries permitted until 2026-09-09 15:04:17.135473731 +0000 UTC m=+1872.146389963 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/64c697b1-4214-4bb1-b578-cff4bc1deba8-offline-tls") pod "feast-simple-feast-remote-setup-c5d49d8c6-7cdwp" (UID: "64c697b1-4214-4bb1-b578-cff4bc1deba8") : secret "feast-simple-feast-remote-setup-offline-tls" not found Sep 09 15:04:16.635641 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:04:16.635566 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/64c697b1-4214-4bb1-b578-cff4bc1deba8-online-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-7cdwp\" (UID: \"64c697b1-4214-4bb1-b578-cff4bc1deba8\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7cdwp" Sep 09 15:04:16.636008 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:04:16.635982 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/64c697b1-4214-4bb1-b578-cff4bc1deba8-feast-data\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-7cdwp\" (UID: \"64c697b1-4214-4bb1-b578-cff4bc1deba8\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7cdwp" Sep 09 15:04:16.636436 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:04:16.636411 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/64c697b1-4214-4bb1-b578-cff4bc1deba8-registry-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-7cdwp\" (UID: \"64c697b1-4214-4bb1-b578-cff4bc1deba8\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7cdwp" Sep 09 15:04:16.638397 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:04:16.638373 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/64c697b1-4214-4bb1-b578-cff4bc1deba8-online-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-7cdwp\" (UID: \"64c697b1-4214-4bb1-b578-cff4bc1deba8\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7cdwp" Sep 09 15:04:16.638522 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:04:16.638421 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/64c697b1-4214-4bb1-b578-cff4bc1deba8-ui-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-7cdwp\" (UID: \"64c697b1-4214-4bb1-b578-cff4bc1deba8\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7cdwp" Sep 09 15:04:16.646760 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:04:16.646727 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9q2c\" (UniqueName: \"kubernetes.io/projected/64c697b1-4214-4bb1-b578-cff4bc1deba8-kube-api-access-p9q2c\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-7cdwp\" (UID: \"64c697b1-4214-4bb1-b578-cff4bc1deba8\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7cdwp" Sep 09 15:04:17.140693 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:04:17.140654 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/64c697b1-4214-4bb1-b578-cff4bc1deba8-offline-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-7cdwp\" (UID: \"64c697b1-4214-4bb1-b578-cff4bc1deba8\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7cdwp" Sep 09 15:04:17.143428 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:04:17.143396 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/64c697b1-4214-4bb1-b578-cff4bc1deba8-offline-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-7cdwp\" (UID: \"64c697b1-4214-4bb1-b578-cff4bc1deba8\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7cdwp" Sep 09 15:04:17.298103 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:04:17.298066 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7cdwp" Sep 09 15:04:17.579752 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:04:17.579721 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7cdwp"] Sep 09 15:04:17.581519 ip-10-0-133-66 kubenswrapper[2576]: W0909 15:04:17.581490 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod64c697b1_4214_4bb1_b578_cff4bc1deba8.slice/crio-7b10e28a981e22686c407036f4d815a3455f11835e81f97b63c0a0e2b258dbd5 WatchSource:0}: Error finding container 7b10e28a981e22686c407036f4d815a3455f11835e81f97b63c0a0e2b258dbd5: Status 404 returned error can't find the container with id 7b10e28a981e22686c407036f4d815a3455f11835e81f97b63c0a0e2b258dbd5 Sep 09 15:04:17.818620 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:04:17.818585 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7cdwp" event={"ID":"64c697b1-4214-4bb1-b578-cff4bc1deba8","Type":"ContainerStarted","Data":"c2265fc41875bef81ecb293ccc3244435bda2dc8aa0a01787cce7f7d950a24ea"} Sep 09 15:04:17.818620 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:04:17.818627 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7cdwp" event={"ID":"64c697b1-4214-4bb1-b578-cff4bc1deba8","Type":"ContainerStarted","Data":"7b10e28a981e22686c407036f4d815a3455f11835e81f97b63c0a0e2b258dbd5"} Sep 09 15:04:23.845837 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:04:23.845798 2576 generic.go:356] "Generic (PLEG): container finished" podID="64c697b1-4214-4bb1-b578-cff4bc1deba8" containerID="c2265fc41875bef81ecb293ccc3244435bda2dc8aa0a01787cce7f7d950a24ea" exitCode=0 Sep 09 15:04:23.846419 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:04:23.845875 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7cdwp" event={"ID":"64c697b1-4214-4bb1-b578-cff4bc1deba8","Type":"ContainerDied","Data":"c2265fc41875bef81ecb293ccc3244435bda2dc8aa0a01787cce7f7d950a24ea"} Sep 09 15:04:24.852117 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:04:24.852074 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7cdwp" event={"ID":"64c697b1-4214-4bb1-b578-cff4bc1deba8","Type":"ContainerStarted","Data":"13643634ffad6fd3b977b8277757e0f8e678fe72fb54872ffdf0d1b1d5d55ef5"} Sep 09 15:04:44.940230 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:04:44.940197 2576 generic.go:356] "Generic (PLEG): container finished" podID="64c697b1-4214-4bb1-b578-cff4bc1deba8" containerID="13643634ffad6fd3b977b8277757e0f8e678fe72fb54872ffdf0d1b1d5d55ef5" exitCode=0 Sep 09 15:04:44.942632 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:04:44.940275 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7cdwp" event={"ID":"64c697b1-4214-4bb1-b578-cff4bc1deba8","Type":"ContainerDied","Data":"13643634ffad6fd3b977b8277757e0f8e678fe72fb54872ffdf0d1b1d5d55ef5"} Sep 09 15:04:45.948364 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:04:45.948292 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7cdwp" event={"ID":"64c697b1-4214-4bb1-b578-cff4bc1deba8","Type":"ContainerStarted","Data":"4baa22ad8b163f2e561ca352188d441ee078ea1ae7513e572e08e55e9392e3a7"} Sep 09 15:04:45.948863 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:04:45.948385 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7cdwp" event={"ID":"64c697b1-4214-4bb1-b578-cff4bc1deba8","Type":"ContainerStarted","Data":"70754f622dcf8e4fe11db9bbbfd8ea97bc05e167cf9e611aeb4d786289981458"} Sep 09 15:04:45.948863 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:04:45.948412 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7cdwp" event={"ID":"64c697b1-4214-4bb1-b578-cff4bc1deba8","Type":"ContainerStarted","Data":"1155c564304cb36d1678119324a052e3f6e9fbd59b9380032f25a6be3a1e3dfd"} Sep 09 15:04:45.976413 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:04:45.976351 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7cdwp" podStartSLOduration=29.976331204 podStartE2EDuration="29.976331204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:04:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:04:45.972990334 +0000 UTC m=+1900.983906586" watchObservedRunningTime="2026-09-09 15:04:45.976331204 +0000 UTC m=+1900.987247452" Sep 09 15:04:47.298239 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:04:47.298199 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7cdwp" Sep 09 15:04:47.298749 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:04:47.298381 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7cdwp" Sep 09 15:04:47.298749 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:04:47.298427 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7cdwp" Sep 09 15:04:47.298749 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:04:47.298447 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7cdwp" Sep 09 15:04:47.298749 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:04:47.298465 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7cdwp" Sep 09 15:04:47.298749 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:04:47.298480 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7cdwp" Sep 09 15:04:47.299949 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:04:47.299910 2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7cdwp" podUID="64c697b1-4214-4bb1-b578-cff4bc1deba8" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.30:8816: connect: connection refused" Sep 09 15:04:47.300087 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:04:47.299968 2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7cdwp" podUID="64c697b1-4214-4bb1-b578-cff4bc1deba8" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.30:8443: connect: connection refused" Sep 09 15:04:47.300087 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:04:47.299968 2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7cdwp" podUID="64c697b1-4214-4bb1-b578-cff4bc1deba8" containerName="online" probeResult="failure" output="Get \"https://10.132.0.30:6567/health\": dial tcp 10.132.0.30:6567: connect: connection refused" Sep 09 15:04:50.299189 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:04:50.299144 2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7cdwp" podUID="64c697b1-4214-4bb1-b578-cff4bc1deba8" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.30:8443: connect: connection refused" Sep 09 15:04:50.299806 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:04:50.299210 2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7cdwp" podUID="64c697b1-4214-4bb1-b578-cff4bc1deba8" containerName="online" probeResult="failure" output="Get \"https://10.132.0.30:6567/health\": dial tcp 10.132.0.30:6567: connect: connection refused" Sep 09 15:04:50.299806 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:04:50.299144 2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7cdwp" podUID="64c697b1-4214-4bb1-b578-cff4bc1deba8" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.30:8816: connect: connection refused" Sep 09 15:04:53.298785 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:04:53.298724 2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7cdwp" podUID="64c697b1-4214-4bb1-b578-cff4bc1deba8" containerName="online" probeResult="failure" output="Get \"https://10.132.0.30:6567/health\": dial tcp 10.132.0.30:6567: connect: connection refused" Sep 09 15:04:53.299556 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:04:53.299533 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7cdwp" Sep 09 15:04:53.299671 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:04:53.299661 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7cdwp" Sep 09 15:04:53.300369 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:04:53.300347 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7cdwp" Sep 09 15:04:53.300465 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:04:53.300416 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7cdwp" Sep 09 15:04:56.987518 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:04:56.987426 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7cdwp" Sep 09 15:04:57.651819 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:04:57.651787 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7cdwp" Sep 09 15:04:58.302492 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:04:58.302444 2576 prober.go:120] "Probe failed" probeType="Liveness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7cdwp" podUID="64c697b1-4214-4bb1-b578-cff4bc1deba8" containerName="online" probeResult="failure" output="Get \"https://10.132.0.30:6567/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Sep 09 15:05:18.302451 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:05:18.302406 2576 prober.go:120] "Probe failed" probeType="Liveness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7cdwp" podUID="64c697b1-4214-4bb1-b578-cff4bc1deba8" containerName="online" probeResult="failure" output="Get \"https://10.132.0.30:6567/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 09 15:05:38.303495 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:05:38.303442 2576 prober.go:120] "Probe failed" probeType="Liveness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7cdwp" podUID="64c697b1-4214-4bb1-b578-cff4bc1deba8" containerName="online" probeResult="failure" output="Get \"https://10.132.0.30:6567/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 09 15:05:58.595035 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:05:58.594998 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7cdwp"] Sep 09 15:05:58.595581 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:05:58.595270 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7cdwp" podUID="64c697b1-4214-4bb1-b578-cff4bc1deba8" containerName="online" containerID="cri-o://1155c564304cb36d1678119324a052e3f6e9fbd59b9380032f25a6be3a1e3dfd" gracePeriod=30 Sep 09 15:05:58.595581 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:05:58.595333 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7cdwp" podUID="64c697b1-4214-4bb1-b578-cff4bc1deba8" containerName="ui" containerID="cri-o://4baa22ad8b163f2e561ca352188d441ee078ea1ae7513e572e08e55e9392e3a7" gracePeriod=30 Sep 09 15:05:58.595581 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:05:58.595410 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7cdwp" podUID="64c697b1-4214-4bb1-b578-cff4bc1deba8" containerName="offline" containerID="cri-o://70754f622dcf8e4fe11db9bbbfd8ea97bc05e167cf9e611aeb4d786289981458" gracePeriod=30 Sep 09 15:05:58.740496 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:05:58.740457 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-l2v8b"] Sep 09 15:05:58.740783 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:05:58.740754 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l2v8b" podUID="27a978af-ee9e-4dfc-bca6-d877ddd92cc1" containerName="registry" containerID="cri-o://23b3de2eec5fdcf3642621cad03a554f2dd1ced52e873ec250acc6b19c82323f" gracePeriod=30 Sep 09 15:05:58.740876 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:05:58.740823 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l2v8b" podUID="27a978af-ee9e-4dfc-bca6-d877ddd92cc1" containerName="offline" containerID="cri-o://1f99ad0b9f0bab53d69f244c17a9367906ed6c7a055dade5ea8de0bef888b257" gracePeriod=30 Sep 09 15:05:58.740949 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:05:58.740866 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l2v8b" podUID="27a978af-ee9e-4dfc-bca6-d877ddd92cc1" containerName="ui" containerID="cri-o://83be447974cb836d6d7d3d3dd8633e3d9ed45c30d5a7e311011b07e3019aed4d" gracePeriod=30 Sep 09 15:05:58.740949 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:05:58.740931 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l2v8b" podUID="27a978af-ee9e-4dfc-bca6-d877ddd92cc1" containerName="online" containerID="cri-o://78e3d67c6fb1dfe2db1c3c510a660aad8efbb36694dbce09a044c1c8fd46e0ab" gracePeriod=30 Sep 09 15:06:00.283085 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:06:00.283047 2576 generic.go:356] "Generic (PLEG): container finished" podID="64c697b1-4214-4bb1-b578-cff4bc1deba8" containerID="4baa22ad8b163f2e561ca352188d441ee078ea1ae7513e572e08e55e9392e3a7" exitCode=0 Sep 09 15:06:00.283085 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:06:00.283074 2576 generic.go:356] "Generic (PLEG): container finished" podID="64c697b1-4214-4bb1-b578-cff4bc1deba8" containerID="1155c564304cb36d1678119324a052e3f6e9fbd59b9380032f25a6be3a1e3dfd" exitCode=0 Sep 09 15:06:00.283559 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:06:00.283121 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7cdwp" event={"ID":"64c697b1-4214-4bb1-b578-cff4bc1deba8","Type":"ContainerDied","Data":"4baa22ad8b163f2e561ca352188d441ee078ea1ae7513e572e08e55e9392e3a7"} Sep 09 15:06:00.283559 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:06:00.283168 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7cdwp" event={"ID":"64c697b1-4214-4bb1-b578-cff4bc1deba8","Type":"ContainerDied","Data":"1155c564304cb36d1678119324a052e3f6e9fbd59b9380032f25a6be3a1e3dfd"} Sep 09 15:06:00.286058 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:06:00.286033 2576 generic.go:356] "Generic (PLEG): container finished" podID="27a978af-ee9e-4dfc-bca6-d877ddd92cc1" containerID="83be447974cb836d6d7d3d3dd8633e3d9ed45c30d5a7e311011b07e3019aed4d" exitCode=0 Sep 09 15:06:00.286058 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:06:00.286052 2576 generic.go:356] "Generic (PLEG): container finished" podID="27a978af-ee9e-4dfc-bca6-d877ddd92cc1" containerID="78e3d67c6fb1dfe2db1c3c510a660aad8efbb36694dbce09a044c1c8fd46e0ab" exitCode=0 Sep 09 15:06:00.286245 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:06:00.286105 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l2v8b" event={"ID":"27a978af-ee9e-4dfc-bca6-d877ddd92cc1","Type":"ContainerDied","Data":"83be447974cb836d6d7d3d3dd8633e3d9ed45c30d5a7e311011b07e3019aed4d"} Sep 09 15:06:00.286245 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:06:00.286139 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l2v8b" event={"ID":"27a978af-ee9e-4dfc-bca6-d877ddd92cc1","Type":"ContainerDied","Data":"78e3d67c6fb1dfe2db1c3c510a660aad8efbb36694dbce09a044c1c8fd46e0ab"} Sep 09 15:06:02.049274 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:06:02.049219 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l2v8b" podUID="27a978af-ee9e-4dfc-bca6-d877ddd92cc1" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.29:8443: connect: connection refused" Sep 09 15:06:02.529387 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:06:02.529343 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l2v8b" podUID="27a978af-ee9e-4dfc-bca6-d877ddd92cc1" containerName="online" probeResult="failure" output="Get \"https://10.132.0.29:6567/health\": dial tcp 10.132.0.29:6567: connect: connection refused" Sep 09 15:06:03.300508 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:06:03.300463 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7cdwp" podUID="64c697b1-4214-4bb1-b578-cff4bc1deba8" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.30:8443: connect: connection refused" Sep 09 15:06:06.988577 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:06:06.988530 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7cdwp" podUID="64c697b1-4214-4bb1-b578-cff4bc1deba8" containerName="online" probeResult="failure" output="Get \"https://10.132.0.30:6567/health\": dial tcp 10.132.0.30:6567: connect: connection refused" Sep 09 15:06:12.049818 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:06:12.049766 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l2v8b" podUID="27a978af-ee9e-4dfc-bca6-d877ddd92cc1" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.29:8443: connect: connection refused" Sep 09 15:06:12.529638 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:06:12.529599 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l2v8b" podUID="27a978af-ee9e-4dfc-bca6-d877ddd92cc1" containerName="online" probeResult="failure" output="Get \"https://10.132.0.29:6567/health\": dial tcp 10.132.0.29:6567: connect: connection refused" Sep 09 15:06:13.300469 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:06:13.300423 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7cdwp" podUID="64c697b1-4214-4bb1-b578-cff4bc1deba8" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.30:8443: connect: connection refused" Sep 09 15:06:16.989007 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:06:16.988952 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7cdwp" podUID="64c697b1-4214-4bb1-b578-cff4bc1deba8" containerName="online" probeResult="failure" output="Get \"https://10.132.0.30:6567/health\": dial tcp 10.132.0.30:6567: connect: connection refused" Sep 09 15:06:22.049877 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:06:22.049827 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l2v8b" podUID="27a978af-ee9e-4dfc-bca6-d877ddd92cc1" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.29:8443: connect: connection refused" Sep 09 15:06:22.050358 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:06:22.049955 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l2v8b" Sep 09 15:06:22.528813 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:06:22.528763 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l2v8b" podUID="27a978af-ee9e-4dfc-bca6-d877ddd92cc1" containerName="online" probeResult="failure" output="Get \"https://10.132.0.29:6567/health\": dial tcp 10.132.0.29:6567: connect: connection refused" Sep 09 15:06:22.529018 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:06:22.528892 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l2v8b" Sep 09 15:06:23.300799 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:06:23.300750 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7cdwp" podUID="64c697b1-4214-4bb1-b578-cff4bc1deba8" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.30:8443: connect: connection refused" Sep 09 15:06:23.301258 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:06:23.300902 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7cdwp" Sep 09 15:06:26.988573 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:06:26.988525 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7cdwp" podUID="64c697b1-4214-4bb1-b578-cff4bc1deba8" containerName="online" probeResult="failure" output="Get \"https://10.132.0.30:6567/health\": dial tcp 10.132.0.30:6567: connect: connection refused" Sep 09 15:06:26.988985 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:06:26.988652 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7cdwp" Sep 09 15:06:29.248871 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:06:29.248847 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7cdwp" Sep 09 15:06:29.274741 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:06:29.274710 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/64c697b1-4214-4bb1-b578-cff4bc1deba8-offline-tls\") pod \"64c697b1-4214-4bb1-b578-cff4bc1deba8\" (UID: \"64c697b1-4214-4bb1-b578-cff4bc1deba8\") " Sep 09 15:06:29.274903 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:06:29.274775 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/64c697b1-4214-4bb1-b578-cff4bc1deba8-registry-tls\") pod \"64c697b1-4214-4bb1-b578-cff4bc1deba8\" (UID: \"64c697b1-4214-4bb1-b578-cff4bc1deba8\") " Sep 09 15:06:29.274903 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:06:29.274827 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/64c697b1-4214-4bb1-b578-cff4bc1deba8-ui-tls\") pod \"64c697b1-4214-4bb1-b578-cff4bc1deba8\" (UID: \"64c697b1-4214-4bb1-b578-cff4bc1deba8\") " Sep 09 15:06:29.274903 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:06:29.274861 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p9q2c\" (UniqueName: \"kubernetes.io/projected/64c697b1-4214-4bb1-b578-cff4bc1deba8-kube-api-access-p9q2c\") pod \"64c697b1-4214-4bb1-b578-cff4bc1deba8\" (UID: \"64c697b1-4214-4bb1-b578-cff4bc1deba8\") " Sep 09 15:06:29.274903 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:06:29.274899 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/64c697b1-4214-4bb1-b578-cff4bc1deba8-feast-data\") pod \"64c697b1-4214-4bb1-b578-cff4bc1deba8\" (UID: \"64c697b1-4214-4bb1-b578-cff4bc1deba8\") " Sep 09 15:06:29.275142 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:06:29.274931 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/64c697b1-4214-4bb1-b578-cff4bc1deba8-online-tls\") pod \"64c697b1-4214-4bb1-b578-cff4bc1deba8\" (UID: \"64c697b1-4214-4bb1-b578-cff4bc1deba8\") " Sep 09 15:06:29.275407 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:06:29.275366 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64c697b1-4214-4bb1-b578-cff4bc1deba8-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "64c697b1-4214-4bb1-b578-cff4bc1deba8" (UID: "64c697b1-4214-4bb1-b578-cff4bc1deba8"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 15:06:29.275688 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:06:29.275645 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64c697b1-4214-4bb1-b578-cff4bc1deba8-feast-data" (OuterVolumeSpecName: "feast-data") pod "64c697b1-4214-4bb1-b578-cff4bc1deba8" (UID: "64c697b1-4214-4bb1-b578-cff4bc1deba8"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 15:06:29.277630 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:06:29.277601 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64c697b1-4214-4bb1-b578-cff4bc1deba8-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "64c697b1-4214-4bb1-b578-cff4bc1deba8" (UID: "64c697b1-4214-4bb1-b578-cff4bc1deba8"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 15:06:29.277780 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:06:29.277751 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64c697b1-4214-4bb1-b578-cff4bc1deba8-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "64c697b1-4214-4bb1-b578-cff4bc1deba8" (UID: "64c697b1-4214-4bb1-b578-cff4bc1deba8"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 15:06:29.278006 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:06:29.277974 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64c697b1-4214-4bb1-b578-cff4bc1deba8-kube-api-access-p9q2c" (OuterVolumeSpecName: "kube-api-access-p9q2c") pod "64c697b1-4214-4bb1-b578-cff4bc1deba8" (UID: "64c697b1-4214-4bb1-b578-cff4bc1deba8"). InnerVolumeSpecName "kube-api-access-p9q2c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 15:06:29.278259 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:06:29.278238 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64c697b1-4214-4bb1-b578-cff4bc1deba8-online-tls" (OuterVolumeSpecName: "online-tls") pod "64c697b1-4214-4bb1-b578-cff4bc1deba8" (UID: "64c697b1-4214-4bb1-b578-cff4bc1deba8"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 15:06:29.375671 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:06:29.375642 2576 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/64c697b1-4214-4bb1-b578-cff4bc1deba8-feast-data\") on node \"ip-10-0-133-66.ec2.internal\" DevicePath \"\"" Sep 09 15:06:29.375671 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:06:29.375675 2576 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/64c697b1-4214-4bb1-b578-cff4bc1deba8-online-tls\") on node \"ip-10-0-133-66.ec2.internal\" DevicePath \"\"" Sep 09 15:06:29.375824 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:06:29.375689 2576 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/64c697b1-4214-4bb1-b578-cff4bc1deba8-offline-tls\") on node \"ip-10-0-133-66.ec2.internal\" DevicePath \"\"" Sep 09 15:06:29.375824 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:06:29.375700 2576 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/64c697b1-4214-4bb1-b578-cff4bc1deba8-registry-tls\") on node \"ip-10-0-133-66.ec2.internal\" DevicePath \"\"" Sep 09 15:06:29.375824 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:06:29.375712 2576 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/64c697b1-4214-4bb1-b578-cff4bc1deba8-ui-tls\") on node \"ip-10-0-133-66.ec2.internal\" DevicePath \"\"" Sep 09 15:06:29.375824 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:06:29.375723 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9q2c\" (UniqueName: \"kubernetes.io/projected/64c697b1-4214-4bb1-b578-cff4bc1deba8-kube-api-access-p9q2c\") on node \"ip-10-0-133-66.ec2.internal\" DevicePath \"\"" Sep 09 15:06:29.384573 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:06:29.384547 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l2v8b" Sep 09 15:06:29.407668 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:06:29.407626 2576 generic.go:356] "Generic (PLEG): container finished" podID="64c697b1-4214-4bb1-b578-cff4bc1deba8" containerID="70754f622dcf8e4fe11db9bbbfd8ea97bc05e167cf9e611aeb4d786289981458" exitCode=137 Sep 09 15:06:29.407846 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:06:29.407704 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7cdwp" Sep 09 15:06:29.407846 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:06:29.407712 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7cdwp" event={"ID":"64c697b1-4214-4bb1-b578-cff4bc1deba8","Type":"ContainerDied","Data":"70754f622dcf8e4fe11db9bbbfd8ea97bc05e167cf9e611aeb4d786289981458"} Sep 09 15:06:29.407846 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:06:29.407767 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7cdwp" event={"ID":"64c697b1-4214-4bb1-b578-cff4bc1deba8","Type":"ContainerDied","Data":"7b10e28a981e22686c407036f4d815a3455f11835e81f97b63c0a0e2b258dbd5"} Sep 09 15:06:29.407846 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:06:29.407802 2576 scope.go:117] "RemoveContainer" containerID="4baa22ad8b163f2e561ca352188d441ee078ea1ae7513e572e08e55e9392e3a7" Sep 09 15:06:29.411724 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:06:29.411695 2576 generic.go:356] "Generic (PLEG): container finished" podID="27a978af-ee9e-4dfc-bca6-d877ddd92cc1" containerID="1f99ad0b9f0bab53d69f244c17a9367906ed6c7a055dade5ea8de0bef888b257" exitCode=137 Sep 09 15:06:29.411857 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:06:29.411730 2576 generic.go:356] "Generic (PLEG): container finished" podID="27a978af-ee9e-4dfc-bca6-d877ddd92cc1" containerID="23b3de2eec5fdcf3642621cad03a554f2dd1ced52e873ec250acc6b19c82323f" exitCode=137 Sep 09 15:06:29.411857 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:06:29.411736 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l2v8b" event={"ID":"27a978af-ee9e-4dfc-bca6-d877ddd92cc1","Type":"ContainerDied","Data":"1f99ad0b9f0bab53d69f244c17a9367906ed6c7a055dade5ea8de0bef888b257"} Sep 09 15:06:29.411857 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:06:29.411771 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l2v8b" Sep 09 15:06:29.411857 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:06:29.411784 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l2v8b" event={"ID":"27a978af-ee9e-4dfc-bca6-d877ddd92cc1","Type":"ContainerDied","Data":"23b3de2eec5fdcf3642621cad03a554f2dd1ced52e873ec250acc6b19c82323f"} Sep 09 15:06:29.411857 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:06:29.411806 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l2v8b" event={"ID":"27a978af-ee9e-4dfc-bca6-d877ddd92cc1","Type":"ContainerDied","Data":"323b3ef507c24f2cce2e14fe3de9ed3fbbb62f84635fff4668daf6379c58d402"} Sep 09 15:06:29.418202 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:06:29.418184 2576 scope.go:117] "RemoveContainer" containerID="70754f622dcf8e4fe11db9bbbfd8ea97bc05e167cf9e611aeb4d786289981458" Sep 09 15:06:29.426627 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:06:29.426606 2576 scope.go:117] "RemoveContainer" containerID="1155c564304cb36d1678119324a052e3f6e9fbd59b9380032f25a6be3a1e3dfd" Sep 09 15:06:29.430314 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:06:29.430279 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7cdwp"] Sep 09 15:06:29.434464 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:06:29.434439 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7cdwp"] Sep 09 15:06:29.435554 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:06:29.435538 2576 scope.go:117] "RemoveContainer" containerID="13643634ffad6fd3b977b8277757e0f8e678fe72fb54872ffdf0d1b1d5d55ef5" Sep 09 15:06:29.443587 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:06:29.443563 2576 scope.go:117] "RemoveContainer" containerID="c2265fc41875bef81ecb293ccc3244435bda2dc8aa0a01787cce7f7d950a24ea" Sep 09 15:06:29.451231 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:06:29.451212 2576 scope.go:117] "RemoveContainer" containerID="4baa22ad8b163f2e561ca352188d441ee078ea1ae7513e572e08e55e9392e3a7" Sep 09 15:06:29.451611 ip-10-0-133-66 kubenswrapper[2576]: E0909 15:06:29.451588 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4baa22ad8b163f2e561ca352188d441ee078ea1ae7513e572e08e55e9392e3a7\": container with ID starting with 4baa22ad8b163f2e561ca352188d441ee078ea1ae7513e572e08e55e9392e3a7 not found: ID does not exist" containerID="4baa22ad8b163f2e561ca352188d441ee078ea1ae7513e572e08e55e9392e3a7" Sep 09 15:06:29.451691 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:06:29.451622 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4baa22ad8b163f2e561ca352188d441ee078ea1ae7513e572e08e55e9392e3a7"} err="failed to get container status \"4baa22ad8b163f2e561ca352188d441ee078ea1ae7513e572e08e55e9392e3a7\": rpc error: code = NotFound desc = could not find container \"4baa22ad8b163f2e561ca352188d441ee078ea1ae7513e572e08e55e9392e3a7\": container with ID starting with 4baa22ad8b163f2e561ca352188d441ee078ea1ae7513e572e08e55e9392e3a7 not found: ID does not exist" Sep 09 15:06:29.451691 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:06:29.451640 2576 scope.go:117] "RemoveContainer" containerID="70754f622dcf8e4fe11db9bbbfd8ea97bc05e167cf9e611aeb4d786289981458" Sep 09 15:06:29.451901 ip-10-0-133-66 kubenswrapper[2576]: E0909 15:06:29.451881 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70754f622dcf8e4fe11db9bbbfd8ea97bc05e167cf9e611aeb4d786289981458\": container with ID starting with 70754f622dcf8e4fe11db9bbbfd8ea97bc05e167cf9e611aeb4d786289981458 not found: ID does not exist" containerID="70754f622dcf8e4fe11db9bbbfd8ea97bc05e167cf9e611aeb4d786289981458" Sep 09 15:06:29.451968 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:06:29.451911 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70754f622dcf8e4fe11db9bbbfd8ea97bc05e167cf9e611aeb4d786289981458"} err="failed to get container status \"70754f622dcf8e4fe11db9bbbfd8ea97bc05e167cf9e611aeb4d786289981458\": rpc error: code = NotFound desc = could not find container \"70754f622dcf8e4fe11db9bbbfd8ea97bc05e167cf9e611aeb4d786289981458\": container with ID starting with 70754f622dcf8e4fe11db9bbbfd8ea97bc05e167cf9e611aeb4d786289981458 not found: ID does not exist" Sep 09 15:06:29.451968 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:06:29.451936 2576 scope.go:117] "RemoveContainer" containerID="1155c564304cb36d1678119324a052e3f6e9fbd59b9380032f25a6be3a1e3dfd" Sep 09 15:06:29.452174 ip-10-0-133-66 kubenswrapper[2576]: E0909 15:06:29.452158 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1155c564304cb36d1678119324a052e3f6e9fbd59b9380032f25a6be3a1e3dfd\": container with ID starting with 1155c564304cb36d1678119324a052e3f6e9fbd59b9380032f25a6be3a1e3dfd not found: ID does not exist" containerID="1155c564304cb36d1678119324a052e3f6e9fbd59b9380032f25a6be3a1e3dfd" Sep 09 15:06:29.452217 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:06:29.452179 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1155c564304cb36d1678119324a052e3f6e9fbd59b9380032f25a6be3a1e3dfd"} err="failed to get container status \"1155c564304cb36d1678119324a052e3f6e9fbd59b9380032f25a6be3a1e3dfd\": rpc error: code = NotFound desc = could not find container \"1155c564304cb36d1678119324a052e3f6e9fbd59b9380032f25a6be3a1e3dfd\": container with ID starting with 1155c564304cb36d1678119324a052e3f6e9fbd59b9380032f25a6be3a1e3dfd not found: ID does not exist" Sep 09 15:06:29.452217 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:06:29.452191 2576 scope.go:117] "RemoveContainer" containerID="13643634ffad6fd3b977b8277757e0f8e678fe72fb54872ffdf0d1b1d5d55ef5" Sep 09 15:06:29.452426 ip-10-0-133-66 kubenswrapper[2576]: E0909 15:06:29.452410 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13643634ffad6fd3b977b8277757e0f8e678fe72fb54872ffdf0d1b1d5d55ef5\": container with ID starting with 13643634ffad6fd3b977b8277757e0f8e678fe72fb54872ffdf0d1b1d5d55ef5 not found: ID does not exist" containerID="13643634ffad6fd3b977b8277757e0f8e678fe72fb54872ffdf0d1b1d5d55ef5" Sep 09 15:06:29.452494 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:06:29.452432 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13643634ffad6fd3b977b8277757e0f8e678fe72fb54872ffdf0d1b1d5d55ef5"} err="failed to get container status \"13643634ffad6fd3b977b8277757e0f8e678fe72fb54872ffdf0d1b1d5d55ef5\": rpc error: code = NotFound desc = could not find container \"13643634ffad6fd3b977b8277757e0f8e678fe72fb54872ffdf0d1b1d5d55ef5\": container with ID starting with 13643634ffad6fd3b977b8277757e0f8e678fe72fb54872ffdf0d1b1d5d55ef5 not found: ID does not exist" Sep 09 15:06:29.452494 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:06:29.452451 2576 scope.go:117] "RemoveContainer" containerID="c2265fc41875bef81ecb293ccc3244435bda2dc8aa0a01787cce7f7d950a24ea" Sep 09 15:06:29.452669 ip-10-0-133-66 kubenswrapper[2576]: E0909 15:06:29.452655 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2265fc41875bef81ecb293ccc3244435bda2dc8aa0a01787cce7f7d950a24ea\": container with ID starting with c2265fc41875bef81ecb293ccc3244435bda2dc8aa0a01787cce7f7d950a24ea not found: ID does not exist" containerID="c2265fc41875bef81ecb293ccc3244435bda2dc8aa0a01787cce7f7d950a24ea" Sep 09 15:06:29.452707 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:06:29.452672 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2265fc41875bef81ecb293ccc3244435bda2dc8aa0a01787cce7f7d950a24ea"} err="failed to get container status \"c2265fc41875bef81ecb293ccc3244435bda2dc8aa0a01787cce7f7d950a24ea\": rpc error: code = NotFound desc = could not find container \"c2265fc41875bef81ecb293ccc3244435bda2dc8aa0a01787cce7f7d950a24ea\": container with ID starting with c2265fc41875bef81ecb293ccc3244435bda2dc8aa0a01787cce7f7d950a24ea not found: ID does not exist" Sep 09 15:06:29.452707 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:06:29.452684 2576 scope.go:117] "RemoveContainer" containerID="83be447974cb836d6d7d3d3dd8633e3d9ed45c30d5a7e311011b07e3019aed4d" Sep 09 15:06:29.460163 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:06:29.460145 2576 scope.go:117] "RemoveContainer" containerID="1f99ad0b9f0bab53d69f244c17a9367906ed6c7a055dade5ea8de0bef888b257" Sep 09 15:06:29.468015 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:06:29.467995 2576 scope.go:117] "RemoveContainer" containerID="78e3d67c6fb1dfe2db1c3c510a660aad8efbb36694dbce09a044c1c8fd46e0ab" Sep 09 15:06:29.475696 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:06:29.475678 2576 scope.go:117] "RemoveContainer" containerID="23b3de2eec5fdcf3642621cad03a554f2dd1ced52e873ec250acc6b19c82323f" Sep 09 15:06:29.475973 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:06:29.475955 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/27a978af-ee9e-4dfc-bca6-d877ddd92cc1-registry-tls\") pod \"27a978af-ee9e-4dfc-bca6-d877ddd92cc1\" (UID: \"27a978af-ee9e-4dfc-bca6-d877ddd92cc1\") " Sep 09 15:06:29.476044 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:06:29.475996 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/27a978af-ee9e-4dfc-bca6-d877ddd92cc1-ui-tls\") pod \"27a978af-ee9e-4dfc-bca6-d877ddd92cc1\" (UID: \"27a978af-ee9e-4dfc-bca6-d877ddd92cc1\") " Sep 09 15:06:29.476101 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:06:29.476045 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nkkcg\" (UniqueName: \"kubernetes.io/projected/27a978af-ee9e-4dfc-bca6-d877ddd92cc1-kube-api-access-nkkcg\") pod \"27a978af-ee9e-4dfc-bca6-d877ddd92cc1\" (UID: \"27a978af-ee9e-4dfc-bca6-d877ddd92cc1\") " Sep 09 15:06:29.476195 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:06:29.476175 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/27a978af-ee9e-4dfc-bca6-d877ddd92cc1-offline-tls\") pod \"27a978af-ee9e-4dfc-bca6-d877ddd92cc1\" (UID: \"27a978af-ee9e-4dfc-bca6-d877ddd92cc1\") " Sep 09 15:06:29.476271 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:06:29.476233 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/27a978af-ee9e-4dfc-bca6-d877ddd92cc1-feast-data\") pod \"27a978af-ee9e-4dfc-bca6-d877ddd92cc1\" (UID: \"27a978af-ee9e-4dfc-bca6-d877ddd92cc1\") " Sep 09 15:06:29.476271 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:06:29.476260 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/27a978af-ee9e-4dfc-bca6-d877ddd92cc1-online-tls\") pod \"27a978af-ee9e-4dfc-bca6-d877ddd92cc1\" (UID: \"27a978af-ee9e-4dfc-bca6-d877ddd92cc1\") " Sep 09 15:06:29.477434 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:06:29.477217 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27a978af-ee9e-4dfc-bca6-d877ddd92cc1-feast-data" (OuterVolumeSpecName: "feast-data") pod "27a978af-ee9e-4dfc-bca6-d877ddd92cc1" (UID: "27a978af-ee9e-4dfc-bca6-d877ddd92cc1"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 15:06:29.478456 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:06:29.478430 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27a978af-ee9e-4dfc-bca6-d877ddd92cc1-kube-api-access-nkkcg" (OuterVolumeSpecName: "kube-api-access-nkkcg") pod "27a978af-ee9e-4dfc-bca6-d877ddd92cc1" (UID: "27a978af-ee9e-4dfc-bca6-d877ddd92cc1"). InnerVolumeSpecName "kube-api-access-nkkcg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 15:06:29.478694 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:06:29.478676 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27a978af-ee9e-4dfc-bca6-d877ddd92cc1-online-tls" (OuterVolumeSpecName: "online-tls") pod "27a978af-ee9e-4dfc-bca6-d877ddd92cc1" (UID: "27a978af-ee9e-4dfc-bca6-d877ddd92cc1"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 15:06:29.478839 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:06:29.478819 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27a978af-ee9e-4dfc-bca6-d877ddd92cc1-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "27a978af-ee9e-4dfc-bca6-d877ddd92cc1" (UID: "27a978af-ee9e-4dfc-bca6-d877ddd92cc1"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 15:06:29.478839 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:06:29.478837 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27a978af-ee9e-4dfc-bca6-d877ddd92cc1-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "27a978af-ee9e-4dfc-bca6-d877ddd92cc1" (UID: "27a978af-ee9e-4dfc-bca6-d877ddd92cc1"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 15:06:29.479089 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:06:29.479075 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27a978af-ee9e-4dfc-bca6-d877ddd92cc1-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "27a978af-ee9e-4dfc-bca6-d877ddd92cc1" (UID: "27a978af-ee9e-4dfc-bca6-d877ddd92cc1"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 15:06:29.484483 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:06:29.484466 2576 scope.go:117] "RemoveContainer" containerID="c2f1ab7040e10614b73087e0db10de6bc717406e0161e73df2fc528e4cab4109" Sep 09 15:06:29.491766 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:06:29.491745 2576 scope.go:117] "RemoveContainer" containerID="c765e510ac62a829653001e4e64a09a44f8f772a3c905be79823c9c77dbaac40" Sep 09 15:06:29.499105 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:06:29.499086 2576 scope.go:117] "RemoveContainer" containerID="83be447974cb836d6d7d3d3dd8633e3d9ed45c30d5a7e311011b07e3019aed4d" Sep 09 15:06:29.499405 ip-10-0-133-66 kubenswrapper[2576]: E0909 15:06:29.499382 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83be447974cb836d6d7d3d3dd8633e3d9ed45c30d5a7e311011b07e3019aed4d\": container with ID starting with 83be447974cb836d6d7d3d3dd8633e3d9ed45c30d5a7e311011b07e3019aed4d not found: ID does not exist" containerID="83be447974cb836d6d7d3d3dd8633e3d9ed45c30d5a7e311011b07e3019aed4d" Sep 09 15:06:29.499457 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:06:29.499419 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83be447974cb836d6d7d3d3dd8633e3d9ed45c30d5a7e311011b07e3019aed4d"} err="failed to get container status \"83be447974cb836d6d7d3d3dd8633e3d9ed45c30d5a7e311011b07e3019aed4d\": rpc error: code = NotFound desc = could not find container \"83be447974cb836d6d7d3d3dd8633e3d9ed45c30d5a7e311011b07e3019aed4d\": container with ID starting with 83be447974cb836d6d7d3d3dd8633e3d9ed45c30d5a7e311011b07e3019aed4d not found: ID does not exist" Sep 09 15:06:29.499457 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:06:29.499442 2576 scope.go:117] "RemoveContainer" containerID="1f99ad0b9f0bab53d69f244c17a9367906ed6c7a055dade5ea8de0bef888b257" Sep 09 15:06:29.499674 ip-10-0-133-66 kubenswrapper[2576]: E0909 15:06:29.499657 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f99ad0b9f0bab53d69f244c17a9367906ed6c7a055dade5ea8de0bef888b257\": container with ID starting with 1f99ad0b9f0bab53d69f244c17a9367906ed6c7a055dade5ea8de0bef888b257 not found: ID does not exist" containerID="1f99ad0b9f0bab53d69f244c17a9367906ed6c7a055dade5ea8de0bef888b257" Sep 09 15:06:29.499723 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:06:29.499680 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f99ad0b9f0bab53d69f244c17a9367906ed6c7a055dade5ea8de0bef888b257"} err="failed to get container status \"1f99ad0b9f0bab53d69f244c17a9367906ed6c7a055dade5ea8de0bef888b257\": rpc error: code = NotFound desc = could not find container \"1f99ad0b9f0bab53d69f244c17a9367906ed6c7a055dade5ea8de0bef888b257\": container with ID starting with 1f99ad0b9f0bab53d69f244c17a9367906ed6c7a055dade5ea8de0bef888b257 not found: ID does not exist" Sep 09 15:06:29.499723 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:06:29.499695 2576 scope.go:117] "RemoveContainer" containerID="78e3d67c6fb1dfe2db1c3c510a660aad8efbb36694dbce09a044c1c8fd46e0ab" Sep 09 15:06:29.499944 ip-10-0-133-66 kubenswrapper[2576]: E0909 15:06:29.499927 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78e3d67c6fb1dfe2db1c3c510a660aad8efbb36694dbce09a044c1c8fd46e0ab\": container with ID starting with 78e3d67c6fb1dfe2db1c3c510a660aad8efbb36694dbce09a044c1c8fd46e0ab not found: ID does not exist" containerID="78e3d67c6fb1dfe2db1c3c510a660aad8efbb36694dbce09a044c1c8fd46e0ab" Sep 09 15:06:29.499989 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:06:29.499947 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78e3d67c6fb1dfe2db1c3c510a660aad8efbb36694dbce09a044c1c8fd46e0ab"} err="failed to get container status \"78e3d67c6fb1dfe2db1c3c510a660aad8efbb36694dbce09a044c1c8fd46e0ab\": rpc error: code = NotFound desc = could not find container \"78e3d67c6fb1dfe2db1c3c510a660aad8efbb36694dbce09a044c1c8fd46e0ab\": container with ID starting with 78e3d67c6fb1dfe2db1c3c510a660aad8efbb36694dbce09a044c1c8fd46e0ab not found: ID does not exist" Sep 09 15:06:29.499989 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:06:29.499959 2576 scope.go:117] "RemoveContainer" containerID="23b3de2eec5fdcf3642621cad03a554f2dd1ced52e873ec250acc6b19c82323f" Sep 09 15:06:29.500196 ip-10-0-133-66 kubenswrapper[2576]: E0909 15:06:29.500176 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23b3de2eec5fdcf3642621cad03a554f2dd1ced52e873ec250acc6b19c82323f\": container with ID starting with 23b3de2eec5fdcf3642621cad03a554f2dd1ced52e873ec250acc6b19c82323f not found: ID does not exist" containerID="23b3de2eec5fdcf3642621cad03a554f2dd1ced52e873ec250acc6b19c82323f" Sep 09 15:06:29.500262 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:06:29.500203 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23b3de2eec5fdcf3642621cad03a554f2dd1ced52e873ec250acc6b19c82323f"} err="failed to get container status \"23b3de2eec5fdcf3642621cad03a554f2dd1ced52e873ec250acc6b19c82323f\": rpc error: code = NotFound desc = could not find container \"23b3de2eec5fdcf3642621cad03a554f2dd1ced52e873ec250acc6b19c82323f\": container with ID starting with 23b3de2eec5fdcf3642621cad03a554f2dd1ced52e873ec250acc6b19c82323f not found: ID does not exist" Sep 09 15:06:29.500262 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:06:29.500226 2576 scope.go:117] "RemoveContainer" containerID="c2f1ab7040e10614b73087e0db10de6bc717406e0161e73df2fc528e4cab4109" Sep 09 15:06:29.500496 ip-10-0-133-66 kubenswrapper[2576]: E0909 15:06:29.500477 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2f1ab7040e10614b73087e0db10de6bc717406e0161e73df2fc528e4cab4109\": container with ID starting with c2f1ab7040e10614b73087e0db10de6bc717406e0161e73df2fc528e4cab4109 not found: ID does not exist" containerID="c2f1ab7040e10614b73087e0db10de6bc717406e0161e73df2fc528e4cab4109" Sep 09 15:06:29.500552 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:06:29.500499 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2f1ab7040e10614b73087e0db10de6bc717406e0161e73df2fc528e4cab4109"} err="failed to get container status \"c2f1ab7040e10614b73087e0db10de6bc717406e0161e73df2fc528e4cab4109\": rpc error: code = NotFound desc = could not find container \"c2f1ab7040e10614b73087e0db10de6bc717406e0161e73df2fc528e4cab4109\": container with ID starting with c2f1ab7040e10614b73087e0db10de6bc717406e0161e73df2fc528e4cab4109 not found: ID does not exist" Sep 09 15:06:29.500552 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:06:29.500513 2576 scope.go:117] "RemoveContainer" containerID="c765e510ac62a829653001e4e64a09a44f8f772a3c905be79823c9c77dbaac40" Sep 09 15:06:29.500757 ip-10-0-133-66 kubenswrapper[2576]: E0909 15:06:29.500739 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c765e510ac62a829653001e4e64a09a44f8f772a3c905be79823c9c77dbaac40\": container with ID starting with c765e510ac62a829653001e4e64a09a44f8f772a3c905be79823c9c77dbaac40 not found: ID does not exist" containerID="c765e510ac62a829653001e4e64a09a44f8f772a3c905be79823c9c77dbaac40" Sep 09 15:06:29.500804 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:06:29.500761 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c765e510ac62a829653001e4e64a09a44f8f772a3c905be79823c9c77dbaac40"} err="failed to get container status \"c765e510ac62a829653001e4e64a09a44f8f772a3c905be79823c9c77dbaac40\": rpc error: code = NotFound desc = could not find container \"c765e510ac62a829653001e4e64a09a44f8f772a3c905be79823c9c77dbaac40\": container with ID starting with c765e510ac62a829653001e4e64a09a44f8f772a3c905be79823c9c77dbaac40 not found: ID does not exist" Sep 09 15:06:29.500804 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:06:29.500776 2576 scope.go:117] "RemoveContainer" containerID="83be447974cb836d6d7d3d3dd8633e3d9ed45c30d5a7e311011b07e3019aed4d" Sep 09 15:06:29.501013 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:06:29.500992 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83be447974cb836d6d7d3d3dd8633e3d9ed45c30d5a7e311011b07e3019aed4d"} err="failed to get container status \"83be447974cb836d6d7d3d3dd8633e3d9ed45c30d5a7e311011b07e3019aed4d\": rpc error: code = NotFound desc = could not find container \"83be447974cb836d6d7d3d3dd8633e3d9ed45c30d5a7e311011b07e3019aed4d\": container with ID starting with 83be447974cb836d6d7d3d3dd8633e3d9ed45c30d5a7e311011b07e3019aed4d not found: ID does not exist" Sep 09 15:06:29.501013 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:06:29.501011 2576 scope.go:117] "RemoveContainer" containerID="1f99ad0b9f0bab53d69f244c17a9367906ed6c7a055dade5ea8de0bef888b257" Sep 09 15:06:29.501261 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:06:29.501243 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f99ad0b9f0bab53d69f244c17a9367906ed6c7a055dade5ea8de0bef888b257"} err="failed to get container status \"1f99ad0b9f0bab53d69f244c17a9367906ed6c7a055dade5ea8de0bef888b257\": rpc error: code = NotFound desc = could not find container \"1f99ad0b9f0bab53d69f244c17a9367906ed6c7a055dade5ea8de0bef888b257\": container with ID starting with 1f99ad0b9f0bab53d69f244c17a9367906ed6c7a055dade5ea8de0bef888b257 not found: ID does not exist" Sep 09 15:06:29.501330 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:06:29.501262 2576 scope.go:117] "RemoveContainer" containerID="78e3d67c6fb1dfe2db1c3c510a660aad8efbb36694dbce09a044c1c8fd46e0ab" Sep 09 15:06:29.501519 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:06:29.501501 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78e3d67c6fb1dfe2db1c3c510a660aad8efbb36694dbce09a044c1c8fd46e0ab"} err="failed to get container status \"78e3d67c6fb1dfe2db1c3c510a660aad8efbb36694dbce09a044c1c8fd46e0ab\": rpc error: code = NotFound desc = could not find container \"78e3d67c6fb1dfe2db1c3c510a660aad8efbb36694dbce09a044c1c8fd46e0ab\": container with ID starting with 78e3d67c6fb1dfe2db1c3c510a660aad8efbb36694dbce09a044c1c8fd46e0ab not found: ID does not exist" Sep 09 15:06:29.501586 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:06:29.501520 2576 scope.go:117] "RemoveContainer" containerID="23b3de2eec5fdcf3642621cad03a554f2dd1ced52e873ec250acc6b19c82323f" Sep 09 15:06:29.501764 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:06:29.501747 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23b3de2eec5fdcf3642621cad03a554f2dd1ced52e873ec250acc6b19c82323f"} err="failed to get container status \"23b3de2eec5fdcf3642621cad03a554f2dd1ced52e873ec250acc6b19c82323f\": rpc error: code = NotFound desc = could not find container \"23b3de2eec5fdcf3642621cad03a554f2dd1ced52e873ec250acc6b19c82323f\": container with ID starting with 23b3de2eec5fdcf3642621cad03a554f2dd1ced52e873ec250acc6b19c82323f not found: ID does not exist" Sep 09 15:06:29.501816 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:06:29.501765 2576 scope.go:117] "RemoveContainer" containerID="c2f1ab7040e10614b73087e0db10de6bc717406e0161e73df2fc528e4cab4109" Sep 09 15:06:29.501989 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:06:29.501973 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2f1ab7040e10614b73087e0db10de6bc717406e0161e73df2fc528e4cab4109"} err="failed to get container status \"c2f1ab7040e10614b73087e0db10de6bc717406e0161e73df2fc528e4cab4109\": rpc error: code = NotFound desc = could not find container \"c2f1ab7040e10614b73087e0db10de6bc717406e0161e73df2fc528e4cab4109\": container with ID starting with c2f1ab7040e10614b73087e0db10de6bc717406e0161e73df2fc528e4cab4109 not found: ID does not exist" Sep 09 15:06:29.502036 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:06:29.501989 2576 scope.go:117] "RemoveContainer" containerID="c765e510ac62a829653001e4e64a09a44f8f772a3c905be79823c9c77dbaac40" Sep 09 15:06:29.502226 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:06:29.502207 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c765e510ac62a829653001e4e64a09a44f8f772a3c905be79823c9c77dbaac40"} err="failed to get container status \"c765e510ac62a829653001e4e64a09a44f8f772a3c905be79823c9c77dbaac40\": rpc error: code = NotFound desc = could not find container \"c765e510ac62a829653001e4e64a09a44f8f772a3c905be79823c9c77dbaac40\": container with ID starting with c765e510ac62a829653001e4e64a09a44f8f772a3c905be79823c9c77dbaac40 not found: ID does not exist" Sep 09 15:06:29.514985 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:06:29.514944 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="64c697b1-4214-4bb1-b578-cff4bc1deba8" path="/var/lib/kubelet/pods/64c697b1-4214-4bb1-b578-cff4bc1deba8/volumes" Sep 09 15:06:29.577902 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:06:29.577863 2576 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/27a978af-ee9e-4dfc-bca6-d877ddd92cc1-registry-tls\") on node \"ip-10-0-133-66.ec2.internal\" DevicePath \"\"" Sep 09 15:06:29.577902 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:06:29.577894 2576 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/27a978af-ee9e-4dfc-bca6-d877ddd92cc1-ui-tls\") on node \"ip-10-0-133-66.ec2.internal\" DevicePath \"\"" Sep 09 15:06:29.578091 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:06:29.577929 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nkkcg\" (UniqueName: \"kubernetes.io/projected/27a978af-ee9e-4dfc-bca6-d877ddd92cc1-kube-api-access-nkkcg\") on node \"ip-10-0-133-66.ec2.internal\" DevicePath \"\"" Sep 09 15:06:29.578091 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:06:29.577941 2576 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/27a978af-ee9e-4dfc-bca6-d877ddd92cc1-offline-tls\") on node \"ip-10-0-133-66.ec2.internal\" DevicePath \"\"" Sep 09 15:06:29.578091 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:06:29.577950 2576 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/27a978af-ee9e-4dfc-bca6-d877ddd92cc1-feast-data\") on node \"ip-10-0-133-66.ec2.internal\" DevicePath \"\"" Sep 09 15:06:29.578091 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:06:29.577958 2576 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/27a978af-ee9e-4dfc-bca6-d877ddd92cc1-online-tls\") on node \"ip-10-0-133-66.ec2.internal\" DevicePath \"\"" Sep 09 15:06:29.729327 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:06:29.729224 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-l2v8b"] Sep 09 15:06:29.733050 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:06:29.733023 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-l2v8b"] Sep 09 15:06:31.514108 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:06:31.514074 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="27a978af-ee9e-4dfc-bca6-d877ddd92cc1" path="/var/lib/kubelet/pods/27a978af-ee9e-4dfc-bca6-d877ddd92cc1/volumes" Sep 09 15:07:05.830003 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:07:05.829967 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-sjq67/must-gather-rfxc2"] Sep 09 15:07:05.830451 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:07:05.830259 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64c697b1-4214-4bb1-b578-cff4bc1deba8" containerName="feast-init" Sep 09 15:07:05.830451 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:07:05.830272 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="64c697b1-4214-4bb1-b578-cff4bc1deba8" containerName="feast-init" Sep 09 15:07:05.830451 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:07:05.830281 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27a978af-ee9e-4dfc-bca6-d877ddd92cc1" containerName="online" Sep 09 15:07:05.830451 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:07:05.830287 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="27a978af-ee9e-4dfc-bca6-d877ddd92cc1" containerName="online" Sep 09 15:07:05.830451 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:07:05.830311 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64c697b1-4214-4bb1-b578-cff4bc1deba8" containerName="offline" Sep 09 15:07:05.830451 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:07:05.830317 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="64c697b1-4214-4bb1-b578-cff4bc1deba8" containerName="offline" Sep 09 15:07:05.830451 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:07:05.830324 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27a978af-ee9e-4dfc-bca6-d877ddd92cc1" containerName="offline" Sep 09 15:07:05.830451 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:07:05.830330 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="27a978af-ee9e-4dfc-bca6-d877ddd92cc1" containerName="offline" Sep 09 15:07:05.830451 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:07:05.830340 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27a978af-ee9e-4dfc-bca6-d877ddd92cc1" containerName="feast-init" Sep 09 15:07:05.830451 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:07:05.830345 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="27a978af-ee9e-4dfc-bca6-d877ddd92cc1" containerName="feast-init" Sep 09 15:07:05.830451 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:07:05.830351 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27a978af-ee9e-4dfc-bca6-d877ddd92cc1" containerName="feast-apply" Sep 09 15:07:05.830451 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:07:05.830356 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="27a978af-ee9e-4dfc-bca6-d877ddd92cc1" containerName="feast-apply" Sep 09 15:07:05.830451 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:07:05.830363 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27a978af-ee9e-4dfc-bca6-d877ddd92cc1" containerName="registry" Sep 09 15:07:05.830451 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:07:05.830368 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="27a978af-ee9e-4dfc-bca6-d877ddd92cc1" containerName="registry" Sep 09 15:07:05.830451 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:07:05.830375 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27a978af-ee9e-4dfc-bca6-d877ddd92cc1" containerName="ui" Sep 09 15:07:05.830451 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:07:05.830380 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="27a978af-ee9e-4dfc-bca6-d877ddd92cc1" containerName="ui" Sep 09 15:07:05.830451 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:07:05.830389 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64c697b1-4214-4bb1-b578-cff4bc1deba8" containerName="feast-apply" Sep 09 15:07:05.830451 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:07:05.830394 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="64c697b1-4214-4bb1-b578-cff4bc1deba8" containerName="feast-apply" Sep 09 15:07:05.830451 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:07:05.830402 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64c697b1-4214-4bb1-b578-cff4bc1deba8" containerName="online" Sep 09 15:07:05.830451 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:07:05.830406 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="64c697b1-4214-4bb1-b578-cff4bc1deba8" containerName="online" Sep 09 15:07:05.830451 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:07:05.830415 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64c697b1-4214-4bb1-b578-cff4bc1deba8" containerName="ui" Sep 09 15:07:05.830451 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:07:05.830420 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="64c697b1-4214-4bb1-b578-cff4bc1deba8" containerName="ui" Sep 09 15:07:05.830451 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:07:05.830462 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="27a978af-ee9e-4dfc-bca6-d877ddd92cc1" containerName="ui" Sep 09 15:07:05.831178 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:07:05.830470 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="64c697b1-4214-4bb1-b578-cff4bc1deba8" containerName="offline" Sep 09 15:07:05.831178 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:07:05.830477 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="27a978af-ee9e-4dfc-bca6-d877ddd92cc1" containerName="registry" Sep 09 15:07:05.831178 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:07:05.830483 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="27a978af-ee9e-4dfc-bca6-d877ddd92cc1" containerName="online" Sep 09 15:07:05.831178 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:07:05.830488 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="27a978af-ee9e-4dfc-bca6-d877ddd92cc1" containerName="offline" Sep 09 15:07:05.831178 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:07:05.830493 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="64c697b1-4214-4bb1-b578-cff4bc1deba8" containerName="online" Sep 09 15:07:05.831178 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:07:05.830498 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="64c697b1-4214-4bb1-b578-cff4bc1deba8" containerName="ui" Sep 09 15:07:05.834665 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:07:05.834643 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-sjq67/must-gather-rfxc2" Sep 09 15:07:05.836956 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:07:05.836931 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-sjq67\"/\"default-dockercfg-fnk76\"" Sep 09 15:07:05.837078 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:07:05.836974 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-sjq67\"/\"kube-root-ca.crt\"" Sep 09 15:07:05.837260 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:07:05.837233 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-sjq67\"/\"openshift-service-ca.crt\"" Sep 09 15:07:05.844274 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:07:05.844255 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-sjq67/must-gather-rfxc2"] Sep 09 15:07:05.977529 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:07:05.977489 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7ad2260c-2960-4d71-ba12-9b5d77cd0d83-must-gather-output\") pod \"must-gather-rfxc2\" (UID: \"7ad2260c-2960-4d71-ba12-9b5d77cd0d83\") " pod="openshift-must-gather-sjq67/must-gather-rfxc2" Sep 09 15:07:05.977529 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:07:05.977534 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4j6kj\" (UniqueName: \"kubernetes.io/projected/7ad2260c-2960-4d71-ba12-9b5d77cd0d83-kube-api-access-4j6kj\") pod \"must-gather-rfxc2\" (UID: \"7ad2260c-2960-4d71-ba12-9b5d77cd0d83\") " pod="openshift-must-gather-sjq67/must-gather-rfxc2" Sep 09 15:07:06.078412 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:07:06.078373 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4j6kj\" (UniqueName: \"kubernetes.io/projected/7ad2260c-2960-4d71-ba12-9b5d77cd0d83-kube-api-access-4j6kj\") pod \"must-gather-rfxc2\" (UID: \"7ad2260c-2960-4d71-ba12-9b5d77cd0d83\") " pod="openshift-must-gather-sjq67/must-gather-rfxc2" Sep 09 15:07:06.078580 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:07:06.078480 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7ad2260c-2960-4d71-ba12-9b5d77cd0d83-must-gather-output\") pod \"must-gather-rfxc2\" (UID: \"7ad2260c-2960-4d71-ba12-9b5d77cd0d83\") " pod="openshift-must-gather-sjq67/must-gather-rfxc2" Sep 09 15:07:06.078842 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:07:06.078822 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7ad2260c-2960-4d71-ba12-9b5d77cd0d83-must-gather-output\") pod \"must-gather-rfxc2\" (UID: \"7ad2260c-2960-4d71-ba12-9b5d77cd0d83\") " pod="openshift-must-gather-sjq67/must-gather-rfxc2" Sep 09 15:07:06.086850 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:07:06.086816 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4j6kj\" (UniqueName: \"kubernetes.io/projected/7ad2260c-2960-4d71-ba12-9b5d77cd0d83-kube-api-access-4j6kj\") pod \"must-gather-rfxc2\" (UID: \"7ad2260c-2960-4d71-ba12-9b5d77cd0d83\") " pod="openshift-must-gather-sjq67/must-gather-rfxc2" Sep 09 15:07:06.154765 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:07:06.154663 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-sjq67/must-gather-rfxc2" Sep 09 15:07:06.424115 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:07:06.424089 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-sjq67/must-gather-rfxc2"] Sep 09 15:07:06.426881 ip-10-0-133-66 kubenswrapper[2576]: W0909 15:07:06.426853 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7ad2260c_2960_4d71_ba12_9b5d77cd0d83.slice/crio-b0f37707cd0d01709a070adcd96eb196a81036c705697d08757b8ba472e5d7f1 WatchSource:0}: Error finding container b0f37707cd0d01709a070adcd96eb196a81036c705697d08757b8ba472e5d7f1: Status 404 returned error can't find the container with id b0f37707cd0d01709a070adcd96eb196a81036c705697d08757b8ba472e5d7f1 Sep 09 15:07:06.428769 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:07:06.428753 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 09 15:07:06.554575 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:07:06.554540 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sjq67/must-gather-rfxc2" event={"ID":"7ad2260c-2960-4d71-ba12-9b5d77cd0d83","Type":"ContainerStarted","Data":"b0f37707cd0d01709a070adcd96eb196a81036c705697d08757b8ba472e5d7f1"} Sep 09 15:07:08.565792 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:07:08.565756 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sjq67/must-gather-rfxc2" event={"ID":"7ad2260c-2960-4d71-ba12-9b5d77cd0d83","Type":"ContainerStarted","Data":"b13b659f4b61549ddee31a8864222cd80cd0830284e3e570c7eb30e4e55df5f2"} Sep 09 15:07:08.566341 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:07:08.565804 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sjq67/must-gather-rfxc2" event={"ID":"7ad2260c-2960-4d71-ba12-9b5d77cd0d83","Type":"ContainerStarted","Data":"94394802f87569328911777523220ef6cdd5ffb636fa5a233e95ae006edecfc6"} Sep 09 15:07:08.581005 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:07:08.580948 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-sjq67/must-gather-rfxc2" podStartSLOduration=2.352162677 podStartE2EDuration="3.580929736s" totalImagesPullingTime="1.228767059s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:07:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:07:08.579683068 +0000 UTC m=+2043.590599323" watchObservedRunningTime="2026-09-09 15:07:08.580929736 +0000 UTC m=+2043.591845987" Sep 09 15:07:08.948836 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:07:08.948750 2576 ???:1] "http: TLS handshake error from 10.0.133.66:56218: EOF" Sep 09 15:07:08.954966 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:07:08.954938 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-nqc8q_f286fc80-41f7-4b6f-8506-b5f0eb2629d3/global-pull-secret-syncer/0.log" Sep 09 15:07:09.072595 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:07:09.072557 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-ppvhx_141380ba-1588-47e3-99ea-1de06df135e9/konnectivity-agent/0.log" Sep 09 15:07:09.152689 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:07:09.152652 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-66.ec2.internal_3aba2b4456e66055b3846a53f8393ab4/haproxy/0.log" Sep 09 15:07:13.334295 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:07:13.334253 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-wl2ff_f3feb49b-9668-453e-a4d5-1cfe142baa3d/kube-state-metrics/0.log" Sep 09 15:07:13.354474 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:07:13.354433 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-wl2ff_f3feb49b-9668-453e-a4d5-1cfe142baa3d/kube-rbac-proxy-main/0.log" Sep 09 15:07:13.378762 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:07:13.378722 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-wl2ff_f3feb49b-9668-453e-a4d5-1cfe142baa3d/kube-rbac-proxy-self/0.log" Sep 09 15:07:13.412013 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:07:13.411976 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-6b959dd5d4-bjzf7_f7a0b560-772a-41e6-82cf-8550c30f69d2/metrics-server/0.log" Sep 09 15:07:13.436364 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:07:13.436334 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-85bbbbbf4-dbt55_bd5fd069-dc9b-401d-84ad-450b3fc43886/monitoring-plugin/0.log" Sep 09 15:07:13.548843 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:07:13.548778 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-65dg4_c270a080-896e-4955-84a7-5ce42fdb4367/node-exporter/0.log" Sep 09 15:07:13.577012 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:07:13.576977 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-65dg4_c270a080-896e-4955-84a7-5ce42fdb4367/kube-rbac-proxy/0.log" Sep 09 15:07:13.597748 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:07:13.597669 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-65dg4_c270a080-896e-4955-84a7-5ce42fdb4367/init-textfile/0.log" Sep 09 15:07:13.798710 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:07:13.798676 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2a5f6e2c-2708-4f25-94bd-bc638555dec2/prometheus/0.log" Sep 09 15:07:13.817983 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:07:13.817956 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2a5f6e2c-2708-4f25-94bd-bc638555dec2/config-reloader/0.log" Sep 09 15:07:13.836546 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:07:13.836512 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2a5f6e2c-2708-4f25-94bd-bc638555dec2/thanos-sidecar/0.log" Sep 09 15:07:13.861665 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:07:13.861534 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2a5f6e2c-2708-4f25-94bd-bc638555dec2/kube-rbac-proxy-web/0.log" Sep 09 15:07:13.889896 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:07:13.889860 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2a5f6e2c-2708-4f25-94bd-bc638555dec2/kube-rbac-proxy/0.log" Sep 09 15:07:13.911846 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:07:13.911814 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2a5f6e2c-2708-4f25-94bd-bc638555dec2/kube-rbac-proxy-thanos/0.log" Sep 09 15:07:13.932375 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:07:13.932341 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2a5f6e2c-2708-4f25-94bd-bc638555dec2/init-config-reloader/0.log" Sep 09 15:07:15.520694 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:07:15.520668 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-79977f9844-gpnd5_2177b959-0d75-41f6-b309-06f66757e45b/networking-console-plugin/0.log" Sep 09 15:07:16.333375 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:07:16.333340 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-75cf6f4fdf-vn5ms_637a9e8d-3d50-46dd-8781-58b5528732cd/console/0.log" Sep 09 15:07:17.592602 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:07:17.592571 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-pkvlm_bb6068ac-f5ef-42a9-9d6e-a3ab55b63949/dns/0.log" Sep 09 15:07:17.611289 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:07:17.611259 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-pkvlm_bb6068ac-f5ef-42a9-9d6e-a3ab55b63949/kube-rbac-proxy/0.log" Sep 09 15:07:17.635496 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:07:17.635468 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5qkp6_90785a55-a3bd-41a7-8d33-85e3b82eb41f/dns-node-resolver/0.log" Sep 09 15:07:18.081797 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:07:18.081767 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-cqf8f_edb6129a-6cbf-4d83-82b0-018fd0e73f8f/node-ca/0.log" Sep 09 15:07:18.690783 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:07:18.690752 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-sjq67/perf-node-gather-daemonset-htkx6"] Sep 09 15:07:18.695906 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:07:18.695879 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-sjq67/perf-node-gather-daemonset-htkx6" Sep 09 15:07:18.702834 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:07:18.702809 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-sjq67/perf-node-gather-daemonset-htkx6"] Sep 09 15:07:18.796236 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:07:18.796204 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c6f20b2c-2d60-4116-bee1-32885eb8ac01-sys\") pod \"perf-node-gather-daemonset-htkx6\" (UID: \"c6f20b2c-2d60-4116-bee1-32885eb8ac01\") " pod="openshift-must-gather-sjq67/perf-node-gather-daemonset-htkx6" Sep 09 15:07:18.796455 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:07:18.796241 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7hnw\" (UniqueName: \"kubernetes.io/projected/c6f20b2c-2d60-4116-bee1-32885eb8ac01-kube-api-access-v7hnw\") pod \"perf-node-gather-daemonset-htkx6\" (UID: \"c6f20b2c-2d60-4116-bee1-32885eb8ac01\") " pod="openshift-must-gather-sjq67/perf-node-gather-daemonset-htkx6" Sep 09 15:07:18.796455 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:07:18.796276 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c6f20b2c-2d60-4116-bee1-32885eb8ac01-lib-modules\") pod \"perf-node-gather-daemonset-htkx6\" (UID: \"c6f20b2c-2d60-4116-bee1-32885eb8ac01\") " pod="openshift-must-gather-sjq67/perf-node-gather-daemonset-htkx6" Sep 09 15:07:18.796455 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:07:18.796382 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c6f20b2c-2d60-4116-bee1-32885eb8ac01-proc\") pod \"perf-node-gather-daemonset-htkx6\" (UID: \"c6f20b2c-2d60-4116-bee1-32885eb8ac01\") " pod="openshift-must-gather-sjq67/perf-node-gather-daemonset-htkx6" Sep 09 15:07:18.796610 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:07:18.796473 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c6f20b2c-2d60-4116-bee1-32885eb8ac01-podres\") pod \"perf-node-gather-daemonset-htkx6\" (UID: \"c6f20b2c-2d60-4116-bee1-32885eb8ac01\") " pod="openshift-must-gather-sjq67/perf-node-gather-daemonset-htkx6" Sep 09 15:07:18.897277 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:07:18.897231 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c6f20b2c-2d60-4116-bee1-32885eb8ac01-sys\") pod \"perf-node-gather-daemonset-htkx6\" (UID: \"c6f20b2c-2d60-4116-bee1-32885eb8ac01\") " pod="openshift-must-gather-sjq67/perf-node-gather-daemonset-htkx6" Sep 09 15:07:18.897451 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:07:18.897284 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7hnw\" (UniqueName: \"kubernetes.io/projected/c6f20b2c-2d60-4116-bee1-32885eb8ac01-kube-api-access-v7hnw\") pod \"perf-node-gather-daemonset-htkx6\" (UID: \"c6f20b2c-2d60-4116-bee1-32885eb8ac01\") " pod="openshift-must-gather-sjq67/perf-node-gather-daemonset-htkx6" Sep 09 15:07:18.897451 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:07:18.897365 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c6f20b2c-2d60-4116-bee1-32885eb8ac01-lib-modules\") pod \"perf-node-gather-daemonset-htkx6\" (UID: \"c6f20b2c-2d60-4116-bee1-32885eb8ac01\") " pod="openshift-must-gather-sjq67/perf-node-gather-daemonset-htkx6" Sep 09 15:07:18.897451 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:07:18.897374 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c6f20b2c-2d60-4116-bee1-32885eb8ac01-sys\") pod \"perf-node-gather-daemonset-htkx6\" (UID: \"c6f20b2c-2d60-4116-bee1-32885eb8ac01\") " pod="openshift-must-gather-sjq67/perf-node-gather-daemonset-htkx6" Sep 09 15:07:18.897451 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:07:18.897402 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c6f20b2c-2d60-4116-bee1-32885eb8ac01-proc\") pod \"perf-node-gather-daemonset-htkx6\" (UID: \"c6f20b2c-2d60-4116-bee1-32885eb8ac01\") " pod="openshift-must-gather-sjq67/perf-node-gather-daemonset-htkx6" Sep 09 15:07:18.897651 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:07:18.897469 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c6f20b2c-2d60-4116-bee1-32885eb8ac01-podres\") pod \"perf-node-gather-daemonset-htkx6\" (UID: \"c6f20b2c-2d60-4116-bee1-32885eb8ac01\") " pod="openshift-must-gather-sjq67/perf-node-gather-daemonset-htkx6" Sep 09 15:07:18.897651 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:07:18.897478 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c6f20b2c-2d60-4116-bee1-32885eb8ac01-lib-modules\") pod \"perf-node-gather-daemonset-htkx6\" (UID: \"c6f20b2c-2d60-4116-bee1-32885eb8ac01\") " pod="openshift-must-gather-sjq67/perf-node-gather-daemonset-htkx6" Sep 09 15:07:18.897651 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:07:18.897557 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c6f20b2c-2d60-4116-bee1-32885eb8ac01-proc\") pod \"perf-node-gather-daemonset-htkx6\" (UID: \"c6f20b2c-2d60-4116-bee1-32885eb8ac01\") " pod="openshift-must-gather-sjq67/perf-node-gather-daemonset-htkx6" Sep 09 15:07:18.897651 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:07:18.897614 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c6f20b2c-2d60-4116-bee1-32885eb8ac01-podres\") pod \"perf-node-gather-daemonset-htkx6\" (UID: \"c6f20b2c-2d60-4116-bee1-32885eb8ac01\") " pod="openshift-must-gather-sjq67/perf-node-gather-daemonset-htkx6" Sep 09 15:07:18.906128 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:07:18.906102 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7hnw\" (UniqueName: \"kubernetes.io/projected/c6f20b2c-2d60-4116-bee1-32885eb8ac01-kube-api-access-v7hnw\") pod \"perf-node-gather-daemonset-htkx6\" (UID: \"c6f20b2c-2d60-4116-bee1-32885eb8ac01\") " pod="openshift-must-gather-sjq67/perf-node-gather-daemonset-htkx6" Sep 09 15:07:19.011353 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:07:19.011293 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-sjq67/perf-node-gather-daemonset-htkx6" Sep 09 15:07:19.053555 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:07:19.053514 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-4cj2m_bb7ae7e5-fd88-4894-b4c8-4b8ea88776d4/serve-healthcheck-canary/0.log" Sep 09 15:07:19.290365 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:07:19.290255 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-sjq67/perf-node-gather-daemonset-htkx6"] Sep 09 15:07:19.294090 ip-10-0-133-66 kubenswrapper[2576]: W0909 15:07:19.294053 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podc6f20b2c_2d60_4116_bee1_32885eb8ac01.slice/crio-9603bfad3cc31ed0e485f97fd083305f474ec8a86adda2f359ad0daa4592451c WatchSource:0}: Error finding container 9603bfad3cc31ed0e485f97fd083305f474ec8a86adda2f359ad0daa4592451c: Status 404 returned error can't find the container with id 9603bfad3cc31ed0e485f97fd083305f474ec8a86adda2f359ad0daa4592451c Sep 09 15:07:19.497713 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:07:19.497685 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-c5lwg_23259762-461b-4ee1-911b-d4a595f0ecf0/kube-rbac-proxy/0.log" Sep 09 15:07:19.515116 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:07:19.515088 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-c5lwg_23259762-461b-4ee1-911b-d4a595f0ecf0/exporter/0.log" Sep 09 15:07:19.534097 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:07:19.534069 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-c5lwg_23259762-461b-4ee1-911b-d4a595f0ecf0/extractor/0.log" Sep 09 15:07:19.625325 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:07:19.625191 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sjq67/perf-node-gather-daemonset-htkx6" event={"ID":"c6f20b2c-2d60-4116-bee1-32885eb8ac01","Type":"ContainerStarted","Data":"40f623f3e72f25c0527d8e5d226139df80446e436a0ea1af068c52a7c1a71c1c"} Sep 09 15:07:19.625325 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:07:19.625269 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sjq67/perf-node-gather-daemonset-htkx6" event={"ID":"c6f20b2c-2d60-4116-bee1-32885eb8ac01","Type":"ContainerStarted","Data":"9603bfad3cc31ed0e485f97fd083305f474ec8a86adda2f359ad0daa4592451c"} Sep 09 15:07:19.625939 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:07:19.625917 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-sjq67/perf-node-gather-daemonset-htkx6" Sep 09 15:07:19.643606 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:07:19.643548 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-sjq67/perf-node-gather-daemonset-htkx6" podStartSLOduration=1.6435335009999998 podStartE2EDuration="1.643533501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 15:07:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:07:19.641000781 +0000 UTC m=+2054.651917054" watchObservedRunningTime="2026-09-09 15:07:19.643533501 +0000 UTC m=+2054.654449752" Sep 09 15:07:23.640110 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:07:23.640070 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-mxjvn_afa612b5-3120-49fb-a817-e85de6937bd3/migrator/0.log" Sep 09 15:07:23.657664 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:07:23.657630 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-mxjvn_afa612b5-3120-49fb-a817-e85de6937bd3/graceful-termination/0.log" Sep 09 15:07:23.916609 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:07:23.916532 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-8d7dd4675-xtkrm_342ca2d8-8fa0-45b6-9aaf-4bcf7117b021/kube-storage-version-migrator-operator/1.log" Sep 09 15:07:23.918382 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:07:23.918358 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-8d7dd4675-xtkrm_342ca2d8-8fa0-45b6-9aaf-4bcf7117b021/kube-storage-version-migrator-operator/0.log" Sep 09 15:07:24.810328 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:07:24.810260 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-p5bsf_3c7d6121-a834-42d0-9b78-583260fd4dce/kube-multus-additional-cni-plugins/0.log" Sep 09 15:07:24.831813 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:07:24.831782 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-p5bsf_3c7d6121-a834-42d0-9b78-583260fd4dce/egress-router-binary-copy/0.log" Sep 09 15:07:24.849999 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:07:24.849969 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-p5bsf_3c7d6121-a834-42d0-9b78-583260fd4dce/cni-plugins/0.log" Sep 09 15:07:24.869571 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:07:24.869542 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-p5bsf_3c7d6121-a834-42d0-9b78-583260fd4dce/bond-cni-plugin/0.log" Sep 09 15:07:24.887606 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:07:24.887570 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-p5bsf_3c7d6121-a834-42d0-9b78-583260fd4dce/routeoverride-cni/0.log" Sep 09 15:07:24.905458 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:07:24.905427 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-p5bsf_3c7d6121-a834-42d0-9b78-583260fd4dce/whereabouts-cni-bincopy/0.log" Sep 09 15:07:24.922537 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:07:24.922510 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-p5bsf_3c7d6121-a834-42d0-9b78-583260fd4dce/whereabouts-cni/0.log" Sep 09 15:07:25.124035 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:07:25.123958 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-w5kx5_ab41acab-1926-436b-b208-c18b0b375e9f/kube-multus/0.log" Sep 09 15:07:25.221977 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:07:25.221948 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-wcj2p_ab36c1e4-a65c-4cf9-ada4-f8a87d4b463b/network-metrics-daemon/0.log" Sep 09 15:07:25.238530 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:07:25.238504 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-wcj2p_ab36c1e4-a65c-4cf9-ada4-f8a87d4b463b/kube-rbac-proxy/0.log" Sep 09 15:07:26.085063 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:07:26.085033 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7xcn5_186aab90-266b-4eae-977a-3f053b423ef3/ovn-controller/0.log" Sep 09 15:07:26.099345 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:07:26.099294 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7xcn5_186aab90-266b-4eae-977a-3f053b423ef3/ovn-acl-logging/0.log" Sep 09 15:07:26.099345 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:07:26.099336 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7xcn5_186aab90-266b-4eae-977a-3f053b423ef3/ovn-acl-logging/1.log" Sep 09 15:07:26.123109 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:07:26.123077 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7xcn5_186aab90-266b-4eae-977a-3f053b423ef3/kube-rbac-proxy-node/0.log" Sep 09 15:07:26.146841 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:07:26.146812 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7xcn5_186aab90-266b-4eae-977a-3f053b423ef3/kube-rbac-proxy-ovn-metrics/0.log" Sep 09 15:07:26.161138 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:07:26.161108 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7xcn5_186aab90-266b-4eae-977a-3f053b423ef3/northd/0.log" Sep 09 15:07:26.179499 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:07:26.179477 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7xcn5_186aab90-266b-4eae-977a-3f053b423ef3/nbdb/0.log" Sep 09 15:07:26.197517 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:07:26.197495 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7xcn5_186aab90-266b-4eae-977a-3f053b423ef3/sbdb/0.log" Sep 09 15:07:26.250506 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:07:26.250459 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7xcn5_186aab90-266b-4eae-977a-3f053b423ef3/ovnkube-controller/0.log" Sep 09 15:07:26.644480 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:07:26.644455 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-sjq67/perf-node-gather-daemonset-htkx6" Sep 09 15:07:27.264032 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:07:27.264000 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-source-685456655f-b8gsv_176011b6-b6b3-4b9d-a930-848977cb0006/check-endpoints/0.log" Sep 09 15:07:27.306100 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:07:27.306069 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-5vdl5_ce063353-8e00-4701-8fb5-9b32aa0586e1/network-check-target-container/0.log" Sep 09 15:07:28.147288 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:07:28.147252 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-4kp4s_b134513f-e0e7-452d-8b61-bae4c9007795/iptables-alerter/0.log" Sep 09 15:07:28.890777 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:07:28.890688 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-p5wbc_50c734e0-544e-44f4-9c79-a398e559a276/tuned/0.log" Sep 09 15:07:32.128395 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:07:32.128365 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-service-ca_service-ca-75d99c484-sdzv5_a94612bf-824e-47cf-a60d-ef50cfe29ca5/service-ca-controller/0.log" Sep 09 15:07:32.784128 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:07:32.784098 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-qlxsx_5a445f65-2ccc-44f4-97ad-1eee2597b711/csi-driver/0.log" Sep 09 15:07:32.819258 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:07:32.819228 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-qlxsx_5a445f65-2ccc-44f4-97ad-1eee2597b711/csi-node-driver-registrar/0.log" Sep 09 15:07:32.852732 ip-10-0-133-66 kubenswrapper[2576]: I0909 15:07:32.852664 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-qlxsx_5a445f65-2ccc-44f4-97ad-1eee2597b711/csi-liveness-probe/0.log"