Sep 11 08:44:31.407279 ip-10-0-128-166 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Sep 11 08:44:31.407291 ip-10-0-128-166 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Sep 11 08:44:31.407298 ip-10-0-128-166 systemd[1]: kubelet.service: Failed with result 'resources'. Sep 11 08:44:31.407619 ip-10-0-128-166 systemd[1]: Failed to start Kubernetes Kubelet. Sep 11 08:44:41.442876 ip-10-0-128-166 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Sep 11 08:44:41.442888 ip-10-0-128-166 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 6e57e30e40964e4aa6c0ba868aae0c37 -- Sep 11 08:46:18.070908 ip-10-0-128-166 systemd[1]: Starting Kubernetes Kubelet... Sep 11 08:46:18.501354 ip-10-0-128-166 kubenswrapper[2572]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 11 08:46:18.501354 ip-10-0-128-166 kubenswrapper[2572]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 11 08:46:18.501354 ip-10-0-128-166 kubenswrapper[2572]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 11 08:46:18.501354 ip-10-0-128-166 kubenswrapper[2572]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 11 08:46:18.501354 ip-10-0-128-166 kubenswrapper[2572]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 11 08:46:18.502301 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.502216 2572 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 11 08:46:18.507424 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.507409 2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 08:46:18.507424 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.507425 2572 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 08:46:18.507489 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.507428 2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 08:46:18.507489 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.507432 2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 08:46:18.507489 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.507434 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 08:46:18.507489 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.507437 2572 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 08:46:18.507489 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.507440 2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 08:46:18.507489 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.507443 2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 08:46:18.507489 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.507445 2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 08:46:18.507489 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.507447 2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 08:46:18.507489 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.507450 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 08:46:18.507489 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.507453 2572 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 08:46:18.507489 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.507455 2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 08:46:18.507489 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.507457 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 08:46:18.507489 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.507460 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 08:46:18.507489 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.507463 2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 08:46:18.507489 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.507465 2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 08:46:18.507489 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.507468 2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 08:46:18.507489 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.507471 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 08:46:18.507489 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.507473 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 08:46:18.507489 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.507476 2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 08:46:18.507489 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.507485 2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 08:46:18.507988 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.507489 2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 08:46:18.507988 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.507492 2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 08:46:18.507988 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.507496 2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 08:46:18.507988 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.507500 2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 08:46:18.507988 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.507503 2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 08:46:18.507988 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.507506 2572 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 08:46:18.507988 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.507509 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 08:46:18.507988 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.507511 2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 08:46:18.507988 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.507514 2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 08:46:18.507988 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.507516 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 08:46:18.507988 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.507519 2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 08:46:18.507988 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.507521 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 08:46:18.507988 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.507524 2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 08:46:18.507988 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.507526 2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 08:46:18.507988 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.507528 2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 08:46:18.507988 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.507531 2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 08:46:18.507988 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.507534 2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 08:46:18.507988 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.507536 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 08:46:18.507988 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.507538 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 08:46:18.507988 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.507541 2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 08:46:18.508469 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.507543 2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 08:46:18.508469 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.507545 2572 feature_gate.go:328] unrecognized feature gate: Example Sep 11 08:46:18.508469 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.507548 2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 08:46:18.508469 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.507550 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 08:46:18.508469 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.507553 2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 08:46:18.508469 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.507555 2572 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 08:46:18.508469 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.507557 2572 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 08:46:18.508469 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.507560 2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 08:46:18.508469 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.507562 2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 08:46:18.508469 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.507565 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 08:46:18.508469 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.507567 2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 08:46:18.508469 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.507569 2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 08:46:18.508469 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.507574 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 08:46:18.508469 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.507577 2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 08:46:18.508469 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.507579 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 08:46:18.508469 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.507594 2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 08:46:18.508469 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.507598 2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 08:46:18.508469 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.507602 2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 08:46:18.508469 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.507605 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 08:46:18.508469 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.507608 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 08:46:18.508994 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.507611 2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 08:46:18.508994 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.507614 2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 08:46:18.508994 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.507617 2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 08:46:18.508994 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.507619 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 08:46:18.508994 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.507622 2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 08:46:18.508994 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.507625 2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 08:46:18.508994 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.507629 2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 08:46:18.508994 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.507632 2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 08:46:18.508994 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.507635 2572 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 08:46:18.508994 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.507638 2572 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 08:46:18.508994 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.507641 2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 08:46:18.508994 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.507644 2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 08:46:18.508994 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.507647 2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 08:46:18.508994 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.507649 2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 08:46:18.508994 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.507652 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 08:46:18.508994 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.507654 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 08:46:18.508994 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.507657 2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 08:46:18.508994 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.507659 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 08:46:18.508994 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.507662 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 08:46:18.509453 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.507665 2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 08:46:18.509453 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.507668 2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 08:46:18.509453 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.507671 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 08:46:18.509453 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.507674 2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 08:46:18.509453 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.507676 2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 08:46:18.509453 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.507679 2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 08:46:18.509453 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.507683 2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 08:46:18.509453 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.507685 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 08:46:18.509453 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.507688 2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 08:46:18.509453 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.507691 2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 08:46:18.509453 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.507693 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 08:46:18.509453 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.507696 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 08:46:18.509453 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.507698 2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 08:46:18.509453 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.507701 2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 08:46:18.509453 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.507704 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 08:46:18.509453 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.507706 2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 08:46:18.509453 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.507709 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 08:46:18.509453 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.507711 2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 08:46:18.509453 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.508374 2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 08:46:18.510033 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.508380 2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 08:46:18.510033 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.508383 2572 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 08:46:18.510033 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.508386 2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 08:46:18.510033 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.508391 2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 08:46:18.510033 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.508394 2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 08:46:18.510033 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.508397 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 08:46:18.510033 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.508399 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 08:46:18.510033 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.508403 2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 08:46:18.510033 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.508408 2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 08:46:18.510033 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.508411 2572 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 08:46:18.510033 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.508414 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 08:46:18.510033 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.508416 2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 08:46:18.510033 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.508419 2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 08:46:18.510033 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.508421 2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 08:46:18.510033 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.508424 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 08:46:18.510033 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.508426 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 08:46:18.510033 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.508429 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 08:46:18.510033 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.508432 2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 08:46:18.510033 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.508434 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 08:46:18.510532 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.508437 2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 08:46:18.510532 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.508440 2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 08:46:18.510532 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.508443 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 08:46:18.510532 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.508446 2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 08:46:18.510532 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.508448 2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 08:46:18.510532 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.508451 2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 08:46:18.510532 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.508453 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 08:46:18.510532 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.508456 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 08:46:18.510532 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.508458 2572 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 08:46:18.510532 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.508461 2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 08:46:18.510532 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.508463 2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 08:46:18.510532 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.508465 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 08:46:18.510532 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.508468 2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 08:46:18.510532 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.508470 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 08:46:18.510532 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.508473 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 08:46:18.510532 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.508475 2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 08:46:18.510532 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.508480 2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 08:46:18.510532 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.508482 2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 08:46:18.510532 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.508485 2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 08:46:18.510532 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.508487 2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 08:46:18.511043 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.508490 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 08:46:18.511043 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.508492 2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 08:46:18.511043 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.508494 2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 08:46:18.511043 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.508497 2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 08:46:18.511043 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.508499 2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 08:46:18.511043 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.508502 2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 08:46:18.511043 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.508506 2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 08:46:18.511043 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.508510 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 08:46:18.511043 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.508513 2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 08:46:18.511043 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.508516 2572 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 08:46:18.511043 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.508518 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 08:46:18.511043 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.508521 2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 08:46:18.511043 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.508523 2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 08:46:18.511043 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.508526 2572 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 08:46:18.511043 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.508528 2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 08:46:18.511043 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.508531 2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 08:46:18.511043 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.508533 2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 08:46:18.511043 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.508536 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 08:46:18.511043 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.508539 2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 08:46:18.511509 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.508544 2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 08:46:18.511509 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.508561 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 08:46:18.511509 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.508564 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 08:46:18.511509 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.508567 2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 08:46:18.511509 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.508569 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 08:46:18.511509 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.508573 2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 08:46:18.511509 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.508576 2572 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 08:46:18.511509 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.508578 2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 08:46:18.511509 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.508581 2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 08:46:18.511509 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.508598 2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 08:46:18.511509 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.508601 2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 08:46:18.511509 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.508603 2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 08:46:18.511509 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.508606 2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 08:46:18.511509 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.508608 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 08:46:18.511509 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.508611 2572 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 08:46:18.511509 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.508613 2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 08:46:18.511509 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.508616 2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 08:46:18.511509 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.508619 2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 08:46:18.511509 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.508622 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 08:46:18.511509 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.508624 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 08:46:18.512022 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.508627 2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 08:46:18.512022 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.508629 2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 08:46:18.512022 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.508631 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 08:46:18.512022 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.508634 2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 08:46:18.512022 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.508636 2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 08:46:18.512022 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.508639 2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 08:46:18.512022 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.508641 2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 08:46:18.512022 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.508643 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 08:46:18.512022 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.508647 2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 08:46:18.512022 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.508649 2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 08:46:18.512022 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.508652 2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 08:46:18.512022 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.508654 2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 08:46:18.512022 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.508657 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 08:46:18.512022 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.508659 2572 feature_gate.go:328] unrecognized feature gate: Example Sep 11 08:46:18.512022 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.508661 2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 08:46:18.512022 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.508664 2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 08:46:18.512022 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.508666 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 08:46:18.512022 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.508670 2572 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 08:46:18.512022 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.508673 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 08:46:18.512022 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.508675 2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 08:46:18.512536 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.509868 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 08:46:18.512536 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.509875 2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 08:46:18.512536 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.509878 2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 08:46:18.512536 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.509881 2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 08:46:18.512536 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.509884 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 08:46:18.512536 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.509888 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 08:46:18.512536 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.509890 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 08:46:18.512536 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.509893 2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 08:46:18.512536 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.509896 2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 08:46:18.512536 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.509899 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 08:46:18.512536 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.509903 2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 08:46:18.512536 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.509906 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 08:46:18.512536 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.509909 2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 08:46:18.512536 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.509911 2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 08:46:18.512536 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.509914 2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 08:46:18.512536 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.509916 2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 08:46:18.512536 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.509919 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 08:46:18.512536 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.509921 2572 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 08:46:18.512536 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.509924 2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 08:46:18.512536 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.509926 2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 08:46:18.513037 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.509931 2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 08:46:18.513037 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.509934 2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 08:46:18.513037 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.509937 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 08:46:18.513037 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.509940 2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 08:46:18.513037 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.509942 2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 08:46:18.513037 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.509945 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 08:46:18.513037 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.509947 2572 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 08:46:18.513037 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.509950 2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 08:46:18.513037 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.509952 2572 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 08:46:18.513037 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.509955 2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 08:46:18.513037 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.509958 2572 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 08:46:18.513037 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.509960 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 08:46:18.513037 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.509964 2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 08:46:18.513037 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.509968 2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 08:46:18.513037 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.509971 2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 08:46:18.513037 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.509973 2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 08:46:18.513037 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.509976 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 08:46:18.513037 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.509978 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 08:46:18.513037 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.509981 2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 08:46:18.513522 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.509983 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 08:46:18.513522 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.509986 2572 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 08:46:18.513522 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.509988 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 08:46:18.513522 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.509991 2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 08:46:18.513522 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.509993 2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 08:46:18.513522 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.509996 2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 08:46:18.513522 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.509998 2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 08:46:18.513522 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.510000 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 08:46:18.513522 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.510003 2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 08:46:18.513522 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.510005 2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 08:46:18.513522 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.510007 2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 08:46:18.513522 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.510010 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 08:46:18.513522 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.510012 2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 08:46:18.513522 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.510015 2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 08:46:18.513522 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.510018 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 08:46:18.513522 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.510020 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 08:46:18.513522 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.510023 2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 08:46:18.513522 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.510025 2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 08:46:18.513522 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.510029 2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 08:46:18.513522 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.510032 2572 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 08:46:18.514028 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.510034 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 08:46:18.514028 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.510037 2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 08:46:18.514028 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.510040 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 08:46:18.514028 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.510043 2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 08:46:18.514028 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.510046 2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 08:46:18.514028 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.510048 2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 08:46:18.514028 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.510051 2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 08:46:18.514028 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.510053 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 08:46:18.514028 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.510056 2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 08:46:18.514028 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.510059 2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 08:46:18.514028 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.510061 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 08:46:18.514028 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.510064 2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 08:46:18.514028 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.510066 2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 08:46:18.514028 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.510068 2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 08:46:18.514028 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.510071 2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 08:46:18.514028 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.510073 2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 08:46:18.514028 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.510075 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 08:46:18.514028 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.510078 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 08:46:18.514028 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.510080 2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 08:46:18.514028 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.510082 2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 08:46:18.514505 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.510085 2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 08:46:18.514505 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.510087 2572 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 08:46:18.514505 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.510089 2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 08:46:18.514505 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.510092 2572 feature_gate.go:328] unrecognized feature gate: Example Sep 11 08:46:18.514505 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.510095 2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 08:46:18.514505 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.510097 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 08:46:18.514505 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.510101 2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 08:46:18.514505 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.510105 2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 08:46:18.514505 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.510108 2572 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 08:46:18.514505 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.510111 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 08:46:18.514505 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.510113 2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 08:46:18.514505 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.510116 2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 08:46:18.514505 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.510118 2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 08:46:18.514505 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.510120 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 08:46:18.514505 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.510123 2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 08:46:18.514505 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.510126 2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 08:46:18.514505 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.510128 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 08:46:18.514505 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.510131 2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 08:46:18.514505 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.510133 2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 08:46:18.515010 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.510136 2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 08:46:18.515010 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510197 2572 flags.go:64] FLAG: --address="0.0.0.0" Sep 11 08:46:18.515010 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510233 2572 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 11 08:46:18.515010 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510238 2572 flags.go:64] FLAG: --anonymous-auth="true" Sep 11 08:46:18.515010 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510242 2572 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 11 08:46:18.515010 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510246 2572 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 11 08:46:18.515010 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510249 2572 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 11 08:46:18.515010 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510253 2572 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 11 08:46:18.515010 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510257 2572 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 11 08:46:18.515010 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510260 2572 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 11 08:46:18.515010 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510263 2572 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 11 08:46:18.515010 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510266 2572 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 11 08:46:18.515010 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510270 2572 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 11 08:46:18.515010 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510273 2572 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 11 08:46:18.515010 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510276 2572 flags.go:64] FLAG: --cgroup-root="" Sep 11 08:46:18.515010 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510279 2572 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 11 08:46:18.515010 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510281 2572 flags.go:64] FLAG: --client-ca-file="" Sep 11 08:46:18.515010 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510284 2572 flags.go:64] FLAG: --cloud-provider="external" Sep 11 08:46:18.515010 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510287 2572 flags.go:64] FLAG: --cluster-dns="[]" Sep 11 08:46:18.515010 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510291 2572 flags.go:64] FLAG: --cluster-domain="" Sep 11 08:46:18.515010 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510294 2572 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 11 08:46:18.515010 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510297 2572 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 11 08:46:18.515010 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510303 2572 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 11 08:46:18.515010 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510306 2572 flags.go:64] FLAG: --container-log-max-files="5" Sep 11 08:46:18.515598 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510310 2572 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 11 08:46:18.515598 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510313 2572 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 11 08:46:18.515598 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510316 2572 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 11 08:46:18.515598 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510319 2572 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 11 08:46:18.515598 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510322 2572 flags.go:64] FLAG: --contention-profiling="false" Sep 11 08:46:18.515598 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510325 2572 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 11 08:46:18.515598 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510328 2572 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 11 08:46:18.515598 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510330 2572 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 11 08:46:18.515598 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510333 2572 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 11 08:46:18.515598 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510337 2572 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 11 08:46:18.515598 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510339 2572 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 11 08:46:18.515598 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510342 2572 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 11 08:46:18.515598 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510345 2572 flags.go:64] FLAG: --enable-load-reader="false" Sep 11 08:46:18.515598 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510348 2572 flags.go:64] FLAG: --enable-server="true" Sep 11 08:46:18.515598 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510350 2572 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 11 08:46:18.515598 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510354 2572 flags.go:64] FLAG: --event-burst="100" Sep 11 08:46:18.515598 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510357 2572 flags.go:64] FLAG: --event-qps="50" Sep 11 08:46:18.515598 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510359 2572 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 11 08:46:18.515598 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510375 2572 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 11 08:46:18.515598 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510379 2572 flags.go:64] FLAG: --eviction-hard="" Sep 11 08:46:18.515598 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510383 2572 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 11 08:46:18.515598 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510386 2572 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 11 08:46:18.515598 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510389 2572 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 11 08:46:18.515598 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510392 2572 flags.go:64] FLAG: --eviction-soft="" Sep 11 08:46:18.515598 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510395 2572 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 11 08:46:18.516193 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510397 2572 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 11 08:46:18.516193 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510400 2572 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 11 08:46:18.516193 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510403 2572 flags.go:64] FLAG: --experimental-mounter-path="" Sep 11 08:46:18.516193 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510406 2572 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 11 08:46:18.516193 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510409 2572 flags.go:64] FLAG: --fail-swap-on="true" Sep 11 08:46:18.516193 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510415 2572 flags.go:64] FLAG: --feature-gates="" Sep 11 08:46:18.516193 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510419 2572 flags.go:64] FLAG: --file-check-frequency="20s" Sep 11 08:46:18.516193 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510422 2572 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 11 08:46:18.516193 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510425 2572 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 11 08:46:18.516193 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510429 2572 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 11 08:46:18.516193 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510433 2572 flags.go:64] FLAG: --healthz-port="10248" Sep 11 08:46:18.516193 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510436 2572 flags.go:64] FLAG: --help="false" Sep 11 08:46:18.516193 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510439 2572 flags.go:64] FLAG: --hostname-override="ip-10-0-128-166.ec2.internal" Sep 11 08:46:18.516193 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510442 2572 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 11 08:46:18.516193 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510445 2572 flags.go:64] FLAG: --http-check-frequency="20s" Sep 11 08:46:18.516193 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510448 2572 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 11 08:46:18.516193 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510451 2572 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 11 08:46:18.516193 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510454 2572 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 11 08:46:18.516193 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510457 2572 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 11 08:46:18.516193 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510459 2572 flags.go:64] FLAG: --image-service-endpoint="" Sep 11 08:46:18.516193 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510462 2572 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 11 08:46:18.516193 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510465 2572 flags.go:64] FLAG: --kube-api-burst="100" Sep 11 08:46:18.516193 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510468 2572 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 11 08:46:18.516792 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510471 2572 flags.go:64] FLAG: --kube-api-qps="50" Sep 11 08:46:18.516792 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510474 2572 flags.go:64] FLAG: --kube-reserved="" Sep 11 08:46:18.516792 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510477 2572 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 11 08:46:18.516792 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510479 2572 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 11 08:46:18.516792 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510482 2572 flags.go:64] FLAG: --kubelet-cgroups="" Sep 11 08:46:18.516792 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510485 2572 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 11 08:46:18.516792 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510488 2572 flags.go:64] FLAG: --lock-file="" Sep 11 08:46:18.516792 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510491 2572 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 11 08:46:18.516792 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510493 2572 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 11 08:46:18.516792 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510496 2572 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 11 08:46:18.516792 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510501 2572 flags.go:64] FLAG: --log-json-split-stream="false" Sep 11 08:46:18.516792 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510504 2572 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 11 08:46:18.516792 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510506 2572 flags.go:64] FLAG: --log-text-split-stream="false" Sep 11 08:46:18.516792 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510509 2572 flags.go:64] FLAG: --logging-format="text" Sep 11 08:46:18.516792 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510513 2572 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 11 08:46:18.516792 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510516 2572 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 11 08:46:18.516792 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510519 2572 flags.go:64] FLAG: --manifest-url="" Sep 11 08:46:18.516792 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510522 2572 flags.go:64] FLAG: --manifest-url-header="" Sep 11 08:46:18.516792 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510526 2572 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 11 08:46:18.516792 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510531 2572 flags.go:64] FLAG: --max-open-files="1000000" Sep 11 08:46:18.516792 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510534 2572 flags.go:64] FLAG: --max-pods="110" Sep 11 08:46:18.516792 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510538 2572 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 11 08:46:18.516792 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510541 2572 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 11 08:46:18.516792 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510544 2572 flags.go:64] FLAG: --memory-manager-policy="None" Sep 11 08:46:18.516792 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510547 2572 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 11 08:46:18.517413 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510549 2572 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 11 08:46:18.517413 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510552 2572 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 11 08:46:18.517413 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510555 2572 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 11 08:46:18.517413 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510562 2572 flags.go:64] FLAG: --node-status-max-images="50" Sep 11 08:46:18.517413 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510565 2572 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 11 08:46:18.517413 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510567 2572 flags.go:64] FLAG: --oom-score-adj="-999" Sep 11 08:46:18.517413 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510570 2572 flags.go:64] FLAG: --pod-cidr="" Sep 11 08:46:18.517413 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510573 2572 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10" Sep 11 08:46:18.517413 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510578 2572 flags.go:64] FLAG: --pod-manifest-path="" Sep 11 08:46:18.517413 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510581 2572 flags.go:64] FLAG: --pod-max-pids="-1" Sep 11 08:46:18.517413 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510602 2572 flags.go:64] FLAG: --pods-per-core="0" Sep 11 08:46:18.517413 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510605 2572 flags.go:64] FLAG: --port="10250" Sep 11 08:46:18.517413 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510608 2572 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 11 08:46:18.517413 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510611 2572 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-02c5b1b5604a9a7cd" Sep 11 08:46:18.517413 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510614 2572 flags.go:64] FLAG: --qos-reserved="" Sep 11 08:46:18.517413 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510617 2572 flags.go:64] FLAG: --read-only-port="10255" Sep 11 08:46:18.517413 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510619 2572 flags.go:64] FLAG: --register-node="true" Sep 11 08:46:18.517413 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510622 2572 flags.go:64] FLAG: --register-with-taints="" Sep 11 08:46:18.517413 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510625 2572 flags.go:64] FLAG: --registry-burst="10" Sep 11 08:46:18.517413 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510628 2572 flags.go:64] FLAG: --registry-qps="5" Sep 11 08:46:18.517413 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510631 2572 flags.go:64] FLAG: --reserved-cpus="" Sep 11 08:46:18.517413 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510635 2572 flags.go:64] FLAG: --reserved-memory="" Sep 11 08:46:18.517413 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510639 2572 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 11 08:46:18.517413 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510642 2572 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 11 08:46:18.517413 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510645 2572 flags.go:64] FLAG: --rotate-certificates="false" Sep 11 08:46:18.518053 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510648 2572 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 11 08:46:18.518053 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510651 2572 flags.go:64] FLAG: --runonce="false" Sep 11 08:46:18.518053 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510655 2572 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 11 08:46:18.518053 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510658 2572 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 11 08:46:18.518053 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510661 2572 flags.go:64] FLAG: --seccomp-default="false" Sep 11 08:46:18.518053 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510664 2572 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 11 08:46:18.518053 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510667 2572 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 11 08:46:18.518053 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510670 2572 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 11 08:46:18.518053 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510673 2572 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 11 08:46:18.518053 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510676 2572 flags.go:64] FLAG: --storage-driver-password="root" Sep 11 08:46:18.518053 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510679 2572 flags.go:64] FLAG: --storage-driver-secure="false" Sep 11 08:46:18.518053 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510682 2572 flags.go:64] FLAG: --storage-driver-table="stats" Sep 11 08:46:18.518053 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510685 2572 flags.go:64] FLAG: --storage-driver-user="root" Sep 11 08:46:18.518053 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510688 2572 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 11 08:46:18.518053 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510691 2572 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 11 08:46:18.518053 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510693 2572 flags.go:64] FLAG: --system-cgroups="" Sep 11 08:46:18.518053 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510696 2572 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 11 08:46:18.518053 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510701 2572 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 11 08:46:18.518053 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510704 2572 flags.go:64] FLAG: --tls-cert-file="" Sep 11 08:46:18.518053 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510707 2572 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 11 08:46:18.518053 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510711 2572 flags.go:64] FLAG: --tls-min-version="" Sep 11 08:46:18.518053 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510713 2572 flags.go:64] FLAG: --tls-private-key-file="" Sep 11 08:46:18.518053 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510717 2572 flags.go:64] FLAG: --topology-manager-policy="none" Sep 11 08:46:18.518053 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510719 2572 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 11 08:46:18.518053 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510722 2572 flags.go:64] FLAG: --topology-manager-scope="container" Sep 11 08:46:18.518957 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510725 2572 flags.go:64] FLAG: --v="2" Sep 11 08:46:18.518957 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510728 2572 flags.go:64] FLAG: --version="false" Sep 11 08:46:18.518957 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510732 2572 flags.go:64] FLAG: --vmodule="" Sep 11 08:46:18.518957 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510737 2572 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 11 08:46:18.518957 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.510740 2572 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 11 08:46:18.518957 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.510856 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 08:46:18.518957 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.510859 2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 08:46:18.518957 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.510862 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 08:46:18.518957 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.510865 2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 08:46:18.518957 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.510869 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 08:46:18.518957 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.510872 2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 08:46:18.518957 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.510875 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 08:46:18.518957 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.510877 2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 08:46:18.518957 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.510880 2572 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 08:46:18.518957 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.510882 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 08:46:18.518957 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.510885 2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 08:46:18.518957 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.510887 2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 08:46:18.518957 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.510890 2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 08:46:18.518957 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.510892 2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 08:46:18.518957 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.510895 2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 08:46:18.518957 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.510897 2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 08:46:18.519786 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.510899 2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 08:46:18.519786 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.510902 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 08:46:18.519786 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.510904 2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 08:46:18.519786 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.510907 2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 08:46:18.519786 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.510909 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 08:46:18.519786 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.510911 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 08:46:18.519786 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.510914 2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 08:46:18.519786 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.510916 2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 08:46:18.519786 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.510919 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 08:46:18.519786 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.510922 2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 08:46:18.519786 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.510925 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 08:46:18.519786 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.510928 2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 08:46:18.519786 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.510930 2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 08:46:18.519786 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.510934 2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 08:46:18.519786 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.510938 2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 08:46:18.519786 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.510941 2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 08:46:18.519786 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.510943 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 08:46:18.519786 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.510947 2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 08:46:18.519786 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.510951 2572 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 08:46:18.520255 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.510954 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 08:46:18.520255 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.510958 2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 08:46:18.520255 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.510961 2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 08:46:18.520255 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.510964 2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 08:46:18.520255 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.510967 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 08:46:18.520255 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.510969 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 08:46:18.520255 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.510972 2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 08:46:18.520255 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.510975 2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 08:46:18.520255 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.510977 2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 08:46:18.520255 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.510980 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 08:46:18.520255 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.510982 2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 08:46:18.520255 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.510985 2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 08:46:18.520255 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.510987 2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 08:46:18.520255 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.510990 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 08:46:18.520255 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.510992 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 08:46:18.520255 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.510995 2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 08:46:18.520255 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.510999 2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 08:46:18.520255 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.511002 2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 08:46:18.520255 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.511005 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 08:46:18.520255 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.511008 2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 08:46:18.520758 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.511011 2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 08:46:18.520758 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.511013 2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 08:46:18.520758 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.511016 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 08:46:18.520758 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.511019 2572 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 08:46:18.520758 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.511021 2572 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 08:46:18.520758 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.511024 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 08:46:18.520758 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.511027 2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 08:46:18.520758 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.511031 2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 08:46:18.520758 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.511033 2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 08:46:18.520758 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.511036 2572 feature_gate.go:328] unrecognized feature gate: Example Sep 11 08:46:18.520758 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.511038 2572 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 08:46:18.520758 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.511041 2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 08:46:18.520758 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.511044 2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 08:46:18.520758 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.511050 2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 08:46:18.520758 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.511053 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 08:46:18.520758 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.511055 2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 08:46:18.520758 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.511058 2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 08:46:18.520758 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.511060 2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 08:46:18.520758 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.511063 2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 08:46:18.520758 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.511065 2572 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 08:46:18.521246 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.511068 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 08:46:18.521246 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.511070 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 08:46:18.521246 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.511073 2572 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 08:46:18.521246 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.511075 2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 08:46:18.521246 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.511078 2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 08:46:18.521246 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.511080 2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 08:46:18.521246 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.511083 2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 08:46:18.521246 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.511085 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 08:46:18.521246 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.511087 2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 08:46:18.521246 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.511090 2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 08:46:18.521246 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.511093 2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 08:46:18.521246 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.511095 2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 08:46:18.521246 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.511097 2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 08:46:18.521246 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.511100 2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 08:46:18.521246 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.511102 2572 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 08:46:18.521246 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.511105 2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 08:46:18.521246 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.511107 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 08:46:18.521246 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.511109 2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 08:46:18.521246 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.511112 2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 08:46:18.521246 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.511116 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 08:46:18.521760 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.511118 2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 08:46:18.521760 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.511121 2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 08:46:18.521760 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.511126 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 08:46:18.521760 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.511129 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 08:46:18.521760 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.511878 2572 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 11 08:46:18.521760 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.521157 2572 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 11 08:46:18.521760 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.521172 2572 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 11 08:46:18.521760 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.521189 2572 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 11 08:46:18.521760 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.521195 2572 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 11 08:46:18.521760 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521251 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 08:46:18.521760 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521257 2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 08:46:18.521760 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521261 2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 08:46:18.521760 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521266 2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 08:46:18.521760 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521269 2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 08:46:18.522115 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521271 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 08:46:18.522115 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521274 2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 08:46:18.522115 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521277 2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 08:46:18.522115 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521280 2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 08:46:18.522115 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521283 2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 08:46:18.522115 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521285 2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 08:46:18.522115 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521288 2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 08:46:18.522115 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521290 2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 08:46:18.522115 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521293 2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 08:46:18.522115 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521296 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 08:46:18.522115 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521299 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 08:46:18.522115 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521301 2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 08:46:18.522115 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521304 2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 08:46:18.522115 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521306 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 08:46:18.522115 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521309 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 08:46:18.522115 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521312 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 08:46:18.522115 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521314 2572 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 08:46:18.522115 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521317 2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 08:46:18.522115 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521319 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 08:46:18.522670 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521322 2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 08:46:18.522670 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521332 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 08:46:18.522670 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521335 2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 08:46:18.522670 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521337 2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 08:46:18.522670 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521340 2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 08:46:18.522670 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521343 2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 08:46:18.522670 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521346 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 08:46:18.522670 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521349 2572 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 08:46:18.522670 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521351 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 08:46:18.522670 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521354 2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 08:46:18.522670 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521357 2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 08:46:18.522670 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521359 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 08:46:18.522670 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521362 2572 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 08:46:18.522670 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521364 2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 08:46:18.522670 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521367 2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 08:46:18.522670 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521369 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 08:46:18.522670 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521372 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 08:46:18.522670 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521374 2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 08:46:18.522670 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521377 2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 08:46:18.522670 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521379 2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 08:46:18.523160 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521382 2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 08:46:18.523160 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521385 2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 08:46:18.523160 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521388 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 08:46:18.523160 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521390 2572 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 08:46:18.523160 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521393 2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 08:46:18.523160 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521395 2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 08:46:18.523160 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521397 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 08:46:18.523160 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521400 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 08:46:18.523160 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521402 2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 08:46:18.523160 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521405 2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 08:46:18.523160 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521407 2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 08:46:18.523160 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521410 2572 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 08:46:18.523160 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521412 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 08:46:18.523160 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521414 2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 08:46:18.523160 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521418 2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 08:46:18.523160 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521421 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 08:46:18.523160 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521425 2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 08:46:18.523160 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521430 2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 08:46:18.523160 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521433 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 08:46:18.523629 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521437 2572 feature_gate.go:328] unrecognized feature gate: Example Sep 11 08:46:18.523629 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521439 2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 08:46:18.523629 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521442 2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 08:46:18.523629 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521445 2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 08:46:18.523629 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521447 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 08:46:18.523629 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521450 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 08:46:18.523629 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521452 2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 08:46:18.523629 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521455 2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 08:46:18.523629 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521457 2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 08:46:18.523629 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521460 2572 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 08:46:18.523629 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521462 2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 08:46:18.523629 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521465 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 08:46:18.523629 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521467 2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 08:46:18.523629 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521470 2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 08:46:18.523629 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521473 2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 08:46:18.523629 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521475 2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 08:46:18.523629 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521478 2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 08:46:18.523629 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521481 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 08:46:18.523629 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521483 2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 08:46:18.523629 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521486 2572 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 08:46:18.524109 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521488 2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 08:46:18.524109 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521491 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 08:46:18.524109 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521493 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 08:46:18.524109 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521496 2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 08:46:18.524109 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521498 2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 08:46:18.524109 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521500 2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 08:46:18.524109 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521503 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 08:46:18.524109 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521506 2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 08:46:18.524109 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521508 2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 08:46:18.524109 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521511 2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 08:46:18.524109 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521514 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 08:46:18.524109 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521516 2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 08:46:18.524109 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521519 2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 08:46:18.524109 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521521 2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 08:46:18.524109 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521524 2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 08:46:18.524109 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521526 2572 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 08:46:18.524496 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.521531 2572 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 11 08:46:18.524496 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521670 2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 08:46:18.524496 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521676 2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 08:46:18.524496 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521679 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 08:46:18.524496 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521682 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 08:46:18.524496 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521685 2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 08:46:18.524496 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521688 2572 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 08:46:18.524496 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521691 2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 08:46:18.524496 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521693 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 08:46:18.524496 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521696 2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 08:46:18.524496 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521698 2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 08:46:18.524496 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521701 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 08:46:18.524496 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521703 2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 08:46:18.524496 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521707 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 08:46:18.524496 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521710 2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 08:46:18.524870 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521713 2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 08:46:18.524870 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521715 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 08:46:18.524870 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521717 2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 08:46:18.524870 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521720 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 08:46:18.524870 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521722 2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 08:46:18.524870 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521725 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 08:46:18.524870 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521727 2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 08:46:18.524870 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521730 2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 08:46:18.524870 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521732 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 08:46:18.524870 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521735 2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 08:46:18.524870 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521737 2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 08:46:18.524870 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521740 2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 08:46:18.524870 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521743 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 08:46:18.524870 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521745 2572 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 08:46:18.524870 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521748 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 08:46:18.524870 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521750 2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 08:46:18.524870 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521752 2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 08:46:18.524870 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521755 2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 08:46:18.524870 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521757 2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 08:46:18.524870 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521760 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 08:46:18.525350 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521762 2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 08:46:18.525350 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521766 2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 08:46:18.525350 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521770 2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 08:46:18.525350 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521772 2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 08:46:18.525350 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521775 2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 08:46:18.525350 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521778 2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 08:46:18.525350 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521780 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 08:46:18.525350 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521783 2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 08:46:18.525350 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521785 2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 08:46:18.525350 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521788 2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 08:46:18.525350 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521792 2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 08:46:18.525350 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521799 2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 08:46:18.525350 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521802 2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 08:46:18.525350 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521805 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 08:46:18.525350 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521808 2572 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 08:46:18.525350 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521810 2572 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 08:46:18.525350 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521813 2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 08:46:18.525350 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521815 2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 08:46:18.525350 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521817 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 08:46:18.525833 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521820 2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 08:46:18.525833 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521823 2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 08:46:18.525833 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521826 2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 08:46:18.525833 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521829 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 08:46:18.525833 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521832 2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 08:46:18.525833 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521835 2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 08:46:18.525833 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521838 2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 08:46:18.525833 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521841 2572 feature_gate.go:328] unrecognized feature gate: Example Sep 11 08:46:18.525833 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521843 2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 08:46:18.525833 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521846 2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 08:46:18.525833 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521848 2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 08:46:18.525833 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521851 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 08:46:18.525833 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521853 2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 08:46:18.525833 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521855 2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 08:46:18.525833 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521858 2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 08:46:18.525833 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521860 2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 08:46:18.525833 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521863 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 08:46:18.525833 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521865 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 08:46:18.525833 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521867 2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 08:46:18.526287 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521870 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 08:46:18.526287 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521872 2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 08:46:18.526287 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521875 2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 08:46:18.526287 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521878 2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 08:46:18.526287 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521880 2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 08:46:18.526287 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521882 2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 08:46:18.526287 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521887 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 08:46:18.526287 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521890 2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 08:46:18.526287 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521892 2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 08:46:18.526287 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521894 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 08:46:18.526287 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521897 2572 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 08:46:18.526287 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521899 2572 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 08:46:18.526287 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521902 2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 08:46:18.526287 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521904 2572 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 08:46:18.526287 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521907 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 08:46:18.526287 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521909 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 08:46:18.526287 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521912 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 08:46:18.526287 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521914 2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 08:46:18.526287 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521917 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 08:46:18.526287 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521920 2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 08:46:18.526287 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521922 2572 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 08:46:18.526850 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521924 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 08:46:18.526850 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521927 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 08:46:18.526850 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521929 2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 08:46:18.526850 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521932 2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 08:46:18.526850 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521934 2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 08:46:18.526850 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:18.521936 2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 08:46:18.526850 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.521941 2572 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 11 08:46:18.526850 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.522043 2572 server.go:962] "Client rotation is on, will bootstrap in background" Sep 11 08:46:18.526850 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.524435 2572 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 11 08:46:18.526850 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.525556 2572 server.go:1019] "Starting client certificate rotation" Sep 11 08:46:18.526850 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.525622 2572 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 11 08:46:18.526850 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.525668 2572 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 11 08:46:18.528694 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.528680 2572 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 11 08:46:18.529904 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.529887 2572 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 11 08:46:18.543001 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.542980 2572 log.go:25] "Validated CRI v1 runtime API" Sep 11 08:46:18.549255 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.549240 2572 log.go:25] "Validated CRI v1 image API" Sep 11 08:46:18.550141 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.550128 2572 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 11 08:46:18.555409 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.555386 2572 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 7e62ad12-d574-469e-a357-37dd3630d488:/dev/nvme0n1p3 91fd4615-4ea9-4d8f-b3d8-b93348c3afbd:/dev/nvme0n1p4] Sep 11 08:46:18.555464 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.555410 2572 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 11 08:46:18.562747 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.562633 2572 manager.go:216] Machine: {Timestamp:2026-09-11 08:46:18.561016795 +0000 UTC m=+0.381230618 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100745 MemoryCapacity:32812150784 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec25a050c1d1545aa540a4393ebf98db SystemUUID:ec25a050-c1d1-545a-a540-a4393ebf98db BootID:6e57e30e-4096-4e4a-a6c0-ba868aae0c37 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406073344 Type:vfs Inodes:4005389 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562430976 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406077440 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:1a:40:2f:66:67 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:1a:40:2f:66:67 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:2e:38:a2:ae:96:06 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812150784 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 11 08:46:18.562747 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.562736 2572 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 11 08:46:18.562863 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.562842 2572 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 11 08:46:18.564380 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.564364 2572 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 11 08:46:18.566255 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.566231 2572 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 11 08:46:18.566408 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.566258 2572 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-166.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 11 08:46:18.567190 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.567168 2572 topology_manager.go:138] "Creating topology manager with none policy" Sep 11 08:46:18.567190 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.567193 2572 container_manager_linux.go:309] "Creating device plugin manager" Sep 11 08:46:18.567262 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.567201 2572 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 11 08:46:18.567262 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.567214 2572 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 11 08:46:18.568507 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.568497 2572 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 11 08:46:18.573179 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.573167 2572 state_mem.go:36] "Initialized new in-memory state store" Sep 11 08:46:18.573287 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.573277 2572 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 11 08:46:18.573317 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.573308 2572 kubelet.go:486] "Attempting to sync node with API server" Sep 11 08:46:18.573347 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.573321 2572 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 11 08:46:18.573347 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.573335 2572 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 11 08:46:18.573347 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.573345 2572 kubelet.go:398] "Adding apiserver pod source" Sep 11 08:46:18.573477 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.573360 2572 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 11 08:46:18.573756 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.573745 2572 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 11 08:46:18.573814 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.573764 2572 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 11 08:46:18.576016 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.575994 2572 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 11 08:46:18.578342 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.578328 2572 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 11 08:46:18.578393 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.578359 2572 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 11 08:46:18.580378 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.580366 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 11 08:46:18.580426 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.580384 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 11 08:46:18.580426 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.580390 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 11 08:46:18.580426 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.580395 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 11 08:46:18.580426 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.580403 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 11 08:46:18.580426 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.580408 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 11 08:46:18.580426 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.580421 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 11 08:46:18.580426 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.580426 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 11 08:46:18.580617 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.580432 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 11 08:46:18.580617 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.580437 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 11 08:46:18.580617 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.580446 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 11 08:46:18.580617 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.580451 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 11 08:46:18.581372 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.581362 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 11 08:46:18.581372 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.581372 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 11 08:46:18.585106 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.585089 2572 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-166.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 11 08:46:18.585368 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.585349 2572 server.go:1268] "Started kubelet" Sep 11 08:46:18.585464 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.585407 2572 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 11 08:46:18.585720 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.585665 2572 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 11 08:46:18.585887 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.585859 2572 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 11 08:46:18.585955 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:18.585712 2572 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-166.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 11 08:46:18.585955 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:18.585707 2572 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 11 08:46:18.586095 ip-10-0-128-166 systemd[1]: Started Kubernetes Kubelet. Sep 11 08:46:18.588201 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.588186 2572 server.go:310] "Adding debug handlers to kubelet server" Sep 11 08:46:18.588996 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.588978 2572 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 11 08:46:18.592158 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:18.591069 2572 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-166.ec2.internal.18d437ddff695b4f default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-166.ec2.internal,UID:ip-10-0-128-166.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-166.ec2.internal,},FirstTimestamp:2026-09-11 08:46:18.585324367 +0000 UTC m=+0.405538189,LastTimestamp:2026-09-11 08:46:18.585324367 +0000 UTC m=+0.405538189,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-166.ec2.internal,}" Sep 11 08:46:18.593032 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:18.593016 2572 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 11 08:46:18.594556 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.594538 2572 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 11 08:46:18.594556 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.594551 2572 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 11 08:46:18.594706 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:18.594665 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-166.ec2.internal\" not found" Sep 11 08:46:18.595312 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.595296 2572 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 11 08:46:18.595392 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.595359 2572 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 11 08:46:18.595392 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.595376 2572 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 11 08:46:18.595624 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.595570 2572 reconstruct.go:97] "Volume reconstruction finished" Sep 11 08:46:18.595624 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.595612 2572 reconciler.go:29] "Reconciler: start to sync state" Sep 11 08:46:18.596080 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.595880 2572 factory.go:55] Registering systemd factory Sep 11 08:46:18.596080 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.595894 2572 factory.go:223] Registration of the systemd container factory successfully Sep 11 08:46:18.596080 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.596068 2572 factory.go:153] Registering CRI-O factory Sep 11 08:46:18.596326 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.596085 2572 factory.go:223] Registration of the crio container factory successfully Sep 11 08:46:18.596326 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.596138 2572 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 11 08:46:18.596326 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.596158 2572 factory.go:103] Registering Raw factory Sep 11 08:46:18.596326 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.596173 2572 manager.go:1196] Started watching for new ooms in manager Sep 11 08:46:18.596645 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.596631 2572 manager.go:349] Starting recovery of all containers Sep 11 08:46:18.600450 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:18.600419 2572 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-128-166.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 11 08:46:18.600577 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:18.600550 2572 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 11 08:46:18.607724 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.607709 2572 manager.go:354] Recovery completed Sep 11 08:46:18.611488 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.611476 2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 08:46:18.613887 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.613870 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-166.ec2.internal" event="NodeHasSufficientMemory" Sep 11 08:46:18.613969 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.613897 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-166.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 08:46:18.613969 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.613918 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-166.ec2.internal" event="NodeHasSufficientPID" Sep 11 08:46:18.614178 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.614151 2572 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6gpkw" Sep 11 08:46:18.614446 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.614431 2572 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 11 08:46:18.614446 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.614441 2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 08:46:18.614580 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.614442 2572 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 11 08:46:18.614580 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.614488 2572 state_mem.go:36] "Initialized new in-memory state store" Sep 11 08:46:18.615164 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.615151 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-166.ec2.internal" event="NodeHasSufficientMemory" Sep 11 08:46:18.615229 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.615174 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-166.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 08:46:18.615229 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.615183 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-166.ec2.internal" event="NodeHasSufficientPID" Sep 11 08:46:18.616109 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:18.616048 2572 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-166.ec2.internal.18d437de011d2a3f default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-166.ec2.internal,UID:ip-10-0-128-166.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-128-166.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-128-166.ec2.internal,},FirstTimestamp:2026-09-11 08:46:18.613885503 +0000 UTC m=+0.434099324,LastTimestamp:2026-09-11 08:46:18.613885503 +0000 UTC m=+0.434099324,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-166.ec2.internal,}" Sep 11 08:46:18.616672 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.616660 2572 policy_none.go:49] "None policy: Start" Sep 11 08:46:18.616709 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.616675 2572 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 11 08:46:18.616709 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.616685 2572 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 11 08:46:18.618615 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.618601 2572 policy_none.go:47] "Start" Sep 11 08:46:18.621731 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.621712 2572 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6gpkw" Sep 11 08:46:18.622767 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:18.622681 2572 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-166.ec2.internal.18d437de011d6e23 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-166.ec2.internal,UID:ip-10-0-128-166.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-128-166.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-128-166.ec2.internal,},FirstTimestamp:2026-09-11 08:46:18.613902883 +0000 UTC m=+0.434116705,LastTimestamp:2026-09-11 08:46:18.613902883 +0000 UTC m=+0.434116705,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-166.ec2.internal,}" Sep 11 08:46:18.624507 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:18.624451 2572 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-166.ec2.internal.18d437de011db7f0 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-166.ec2.internal,UID:ip-10-0-128-166.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-128-166.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-128-166.ec2.internal,},FirstTimestamp:2026-09-11 08:46:18.613921776 +0000 UTC m=+0.434135599,LastTimestamp:2026-09-11 08:46:18.613921776 +0000 UTC m=+0.434135599,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-166.ec2.internal,}" Sep 11 08:46:18.650157 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.650143 2572 manager.go:337] "Starting Device Plugin manager" Sep 11 08:46:18.650253 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:18.650192 2572 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 11 08:46:18.650253 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.650205 2572 server.go:87] "Starting device plugin registration server" Sep 11 08:46:18.650446 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.650432 2572 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 11 08:46:18.650502 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.650448 2572 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 11 08:46:18.650555 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.650538 2572 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 11 08:46:18.650668 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.650633 2572 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 11 08:46:18.650668 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.650642 2572 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 11 08:46:18.651179 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:18.651118 2572 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 11 08:46:18.651179 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:18.651165 2572 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-166.ec2.internal\" not found" Sep 11 08:46:18.699392 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.699358 2572 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 11 08:46:18.701672 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.700644 2572 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 11 08:46:18.701672 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.700683 2572 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 11 08:46:18.701672 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.700701 2572 kubelet.go:2448] "Starting kubelet main sync loop" Sep 11 08:46:18.701672 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:18.700733 2572 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 11 08:46:18.703286 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.703267 2572 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 08:46:18.751299 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.751277 2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 08:46:18.752123 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.752067 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-166.ec2.internal" event="NodeHasSufficientMemory" Sep 11 08:46:18.752123 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.752102 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-166.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 08:46:18.752123 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.752119 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-166.ec2.internal" event="NodeHasSufficientPID" Sep 11 08:46:18.752303 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.752149 2572 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-128-166.ec2.internal" Sep 11 08:46:18.761543 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.761527 2572 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-128-166.ec2.internal" Sep 11 08:46:18.761661 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:18.761551 2572 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-166.ec2.internal\": node \"ip-10-0-128-166.ec2.internal\" not found" Sep 11 08:46:18.788181 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:18.788158 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-166.ec2.internal\" not found" Sep 11 08:46:18.801581 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.801555 2572 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-166.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-166.ec2.internal"] Sep 11 08:46:18.801685 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.801635 2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 08:46:18.803016 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.803000 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-166.ec2.internal" event="NodeHasSufficientMemory" Sep 11 08:46:18.803101 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.803028 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-166.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 08:46:18.803101 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.803037 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-166.ec2.internal" event="NodeHasSufficientPID" Sep 11 08:46:18.805352 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.805340 2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 08:46:18.805486 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.805473 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-166.ec2.internal" Sep 11 08:46:18.805525 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.805509 2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 08:46:18.806130 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.806115 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-166.ec2.internal" event="NodeHasSufficientMemory" Sep 11 08:46:18.806244 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.806140 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-166.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 08:46:18.806244 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.806116 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-166.ec2.internal" event="NodeHasSufficientMemory" Sep 11 08:46:18.806244 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.806175 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-166.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 08:46:18.806244 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.806186 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-166.ec2.internal" event="NodeHasSufficientPID" Sep 11 08:46:18.806244 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.806151 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-166.ec2.internal" event="NodeHasSufficientPID" Sep 11 08:46:18.808434 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.808417 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-166.ec2.internal" Sep 11 08:46:18.808492 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.808448 2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 08:46:18.809170 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.809156 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-166.ec2.internal" event="NodeHasSufficientMemory" Sep 11 08:46:18.809253 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.809180 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-166.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 08:46:18.809253 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.809189 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-166.ec2.internal" event="NodeHasSufficientPID" Sep 11 08:46:18.835191 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:18.835165 2572 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-166.ec2.internal\" not found" node="ip-10-0-128-166.ec2.internal" Sep 11 08:46:18.839476 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:18.839459 2572 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-166.ec2.internal\" not found" node="ip-10-0-128-166.ec2.internal" Sep 11 08:46:18.889126 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:18.889100 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-166.ec2.internal\" not found" Sep 11 08:46:18.898295 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.898275 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3e81cc2e86491c3258cb0c986cc8b166-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-166.ec2.internal\" (UID: \"3e81cc2e86491c3258cb0c986cc8b166\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-166.ec2.internal" Sep 11 08:46:18.898368 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.898300 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1f3dec62f93374a659aa01a42bf55935-config\") pod \"kube-apiserver-proxy-ip-10-0-128-166.ec2.internal\" (UID: \"1f3dec62f93374a659aa01a42bf55935\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-166.ec2.internal" Sep 11 08:46:18.898368 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.898322 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3e81cc2e86491c3258cb0c986cc8b166-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-166.ec2.internal\" (UID: \"3e81cc2e86491c3258cb0c986cc8b166\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-166.ec2.internal" Sep 11 08:46:18.989905 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:18.989871 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-166.ec2.internal\" not found" Sep 11 08:46:18.999286 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.999264 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3e81cc2e86491c3258cb0c986cc8b166-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-166.ec2.internal\" (UID: \"3e81cc2e86491c3258cb0c986cc8b166\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-166.ec2.internal" Sep 11 08:46:18.999347 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.999296 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3e81cc2e86491c3258cb0c986cc8b166-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-166.ec2.internal\" (UID: \"3e81cc2e86491c3258cb0c986cc8b166\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-166.ec2.internal" Sep 11 08:46:18.999347 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.999315 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1f3dec62f93374a659aa01a42bf55935-config\") pod \"kube-apiserver-proxy-ip-10-0-128-166.ec2.internal\" (UID: \"1f3dec62f93374a659aa01a42bf55935\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-166.ec2.internal" Sep 11 08:46:18.999347 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.999342 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1f3dec62f93374a659aa01a42bf55935-config\") pod \"kube-apiserver-proxy-ip-10-0-128-166.ec2.internal\" (UID: \"1f3dec62f93374a659aa01a42bf55935\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-166.ec2.internal" Sep 11 08:46:18.999442 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.999368 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3e81cc2e86491c3258cb0c986cc8b166-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-166.ec2.internal\" (UID: \"3e81cc2e86491c3258cb0c986cc8b166\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-166.ec2.internal" Sep 11 08:46:18.999442 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:18.999360 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3e81cc2e86491c3258cb0c986cc8b166-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-166.ec2.internal\" (UID: \"3e81cc2e86491c3258cb0c986cc8b166\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-166.ec2.internal" Sep 11 08:46:19.090762 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:19.090688 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-166.ec2.internal\" not found" Sep 11 08:46:19.137220 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:19.137201 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-166.ec2.internal" Sep 11 08:46:19.142728 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:19.142399 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-166.ec2.internal" Sep 11 08:46:19.191024 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:19.190999 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-166.ec2.internal\" not found" Sep 11 08:46:19.291692 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:19.291508 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-166.ec2.internal\" not found" Sep 11 08:46:19.392012 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:19.391989 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-166.ec2.internal\" not found" Sep 11 08:46:19.492556 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:19.492529 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-166.ec2.internal\" not found" Sep 11 08:46:19.524997 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:19.524977 2572 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 11 08:46:19.525398 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:19.525110 2572 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 11 08:46:19.557157 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:19.557132 2572 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 08:46:19.592670 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:19.592642 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-166.ec2.internal\" not found" Sep 11 08:46:19.627828 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:19.627795 2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-10 08:41:20 +0000 UTC" deadline="2028-04-08 10:08:38.220020265 +0000 UTC" Sep 11 08:46:19.627828 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:19.627824 2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13801h22m18.592199366s" Sep 11 08:46:19.672609 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:19.672540 2572 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 08:46:19.695119 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:19.695094 2572 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-166.ec2.internal" Sep 11 08:46:19.708214 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:19.708192 2572 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 11 08:46:19.709190 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:19.709177 2572 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-166.ec2.internal" Sep 11 08:46:19.717855 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:19.717831 2572 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 11 08:46:19.743245 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:19.743207 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f3dec62f93374a659aa01a42bf55935.slice/crio-17b8b7c8d6bd5f07c1a217c9ec23f4f6d2dc87700eea41847cea371a98a445bf WatchSource:0}: Error finding container 17b8b7c8d6bd5f07c1a217c9ec23f4f6d2dc87700eea41847cea371a98a445bf: Status 404 returned error can't find the container with id 17b8b7c8d6bd5f07c1a217c9ec23f4f6d2dc87700eea41847cea371a98a445bf Sep 11 08:46:19.747935 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:19.747920 2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 08:46:21.404009 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:21.403979 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e81cc2e86491c3258cb0c986cc8b166.slice/crio-4fc5407d9706ae747265ac60f2e83ca6fbcfacf9c4a780357c5f94704b898210 WatchSource:0}: Error finding container 4fc5407d9706ae747265ac60f2e83ca6fbcfacf9c4a780357c5f94704b898210: Status 404 returned error can't find the container with id 4fc5407d9706ae747265ac60f2e83ca6fbcfacf9c4a780357c5f94704b898210 Sep 11 08:46:21.458052 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:21.458029 2572 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 08:46:22.119983 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.119949 2572 apiserver.go:52] "Watching apiserver" Sep 11 08:46:22.124516 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.124492 2572 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 11 08:46:22.124872 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.124849 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-v7fx2","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7m87","openshift-image-registry/node-ca-67p7z","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-166.ec2.internal","openshift-multus/multus-additional-cni-plugins-k599z","openshift-network-diagnostics/network-check-target-82xp9","openshift-ovn-kubernetes/ovnkube-node-lhkkf","kube-system/konnectivity-agent-p5sg8","kube-system/kube-apiserver-proxy-ip-10-0-128-166.ec2.internal","openshift-cluster-node-tuning-operator/tuned-n56xz","openshift-dns/node-resolver-4vdzw","openshift-multus/multus-4w5tl","openshift-multus/network-metrics-daemon-ntkhj","openshift-network-operator/iptables-alerter-v5rw5"] Sep 11 08:46:22.127475 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.127448 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-n56xz" Sep 11 08:46:22.129670 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.129562 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ntkhj" Sep 11 08:46:22.129670 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.129617 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 11 08:46:22.129670 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:22.129631 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ntkhj" podUID="e78a69db-9d98-4923-b082-ccb30b77cc9e" Sep 11 08:46:22.129903 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.129885 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-6hp2t\"" Sep 11 08:46:22.129963 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.129953 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 11 08:46:22.132134 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.132112 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-67p7z" Sep 11 08:46:22.133957 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.133932 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-v5rw5" Sep 11 08:46:22.134066 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.133975 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-5j6q6\"" Sep 11 08:46:22.134066 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.134017 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 11 08:46:22.134066 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.134037 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 11 08:46:22.134261 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.134173 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 11 08:46:22.135839 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.135571 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 11 08:46:22.136206 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.136186 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 11 08:46:22.136314 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.136244 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 11 08:46:22.136415 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.136401 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-k599z" Sep 11 08:46:22.136483 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.136436 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-99s78\"" Sep 11 08:46:22.137925 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.137907 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 11 08:46:22.138022 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.137969 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 11 08:46:22.138080 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.138026 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 11 08:46:22.138257 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.138241 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 11 08:46:22.138385 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.138363 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-j56cv\"" Sep 11 08:46:22.138439 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.138397 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 11 08:46:22.138519 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.138493 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-82xp9" Sep 11 08:46:22.138625 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:22.138558 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-82xp9" podUID="12ac768c-2c6a-43e7-9bd3-44679781ebd7" Sep 11 08:46:22.140262 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.140243 2572 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 08:46:22.140624 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.140604 2572 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 11 08:46:22.141332 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.141301 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-p5sg8" Sep 11 08:46:22.144860 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.144839 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-lhkkf" Sep 11 08:46:22.145340 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.145327 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-shj8d\"" Sep 11 08:46:22.145543 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.145531 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 11 08:46:22.145746 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.145714 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 11 08:46:22.146858 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.146836 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7m87" Sep 11 08:46:22.148423 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.148052 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-96w8j\"" Sep 11 08:46:22.148423 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.148112 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 11 08:46:22.148423 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.148192 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 11 08:46:22.148423 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.148211 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 11 08:46:22.148423 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.148264 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 11 08:46:22.148423 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.148057 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 11 08:46:22.148764 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.148747 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 11 08:46:22.149813 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.149798 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v7fx2" Sep 11 08:46:22.149901 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:22.149854 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v7fx2" podUID="a370ef3a-98cb-4251-86b9-cb3155c587a5" Sep 11 08:46:22.151726 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.151660 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 11 08:46:22.151876 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.151859 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-77ttq\"" Sep 11 08:46:22.152067 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.152052 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 11 08:46:22.152300 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.152283 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-4vdzw" Sep 11 08:46:22.152394 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.152289 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcc27\" (UniqueName: \"kubernetes.io/projected/eb6eba26-53e8-4946-9491-cf154a8286f5-kube-api-access-lcc27\") pod \"node-ca-67p7z\" (UID: \"eb6eba26-53e8-4946-9491-cf154a8286f5\") " pod="openshift-image-registry/node-ca-67p7z" Sep 11 08:46:22.152394 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.152358 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/83409be8-1bb2-43dc-af76-d3d74e569101-systemd-units\") pod \"ovnkube-node-lhkkf\" (UID: \"83409be8-1bb2-43dc-af76-d3d74e569101\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhkkf" Sep 11 08:46:22.152510 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.152417 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/83409be8-1bb2-43dc-af76-d3d74e569101-run-openvswitch\") pod \"ovnkube-node-lhkkf\" (UID: \"83409be8-1bb2-43dc-af76-d3d74e569101\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhkkf" Sep 11 08:46:22.152510 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.152475 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/83409be8-1bb2-43dc-af76-d3d74e569101-log-socket\") pod \"ovnkube-node-lhkkf\" (UID: \"83409be8-1bb2-43dc-af76-d3d74e569101\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhkkf" Sep 11 08:46:22.152510 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.152500 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/83409be8-1bb2-43dc-af76-d3d74e569101-host-cni-netd\") pod \"ovnkube-node-lhkkf\" (UID: \"83409be8-1bb2-43dc-af76-d3d74e569101\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhkkf" Sep 11 08:46:22.152655 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.152521 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/46676d4c-0012-4617-9fe5-73d33015eb8c-host\") pod \"tuned-n56xz\" (UID: \"46676d4c-0012-4617-9fe5-73d33015eb8c\") " pod="openshift-cluster-node-tuning-operator/tuned-n56xz" Sep 11 08:46:22.152655 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.152544 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/83409be8-1bb2-43dc-af76-d3d74e569101-run-systemd\") pod \"ovnkube-node-lhkkf\" (UID: \"83409be8-1bb2-43dc-af76-d3d74e569101\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhkkf" Sep 11 08:46:22.152655 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.152566 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/83409be8-1bb2-43dc-af76-d3d74e569101-node-log\") pod \"ovnkube-node-lhkkf\" (UID: \"83409be8-1bb2-43dc-af76-d3d74e569101\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhkkf" Sep 11 08:46:22.152655 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.152588 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e78a69db-9d98-4923-b082-ccb30b77cc9e-metrics-certs\") pod \"network-metrics-daemon-ntkhj\" (UID: \"e78a69db-9d98-4923-b082-ccb30b77cc9e\") " pod="openshift-multus/network-metrics-daemon-ntkhj" Sep 11 08:46:22.152655 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.152613 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/510eab66-1dcf-4ffc-983d-d570fb9e9fda-iptables-alerter-script\") pod \"iptables-alerter-v5rw5\" (UID: \"510eab66-1dcf-4ffc-983d-d570fb9e9fda\") " pod="openshift-network-operator/iptables-alerter-v5rw5" Sep 11 08:46:22.152655 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.152636 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0c0c64bd-73f4-4cb4-b660-c7232ff8e573-system-cni-dir\") pod \"multus-additional-cni-plugins-k599z\" (UID: \"0c0c64bd-73f4-4cb4-b660-c7232ff8e573\") " pod="openshift-multus/multus-additional-cni-plugins-k599z" Sep 11 08:46:22.152927 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.152672 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0c0c64bd-73f4-4cb4-b660-c7232ff8e573-tuning-conf-dir\") pod \"multus-additional-cni-plugins-k599z\" (UID: \"0c0c64bd-73f4-4cb4-b660-c7232ff8e573\") " pod="openshift-multus/multus-additional-cni-plugins-k599z" Sep 11 08:46:22.152927 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.152696 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0c0c64bd-73f4-4cb4-b660-c7232ff8e573-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-k599z\" (UID: \"0c0c64bd-73f4-4cb4-b660-c7232ff8e573\") " pod="openshift-multus/multus-additional-cni-plugins-k599z" Sep 11 08:46:22.152927 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.152722 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/46676d4c-0012-4617-9fe5-73d33015eb8c-etc-modprobe-d\") pod \"tuned-n56xz\" (UID: \"46676d4c-0012-4617-9fe5-73d33015eb8c\") " pod="openshift-cluster-node-tuning-operator/tuned-n56xz" Sep 11 08:46:22.152927 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.152744 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/46676d4c-0012-4617-9fe5-73d33015eb8c-etc-sysconfig\") pod \"tuned-n56xz\" (UID: \"46676d4c-0012-4617-9fe5-73d33015eb8c\") " pod="openshift-cluster-node-tuning-operator/tuned-n56xz" Sep 11 08:46:22.152927 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.152787 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/46676d4c-0012-4617-9fe5-73d33015eb8c-etc-kubernetes\") pod \"tuned-n56xz\" (UID: \"46676d4c-0012-4617-9fe5-73d33015eb8c\") " pod="openshift-cluster-node-tuning-operator/tuned-n56xz" Sep 11 08:46:22.152927 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.152815 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/46676d4c-0012-4617-9fe5-73d33015eb8c-var-lib-kubelet\") pod \"tuned-n56xz\" (UID: \"46676d4c-0012-4617-9fe5-73d33015eb8c\") " pod="openshift-cluster-node-tuning-operator/tuned-n56xz" Sep 11 08:46:22.152927 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.152838 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpcqf\" (UniqueName: \"kubernetes.io/projected/83409be8-1bb2-43dc-af76-d3d74e569101-kube-api-access-rpcqf\") pod \"ovnkube-node-lhkkf\" (UID: \"83409be8-1bb2-43dc-af76-d3d74e569101\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhkkf" Sep 11 08:46:22.152927 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.152890 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9b7p\" (UniqueName: \"kubernetes.io/projected/e78a69db-9d98-4923-b082-ccb30b77cc9e-kube-api-access-v9b7p\") pod \"network-metrics-daemon-ntkhj\" (UID: \"e78a69db-9d98-4923-b082-ccb30b77cc9e\") " pod="openshift-multus/network-metrics-daemon-ntkhj" Sep 11 08:46:22.152927 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.152915 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clk2v\" (UniqueName: \"kubernetes.io/projected/46676d4c-0012-4617-9fe5-73d33015eb8c-kube-api-access-clk2v\") pod \"tuned-n56xz\" (UID: \"46676d4c-0012-4617-9fe5-73d33015eb8c\") " pod="openshift-cluster-node-tuning-operator/tuned-n56xz" Sep 11 08:46:22.152927 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.152930 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/eb6eba26-53e8-4946-9491-cf154a8286f5-host\") pod \"node-ca-67p7z\" (UID: \"eb6eba26-53e8-4946-9491-cf154a8286f5\") " pod="openshift-image-registry/node-ca-67p7z" Sep 11 08:46:22.153455 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.152946 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/83409be8-1bb2-43dc-af76-d3d74e569101-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lhkkf\" (UID: \"83409be8-1bb2-43dc-af76-d3d74e569101\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhkkf" Sep 11 08:46:22.153455 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.152969 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/83409be8-1bb2-43dc-af76-d3d74e569101-ovnkube-script-lib\") pod \"ovnkube-node-lhkkf\" (UID: \"83409be8-1bb2-43dc-af76-d3d74e569101\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhkkf" Sep 11 08:46:22.153455 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.153024 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lx99\" (UniqueName: \"kubernetes.io/projected/510eab66-1dcf-4ffc-983d-d570fb9e9fda-kube-api-access-6lx99\") pod \"iptables-alerter-v5rw5\" (UID: \"510eab66-1dcf-4ffc-983d-d570fb9e9fda\") " pod="openshift-network-operator/iptables-alerter-v5rw5" Sep 11 08:46:22.153455 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.153052 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0c0c64bd-73f4-4cb4-b660-c7232ff8e573-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-k599z\" (UID: \"0c0c64bd-73f4-4cb4-b660-c7232ff8e573\") " pod="openshift-multus/multus-additional-cni-plugins-k599z" Sep 11 08:46:22.153455 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.153079 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2c0c8738-0dd8-441e-8c6e-3ff6009337ca-agent-certs\") pod \"konnectivity-agent-p5sg8\" (UID: \"2c0c8738-0dd8-441e-8c6e-3ff6009337ca\") " pod="kube-system/konnectivity-agent-p5sg8" Sep 11 08:46:22.153455 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.153106 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2c0c8738-0dd8-441e-8c6e-3ff6009337ca-konnectivity-ca\") pod \"konnectivity-agent-p5sg8\" (UID: \"2c0c8738-0dd8-441e-8c6e-3ff6009337ca\") " pod="kube-system/konnectivity-agent-p5sg8" Sep 11 08:46:22.153455 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.153119 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 11 08:46:22.153455 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.153157 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/46676d4c-0012-4617-9fe5-73d33015eb8c-etc-sysctl-d\") pod \"tuned-n56xz\" (UID: \"46676d4c-0012-4617-9fe5-73d33015eb8c\") " pod="openshift-cluster-node-tuning-operator/tuned-n56xz" Sep 11 08:46:22.153455 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.153196 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/46676d4c-0012-4617-9fe5-73d33015eb8c-etc-systemd\") pod \"tuned-n56xz\" (UID: \"46676d4c-0012-4617-9fe5-73d33015eb8c\") " pod="openshift-cluster-node-tuning-operator/tuned-n56xz" Sep 11 08:46:22.153455 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.153253 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/83409be8-1bb2-43dc-af76-d3d74e569101-var-lib-openvswitch\") pod \"ovnkube-node-lhkkf\" (UID: \"83409be8-1bb2-43dc-af76-d3d74e569101\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhkkf" Sep 11 08:46:22.153455 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.153281 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/83409be8-1bb2-43dc-af76-d3d74e569101-host-run-ovn-kubernetes\") pod \"ovnkube-node-lhkkf\" (UID: \"83409be8-1bb2-43dc-af76-d3d74e569101\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhkkf" Sep 11 08:46:22.153455 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.153303 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/83409be8-1bb2-43dc-af76-d3d74e569101-host-cni-bin\") pod \"ovnkube-node-lhkkf\" (UID: \"83409be8-1bb2-43dc-af76-d3d74e569101\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhkkf" Sep 11 08:46:22.153455 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.153325 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/83409be8-1bb2-43dc-af76-d3d74e569101-ovnkube-config\") pod \"ovnkube-node-lhkkf\" (UID: \"83409be8-1bb2-43dc-af76-d3d74e569101\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhkkf" Sep 11 08:46:22.153455 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.153346 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/83409be8-1bb2-43dc-af76-d3d74e569101-env-overrides\") pod \"ovnkube-node-lhkkf\" (UID: \"83409be8-1bb2-43dc-af76-d3d74e569101\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhkkf" Sep 11 08:46:22.153455 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.153364 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/83409be8-1bb2-43dc-af76-d3d74e569101-host-kubelet\") pod \"ovnkube-node-lhkkf\" (UID: \"83409be8-1bb2-43dc-af76-d3d74e569101\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhkkf" Sep 11 08:46:22.153455 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.153378 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/83409be8-1bb2-43dc-af76-d3d74e569101-host-run-netns\") pod \"ovnkube-node-lhkkf\" (UID: \"83409be8-1bb2-43dc-af76-d3d74e569101\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhkkf" Sep 11 08:46:22.153455 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.153396 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/83409be8-1bb2-43dc-af76-d3d74e569101-etc-openvswitch\") pod \"ovnkube-node-lhkkf\" (UID: \"83409be8-1bb2-43dc-af76-d3d74e569101\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhkkf" Sep 11 08:46:22.154211 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.153434 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/46676d4c-0012-4617-9fe5-73d33015eb8c-tmp\") pod \"tuned-n56xz\" (UID: \"46676d4c-0012-4617-9fe5-73d33015eb8c\") " pod="openshift-cluster-node-tuning-operator/tuned-n56xz" Sep 11 08:46:22.154211 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.153455 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/eb6eba26-53e8-4946-9491-cf154a8286f5-serviceca\") pod \"node-ca-67p7z\" (UID: \"eb6eba26-53e8-4946-9491-cf154a8286f5\") " pod="openshift-image-registry/node-ca-67p7z" Sep 11 08:46:22.154211 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.153473 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/83409be8-1bb2-43dc-af76-d3d74e569101-ovn-node-metrics-cert\") pod \"ovnkube-node-lhkkf\" (UID: \"83409be8-1bb2-43dc-af76-d3d74e569101\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhkkf" Sep 11 08:46:22.154211 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.153493 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/510eab66-1dcf-4ffc-983d-d570fb9e9fda-host-slash\") pod \"iptables-alerter-v5rw5\" (UID: \"510eab66-1dcf-4ffc-983d-d570fb9e9fda\") " pod="openshift-network-operator/iptables-alerter-v5rw5" Sep 11 08:46:22.154211 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.153526 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/46676d4c-0012-4617-9fe5-73d33015eb8c-etc-sysctl-conf\") pod \"tuned-n56xz\" (UID: \"46676d4c-0012-4617-9fe5-73d33015eb8c\") " pod="openshift-cluster-node-tuning-operator/tuned-n56xz" Sep 11 08:46:22.154211 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.153552 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/46676d4c-0012-4617-9fe5-73d33015eb8c-sys\") pod \"tuned-n56xz\" (UID: \"46676d4c-0012-4617-9fe5-73d33015eb8c\") " pod="openshift-cluster-node-tuning-operator/tuned-n56xz" Sep 11 08:46:22.154211 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.153574 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwc96\" (UniqueName: \"kubernetes.io/projected/12ac768c-2c6a-43e7-9bd3-44679781ebd7-kube-api-access-zwc96\") pod \"network-check-target-82xp9\" (UID: \"12ac768c-2c6a-43e7-9bd3-44679781ebd7\") " pod="openshift-network-diagnostics/network-check-target-82xp9" Sep 11 08:46:22.154211 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.153595 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/83409be8-1bb2-43dc-af76-d3d74e569101-host-slash\") pod \"ovnkube-node-lhkkf\" (UID: \"83409be8-1bb2-43dc-af76-d3d74e569101\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhkkf" Sep 11 08:46:22.154211 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.153616 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/83409be8-1bb2-43dc-af76-d3d74e569101-run-ovn\") pod \"ovnkube-node-lhkkf\" (UID: \"83409be8-1bb2-43dc-af76-d3d74e569101\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhkkf" Sep 11 08:46:22.154211 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.153638 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0c0c64bd-73f4-4cb4-b660-c7232ff8e573-cnibin\") pod \"multus-additional-cni-plugins-k599z\" (UID: \"0c0c64bd-73f4-4cb4-b660-c7232ff8e573\") " pod="openshift-multus/multus-additional-cni-plugins-k599z" Sep 11 08:46:22.154211 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.153660 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0c0c64bd-73f4-4cb4-b660-c7232ff8e573-cni-binary-copy\") pod \"multus-additional-cni-plugins-k599z\" (UID: \"0c0c64bd-73f4-4cb4-b660-c7232ff8e573\") " pod="openshift-multus/multus-additional-cni-plugins-k599z" Sep 11 08:46:22.154211 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.153684 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwpjw\" (UniqueName: \"kubernetes.io/projected/0c0c64bd-73f4-4cb4-b660-c7232ff8e573-kube-api-access-qwpjw\") pod \"multus-additional-cni-plugins-k599z\" (UID: \"0c0c64bd-73f4-4cb4-b660-c7232ff8e573\") " pod="openshift-multus/multus-additional-cni-plugins-k599z" Sep 11 08:46:22.154211 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.153704 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/46676d4c-0012-4617-9fe5-73d33015eb8c-run\") pod \"tuned-n56xz\" (UID: \"46676d4c-0012-4617-9fe5-73d33015eb8c\") " pod="openshift-cluster-node-tuning-operator/tuned-n56xz" Sep 11 08:46:22.154211 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.153731 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/46676d4c-0012-4617-9fe5-73d33015eb8c-lib-modules\") pod \"tuned-n56xz\" (UID: \"46676d4c-0012-4617-9fe5-73d33015eb8c\") " pod="openshift-cluster-node-tuning-operator/tuned-n56xz" Sep 11 08:46:22.154211 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.153763 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/46676d4c-0012-4617-9fe5-73d33015eb8c-etc-tuned\") pod \"tuned-n56xz\" (UID: \"46676d4c-0012-4617-9fe5-73d33015eb8c\") " pod="openshift-cluster-node-tuning-operator/tuned-n56xz" Sep 11 08:46:22.154944 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.154667 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 11 08:46:22.154944 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.154714 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 11 08:46:22.155043 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.154992 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-6dk6p\"" Sep 11 08:46:22.155356 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.155339 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4w5tl" Sep 11 08:46:22.158181 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.158032 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-x579w\"" Sep 11 08:46:22.158181 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.158072 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 11 08:46:22.160996 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.160977 2572 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 11 08:46:22.181026 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.181007 2572 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-bvrh5" Sep 11 08:46:22.191973 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.191952 2572 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-bvrh5" Sep 11 08:46:22.242113 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.242082 2572 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 11 08:46:22.251610 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.251562 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-166.ec2.internal" event={"ID":"3e81cc2e86491c3258cb0c986cc8b166","Type":"ContainerStarted","Data":"4fc5407d9706ae747265ac60f2e83ca6fbcfacf9c4a780357c5f94704b898210"} Sep 11 08:46:22.253278 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.253254 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-166.ec2.internal" event={"ID":"1f3dec62f93374a659aa01a42bf55935","Type":"ContainerStarted","Data":"17b8b7c8d6bd5f07c1a217c9ec23f4f6d2dc87700eea41847cea371a98a445bf"} Sep 11 08:46:22.254466 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.254440 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a370ef3a-98cb-4251-86b9-cb3155c587a5-original-pull-secret\") pod \"global-pull-secret-syncer-v7fx2\" (UID: \"a370ef3a-98cb-4251-86b9-cb3155c587a5\") " pod="kube-system/global-pull-secret-syncer-v7fx2" Sep 11 08:46:22.254569 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.254484 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/46676d4c-0012-4617-9fe5-73d33015eb8c-var-lib-kubelet\") pod \"tuned-n56xz\" (UID: \"46676d4c-0012-4617-9fe5-73d33015eb8c\") " pod="openshift-cluster-node-tuning-operator/tuned-n56xz" Sep 11 08:46:22.254569 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.254514 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpcqf\" (UniqueName: \"kubernetes.io/projected/83409be8-1bb2-43dc-af76-d3d74e569101-kube-api-access-rpcqf\") pod \"ovnkube-node-lhkkf\" (UID: \"83409be8-1bb2-43dc-af76-d3d74e569101\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhkkf" Sep 11 08:46:22.254569 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.254538 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/83409be8-1bb2-43dc-af76-d3d74e569101-ovnkube-script-lib\") pod \"ovnkube-node-lhkkf\" (UID: \"83409be8-1bb2-43dc-af76-d3d74e569101\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhkkf" Sep 11 08:46:22.254569 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.254564 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2c0c8738-0dd8-441e-8c6e-3ff6009337ca-konnectivity-ca\") pod \"konnectivity-agent-p5sg8\" (UID: \"2c0c8738-0dd8-441e-8c6e-3ff6009337ca\") " pod="kube-system/konnectivity-agent-p5sg8" Sep 11 08:46:22.254800 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.254578 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/46676d4c-0012-4617-9fe5-73d33015eb8c-var-lib-kubelet\") pod \"tuned-n56xz\" (UID: \"46676d4c-0012-4617-9fe5-73d33015eb8c\") " pod="openshift-cluster-node-tuning-operator/tuned-n56xz" Sep 11 08:46:22.254800 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.254587 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bb4b69fd-d8d3-4134-94c5-16faabe2ec30-system-cni-dir\") pod \"multus-4w5tl\" (UID: \"bb4b69fd-d8d3-4134-94c5-16faabe2ec30\") " pod="openshift-multus/multus-4w5tl" Sep 11 08:46:22.254800 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.254612 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bb4b69fd-d8d3-4134-94c5-16faabe2ec30-host-var-lib-kubelet\") pod \"multus-4w5tl\" (UID: \"bb4b69fd-d8d3-4134-94c5-16faabe2ec30\") " pod="openshift-multus/multus-4w5tl" Sep 11 08:46:22.254800 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.254642 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bb4b69fd-d8d3-4134-94c5-16faabe2ec30-host-run-multus-certs\") pod \"multus-4w5tl\" (UID: \"bb4b69fd-d8d3-4134-94c5-16faabe2ec30\") " pod="openshift-multus/multus-4w5tl" Sep 11 08:46:22.254800 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.254665 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bb4b69fd-d8d3-4134-94c5-16faabe2ec30-etc-kubernetes\") pod \"multus-4w5tl\" (UID: \"bb4b69fd-d8d3-4134-94c5-16faabe2ec30\") " pod="openshift-multus/multus-4w5tl" Sep 11 08:46:22.254800 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.254687 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0c0c64bd-73f4-4cb4-b660-c7232ff8e573-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-k599z\" (UID: \"0c0c64bd-73f4-4cb4-b660-c7232ff8e573\") " pod="openshift-multus/multus-additional-cni-plugins-k599z" Sep 11 08:46:22.254800 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.254703 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/46676d4c-0012-4617-9fe5-73d33015eb8c-etc-systemd\") pod \"tuned-n56xz\" (UID: \"46676d4c-0012-4617-9fe5-73d33015eb8c\") " pod="openshift-cluster-node-tuning-operator/tuned-n56xz" Sep 11 08:46:22.254800 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.254719 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/83409be8-1bb2-43dc-af76-d3d74e569101-var-lib-openvswitch\") pod \"ovnkube-node-lhkkf\" (UID: \"83409be8-1bb2-43dc-af76-d3d74e569101\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhkkf" Sep 11 08:46:22.254800 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.254734 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/83409be8-1bb2-43dc-af76-d3d74e569101-host-run-ovn-kubernetes\") pod \"ovnkube-node-lhkkf\" (UID: \"83409be8-1bb2-43dc-af76-d3d74e569101\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhkkf" Sep 11 08:46:22.254800 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.254750 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/83409be8-1bb2-43dc-af76-d3d74e569101-env-overrides\") pod \"ovnkube-node-lhkkf\" (UID: \"83409be8-1bb2-43dc-af76-d3d74e569101\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhkkf" Sep 11 08:46:22.254800 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.254773 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/510eab66-1dcf-4ffc-983d-d570fb9e9fda-host-slash\") pod \"iptables-alerter-v5rw5\" (UID: \"510eab66-1dcf-4ffc-983d-d570fb9e9fda\") " pod="openshift-network-operator/iptables-alerter-v5rw5" Sep 11 08:46:22.254800 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.254796 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bb4b69fd-d8d3-4134-94c5-16faabe2ec30-host-var-lib-cni-multus\") pod \"multus-4w5tl\" (UID: \"bb4b69fd-d8d3-4134-94c5-16faabe2ec30\") " pod="openshift-multus/multus-4w5tl" Sep 11 08:46:22.255399 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.254816 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0c0c64bd-73f4-4cb4-b660-c7232ff8e573-cni-binary-copy\") pod \"multus-additional-cni-plugins-k599z\" (UID: \"0c0c64bd-73f4-4cb4-b660-c7232ff8e573\") " pod="openshift-multus/multus-additional-cni-plugins-k599z" Sep 11 08:46:22.255399 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.254841 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwpjw\" (UniqueName: \"kubernetes.io/projected/0c0c64bd-73f4-4cb4-b660-c7232ff8e573-kube-api-access-qwpjw\") pod \"multus-additional-cni-plugins-k599z\" (UID: \"0c0c64bd-73f4-4cb4-b660-c7232ff8e573\") " pod="openshift-multus/multus-additional-cni-plugins-k599z" Sep 11 08:46:22.255399 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.254856 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/83409be8-1bb2-43dc-af76-d3d74e569101-host-kubelet\") pod \"ovnkube-node-lhkkf\" (UID: \"83409be8-1bb2-43dc-af76-d3d74e569101\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhkkf" Sep 11 08:46:22.255399 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.254876 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/83409be8-1bb2-43dc-af76-d3d74e569101-etc-openvswitch\") pod \"ovnkube-node-lhkkf\" (UID: \"83409be8-1bb2-43dc-af76-d3d74e569101\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhkkf" Sep 11 08:46:22.255399 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.254902 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/eb6eba26-53e8-4946-9491-cf154a8286f5-serviceca\") pod \"node-ca-67p7z\" (UID: \"eb6eba26-53e8-4946-9491-cf154a8286f5\") " pod="openshift-image-registry/node-ca-67p7z" Sep 11 08:46:22.255399 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.254916 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/83409be8-1bb2-43dc-af76-d3d74e569101-ovn-node-metrics-cert\") pod \"ovnkube-node-lhkkf\" (UID: \"83409be8-1bb2-43dc-af76-d3d74e569101\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhkkf" Sep 11 08:46:22.255399 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.254931 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/83409be8-1bb2-43dc-af76-d3d74e569101-run-ovn\") pod \"ovnkube-node-lhkkf\" (UID: \"83409be8-1bb2-43dc-af76-d3d74e569101\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhkkf" Sep 11 08:46:22.255399 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.254945 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcbdh\" (UniqueName: \"kubernetes.io/projected/82687de7-ba90-4516-99cc-43f545df75bd-kube-api-access-tcbdh\") pod \"node-resolver-4vdzw\" (UID: \"82687de7-ba90-4516-99cc-43f545df75bd\") " pod="openshift-dns/node-resolver-4vdzw" Sep 11 08:46:22.255399 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.254959 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/46676d4c-0012-4617-9fe5-73d33015eb8c-run\") pod \"tuned-n56xz\" (UID: \"46676d4c-0012-4617-9fe5-73d33015eb8c\") " pod="openshift-cluster-node-tuning-operator/tuned-n56xz" Sep 11 08:46:22.255399 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.254976 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcc27\" (UniqueName: \"kubernetes.io/projected/eb6eba26-53e8-4946-9491-cf154a8286f5-kube-api-access-lcc27\") pod \"node-ca-67p7z\" (UID: \"eb6eba26-53e8-4946-9491-cf154a8286f5\") " pod="openshift-image-registry/node-ca-67p7z" Sep 11 08:46:22.255399 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.254990 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/83409be8-1bb2-43dc-af76-d3d74e569101-systemd-units\") pod \"ovnkube-node-lhkkf\" (UID: \"83409be8-1bb2-43dc-af76-d3d74e569101\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhkkf" Sep 11 08:46:22.255399 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.255004 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/83409be8-1bb2-43dc-af76-d3d74e569101-log-socket\") pod \"ovnkube-node-lhkkf\" (UID: \"83409be8-1bb2-43dc-af76-d3d74e569101\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhkkf" Sep 11 08:46:22.255399 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.255018 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/83409be8-1bb2-43dc-af76-d3d74e569101-host-cni-netd\") pod \"ovnkube-node-lhkkf\" (UID: \"83409be8-1bb2-43dc-af76-d3d74e569101\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhkkf" Sep 11 08:46:22.255399 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.255038 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0c0c64bd-73f4-4cb4-b660-c7232ff8e573-tuning-conf-dir\") pod \"multus-additional-cni-plugins-k599z\" (UID: \"0c0c64bd-73f4-4cb4-b660-c7232ff8e573\") " pod="openshift-multus/multus-additional-cni-plugins-k599z" Sep 11 08:46:22.255399 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.255062 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2c0c8738-0dd8-441e-8c6e-3ff6009337ca-agent-certs\") pod \"konnectivity-agent-p5sg8\" (UID: \"2c0c8738-0dd8-441e-8c6e-3ff6009337ca\") " pod="kube-system/konnectivity-agent-p5sg8" Sep 11 08:46:22.255399 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.255088 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bb4b69fd-d8d3-4134-94c5-16faabe2ec30-host-run-netns\") pod \"multus-4w5tl\" (UID: \"bb4b69fd-d8d3-4134-94c5-16faabe2ec30\") " pod="openshift-multus/multus-4w5tl" Sep 11 08:46:22.255399 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.255106 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/83409be8-1bb2-43dc-af76-d3d74e569101-run-systemd\") pod \"ovnkube-node-lhkkf\" (UID: \"83409be8-1bb2-43dc-af76-d3d74e569101\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhkkf" Sep 11 08:46:22.256250 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.255121 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/83409be8-1bb2-43dc-af76-d3d74e569101-node-log\") pod \"ovnkube-node-lhkkf\" (UID: \"83409be8-1bb2-43dc-af76-d3d74e569101\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhkkf" Sep 11 08:46:22.256250 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.255136 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0c0c64bd-73f4-4cb4-b660-c7232ff8e573-system-cni-dir\") pod \"multus-additional-cni-plugins-k599z\" (UID: \"0c0c64bd-73f4-4cb4-b660-c7232ff8e573\") " pod="openshift-multus/multus-additional-cni-plugins-k599z" Sep 11 08:46:22.256250 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.255152 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/82dd9d02-85c9-415b-9db4-d4e8e189ed66-device-dir\") pod \"aws-ebs-csi-driver-node-f7m87\" (UID: \"82dd9d02-85c9-415b-9db4-d4e8e189ed66\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7m87" Sep 11 08:46:22.256250 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.255171 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a370ef3a-98cb-4251-86b9-cb3155c587a5-kubelet-config\") pod \"global-pull-secret-syncer-v7fx2\" (UID: \"a370ef3a-98cb-4251-86b9-cb3155c587a5\") " pod="kube-system/global-pull-secret-syncer-v7fx2" Sep 11 08:46:22.256250 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.255186 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/46676d4c-0012-4617-9fe5-73d33015eb8c-etc-modprobe-d\") pod \"tuned-n56xz\" (UID: \"46676d4c-0012-4617-9fe5-73d33015eb8c\") " pod="openshift-cluster-node-tuning-operator/tuned-n56xz" Sep 11 08:46:22.256250 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.255211 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/46676d4c-0012-4617-9fe5-73d33015eb8c-etc-sysconfig\") pod \"tuned-n56xz\" (UID: \"46676d4c-0012-4617-9fe5-73d33015eb8c\") " pod="openshift-cluster-node-tuning-operator/tuned-n56xz" Sep 11 08:46:22.256250 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.255242 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/46676d4c-0012-4617-9fe5-73d33015eb8c-etc-kubernetes\") pod \"tuned-n56xz\" (UID: \"46676d4c-0012-4617-9fe5-73d33015eb8c\") " pod="openshift-cluster-node-tuning-operator/tuned-n56xz" Sep 11 08:46:22.256250 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.255278 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9b7p\" (UniqueName: \"kubernetes.io/projected/e78a69db-9d98-4923-b082-ccb30b77cc9e-kube-api-access-v9b7p\") pod \"network-metrics-daemon-ntkhj\" (UID: \"e78a69db-9d98-4923-b082-ccb30b77cc9e\") " pod="openshift-multus/network-metrics-daemon-ntkhj" Sep 11 08:46:22.256250 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.255287 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2c0c8738-0dd8-441e-8c6e-3ff6009337ca-konnectivity-ca\") pod \"konnectivity-agent-p5sg8\" (UID: \"2c0c8738-0dd8-441e-8c6e-3ff6009337ca\") " pod="kube-system/konnectivity-agent-p5sg8" Sep 11 08:46:22.256250 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.255369 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/83409be8-1bb2-43dc-af76-d3d74e569101-ovnkube-script-lib\") pod \"ovnkube-node-lhkkf\" (UID: \"83409be8-1bb2-43dc-af76-d3d74e569101\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhkkf" Sep 11 08:46:22.256250 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.255294 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82687de7-ba90-4516-99cc-43f545df75bd-tmp-dir\") pod \"node-resolver-4vdzw\" (UID: \"82687de7-ba90-4516-99cc-43f545df75bd\") " pod="openshift-dns/node-resolver-4vdzw" Sep 11 08:46:22.256250 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.255419 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/83409be8-1bb2-43dc-af76-d3d74e569101-var-lib-openvswitch\") pod \"ovnkube-node-lhkkf\" (UID: \"83409be8-1bb2-43dc-af76-d3d74e569101\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhkkf" Sep 11 08:46:22.256250 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.255428 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bb4b69fd-d8d3-4134-94c5-16faabe2ec30-multus-socket-dir-parent\") pod \"multus-4w5tl\" (UID: \"bb4b69fd-d8d3-4134-94c5-16faabe2ec30\") " pod="openshift-multus/multus-4w5tl" Sep 11 08:46:22.256250 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.255450 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/83409be8-1bb2-43dc-af76-d3d74e569101-host-run-ovn-kubernetes\") pod \"ovnkube-node-lhkkf\" (UID: \"83409be8-1bb2-43dc-af76-d3d74e569101\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhkkf" Sep 11 08:46:22.256250 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.255457 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/82dd9d02-85c9-415b-9db4-d4e8e189ed66-socket-dir\") pod \"aws-ebs-csi-driver-node-f7m87\" (UID: \"82dd9d02-85c9-415b-9db4-d4e8e189ed66\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7m87" Sep 11 08:46:22.256250 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.255488 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clk2v\" (UniqueName: \"kubernetes.io/projected/46676d4c-0012-4617-9fe5-73d33015eb8c-kube-api-access-clk2v\") pod \"tuned-n56xz\" (UID: \"46676d4c-0012-4617-9fe5-73d33015eb8c\") " pod="openshift-cluster-node-tuning-operator/tuned-n56xz" Sep 11 08:46:22.256250 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.255529 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/eb6eba26-53e8-4946-9491-cf154a8286f5-host\") pod \"node-ca-67p7z\" (UID: \"eb6eba26-53e8-4946-9491-cf154a8286f5\") " pod="openshift-image-registry/node-ca-67p7z" Sep 11 08:46:22.257054 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.255581 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/83409be8-1bb2-43dc-af76-d3d74e569101-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lhkkf\" (UID: \"83409be8-1bb2-43dc-af76-d3d74e569101\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhkkf" Sep 11 08:46:22.257054 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.255607 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bb4b69fd-d8d3-4134-94c5-16faabe2ec30-multus-cni-dir\") pod \"multus-4w5tl\" (UID: \"bb4b69fd-d8d3-4134-94c5-16faabe2ec30\") " pod="openshift-multus/multus-4w5tl" Sep 11 08:46:22.257054 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.255641 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lx99\" (UniqueName: \"kubernetes.io/projected/510eab66-1dcf-4ffc-983d-d570fb9e9fda-kube-api-access-6lx99\") pod \"iptables-alerter-v5rw5\" (UID: \"510eab66-1dcf-4ffc-983d-d570fb9e9fda\") " pod="openshift-network-operator/iptables-alerter-v5rw5" Sep 11 08:46:22.257054 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.255667 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/46676d4c-0012-4617-9fe5-73d33015eb8c-etc-sysctl-d\") pod \"tuned-n56xz\" (UID: \"46676d4c-0012-4617-9fe5-73d33015eb8c\") " pod="openshift-cluster-node-tuning-operator/tuned-n56xz" Sep 11 08:46:22.257054 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.255691 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/83409be8-1bb2-43dc-af76-d3d74e569101-host-cni-bin\") pod \"ovnkube-node-lhkkf\" (UID: \"83409be8-1bb2-43dc-af76-d3d74e569101\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhkkf" Sep 11 08:46:22.257054 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.255733 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/83409be8-1bb2-43dc-af76-d3d74e569101-ovnkube-config\") pod \"ovnkube-node-lhkkf\" (UID: \"83409be8-1bb2-43dc-af76-d3d74e569101\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhkkf" Sep 11 08:46:22.257054 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.255766 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bb4b69fd-d8d3-4134-94c5-16faabe2ec30-multus-conf-dir\") pod \"multus-4w5tl\" (UID: \"bb4b69fd-d8d3-4134-94c5-16faabe2ec30\") " pod="openshift-multus/multus-4w5tl" Sep 11 08:46:22.257054 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.255796 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/83409be8-1bb2-43dc-af76-d3d74e569101-env-overrides\") pod \"ovnkube-node-lhkkf\" (UID: \"83409be8-1bb2-43dc-af76-d3d74e569101\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhkkf" Sep 11 08:46:22.257054 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.255805 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bb4b69fd-d8d3-4134-94c5-16faabe2ec30-multus-daemon-config\") pod \"multus-4w5tl\" (UID: \"bb4b69fd-d8d3-4134-94c5-16faabe2ec30\") " pod="openshift-multus/multus-4w5tl" Sep 11 08:46:22.257054 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.255829 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/82dd9d02-85c9-415b-9db4-d4e8e189ed66-kubelet-dir\") pod \"aws-ebs-csi-driver-node-f7m87\" (UID: \"82dd9d02-85c9-415b-9db4-d4e8e189ed66\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7m87" Sep 11 08:46:22.257054 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.255853 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/82dd9d02-85c9-415b-9db4-d4e8e189ed66-sys-fs\") pod \"aws-ebs-csi-driver-node-f7m87\" (UID: \"82dd9d02-85c9-415b-9db4-d4e8e189ed66\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7m87" Sep 11 08:46:22.257054 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.255855 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/510eab66-1dcf-4ffc-983d-d570fb9e9fda-host-slash\") pod \"iptables-alerter-v5rw5\" (UID: \"510eab66-1dcf-4ffc-983d-d570fb9e9fda\") " pod="openshift-network-operator/iptables-alerter-v5rw5" Sep 11 08:46:22.257054 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.255879 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0c0c64bd-73f4-4cb4-b660-c7232ff8e573-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-k599z\" (UID: \"0c0c64bd-73f4-4cb4-b660-c7232ff8e573\") " pod="openshift-multus/multus-additional-cni-plugins-k599z" Sep 11 08:46:22.257054 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.255880 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0c0c64bd-73f4-4cb4-b660-c7232ff8e573-cnibin\") pod \"multus-additional-cni-plugins-k599z\" (UID: \"0c0c64bd-73f4-4cb4-b660-c7232ff8e573\") " pod="openshift-multus/multus-additional-cni-plugins-k599z" Sep 11 08:46:22.257054 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.256012 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/83409be8-1bb2-43dc-af76-d3d74e569101-host-run-netns\") pod \"ovnkube-node-lhkkf\" (UID: \"83409be8-1bb2-43dc-af76-d3d74e569101\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhkkf" Sep 11 08:46:22.257054 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.256042 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/46676d4c-0012-4617-9fe5-73d33015eb8c-tmp\") pod \"tuned-n56xz\" (UID: \"46676d4c-0012-4617-9fe5-73d33015eb8c\") " pod="openshift-cluster-node-tuning-operator/tuned-n56xz" Sep 11 08:46:22.257054 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.256064 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bb4b69fd-d8d3-4134-94c5-16faabe2ec30-cnibin\") pod \"multus-4w5tl\" (UID: \"bb4b69fd-d8d3-4134-94c5-16faabe2ec30\") " pod="openshift-multus/multus-4w5tl" Sep 11 08:46:22.257841 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.255396 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/46676d4c-0012-4617-9fe5-73d33015eb8c-etc-systemd\") pod \"tuned-n56xz\" (UID: \"46676d4c-0012-4617-9fe5-73d33015eb8c\") " pod="openshift-cluster-node-tuning-operator/tuned-n56xz" Sep 11 08:46:22.257841 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.256191 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/83409be8-1bb2-43dc-af76-d3d74e569101-host-cni-bin\") pod \"ovnkube-node-lhkkf\" (UID: \"83409be8-1bb2-43dc-af76-d3d74e569101\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhkkf" Sep 11 08:46:22.257841 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.256152 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bb4b69fd-d8d3-4134-94c5-16faabe2ec30-cni-binary-copy\") pod \"multus-4w5tl\" (UID: \"bb4b69fd-d8d3-4134-94c5-16faabe2ec30\") " pod="openshift-multus/multus-4w5tl" Sep 11 08:46:22.257841 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.256250 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/82dd9d02-85c9-415b-9db4-d4e8e189ed66-etc-selinux\") pod \"aws-ebs-csi-driver-node-f7m87\" (UID: \"82dd9d02-85c9-415b-9db4-d4e8e189ed66\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7m87" Sep 11 08:46:22.257841 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.256296 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/46676d4c-0012-4617-9fe5-73d33015eb8c-etc-sysctl-conf\") pod \"tuned-n56xz\" (UID: \"46676d4c-0012-4617-9fe5-73d33015eb8c\") " pod="openshift-cluster-node-tuning-operator/tuned-n56xz" Sep 11 08:46:22.257841 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.256323 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/46676d4c-0012-4617-9fe5-73d33015eb8c-sys\") pod \"tuned-n56xz\" (UID: \"46676d4c-0012-4617-9fe5-73d33015eb8c\") " pod="openshift-cluster-node-tuning-operator/tuned-n56xz" Sep 11 08:46:22.257841 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.256378 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwc96\" (UniqueName: \"kubernetes.io/projected/12ac768c-2c6a-43e7-9bd3-44679781ebd7-kube-api-access-zwc96\") pod \"network-check-target-82xp9\" (UID: \"12ac768c-2c6a-43e7-9bd3-44679781ebd7\") " pod="openshift-network-diagnostics/network-check-target-82xp9" Sep 11 08:46:22.257841 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.256406 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/83409be8-1bb2-43dc-af76-d3d74e569101-host-slash\") pod \"ovnkube-node-lhkkf\" (UID: \"83409be8-1bb2-43dc-af76-d3d74e569101\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhkkf" Sep 11 08:46:22.257841 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.256431 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bb4b69fd-d8d3-4134-94c5-16faabe2ec30-hostroot\") pod \"multus-4w5tl\" (UID: \"bb4b69fd-d8d3-4134-94c5-16faabe2ec30\") " pod="openshift-multus/multus-4w5tl" Sep 11 08:46:22.257841 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.256471 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0c0c64bd-73f4-4cb4-b660-c7232ff8e573-cni-binary-copy\") pod \"multus-additional-cni-plugins-k599z\" (UID: \"0c0c64bd-73f4-4cb4-b660-c7232ff8e573\") " pod="openshift-multus/multus-additional-cni-plugins-k599z" Sep 11 08:46:22.257841 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.256459 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzvn7\" (UniqueName: \"kubernetes.io/projected/82dd9d02-85c9-415b-9db4-d4e8e189ed66-kube-api-access-fzvn7\") pod \"aws-ebs-csi-driver-node-f7m87\" (UID: \"82dd9d02-85c9-415b-9db4-d4e8e189ed66\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7m87" Sep 11 08:46:22.257841 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.256676 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/46676d4c-0012-4617-9fe5-73d33015eb8c-etc-sysctl-d\") pod \"tuned-n56xz\" (UID: \"46676d4c-0012-4617-9fe5-73d33015eb8c\") " pod="openshift-cluster-node-tuning-operator/tuned-n56xz" Sep 11 08:46:22.257841 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.256792 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/83409be8-1bb2-43dc-af76-d3d74e569101-ovnkube-config\") pod \"ovnkube-node-lhkkf\" (UID: \"83409be8-1bb2-43dc-af76-d3d74e569101\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhkkf" Sep 11 08:46:22.257841 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.256812 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/46676d4c-0012-4617-9fe5-73d33015eb8c-etc-sysctl-conf\") pod \"tuned-n56xz\" (UID: \"46676d4c-0012-4617-9fe5-73d33015eb8c\") " pod="openshift-cluster-node-tuning-operator/tuned-n56xz" Sep 11 08:46:22.257841 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.256889 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/46676d4c-0012-4617-9fe5-73d33015eb8c-sys\") pod \"tuned-n56xz\" (UID: \"46676d4c-0012-4617-9fe5-73d33015eb8c\") " pod="openshift-cluster-node-tuning-operator/tuned-n56xz" Sep 11 08:46:22.257841 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.257005 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0c0c64bd-73f4-4cb4-b660-c7232ff8e573-cnibin\") pod \"multus-additional-cni-plugins-k599z\" (UID: \"0c0c64bd-73f4-4cb4-b660-c7232ff8e573\") " pod="openshift-multus/multus-additional-cni-plugins-k599z" Sep 11 08:46:22.257841 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.257010 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/eb6eba26-53e8-4946-9491-cf154a8286f5-serviceca\") pod \"node-ca-67p7z\" (UID: \"eb6eba26-53e8-4946-9491-cf154a8286f5\") " pod="openshift-image-registry/node-ca-67p7z" Sep 11 08:46:22.258632 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.257053 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/83409be8-1bb2-43dc-af76-d3d74e569101-host-run-netns\") pod \"ovnkube-node-lhkkf\" (UID: \"83409be8-1bb2-43dc-af76-d3d74e569101\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhkkf" Sep 11 08:46:22.258632 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.257080 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a370ef3a-98cb-4251-86b9-cb3155c587a5-dbus\") pod \"global-pull-secret-syncer-v7fx2\" (UID: \"a370ef3a-98cb-4251-86b9-cb3155c587a5\") " pod="kube-system/global-pull-secret-syncer-v7fx2" Sep 11 08:46:22.258632 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.257123 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/46676d4c-0012-4617-9fe5-73d33015eb8c-lib-modules\") pod \"tuned-n56xz\" (UID: \"46676d4c-0012-4617-9fe5-73d33015eb8c\") " pod="openshift-cluster-node-tuning-operator/tuned-n56xz" Sep 11 08:46:22.258632 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.257140 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/46676d4c-0012-4617-9fe5-73d33015eb8c-etc-tuned\") pod \"tuned-n56xz\" (UID: \"46676d4c-0012-4617-9fe5-73d33015eb8c\") " pod="openshift-cluster-node-tuning-operator/tuned-n56xz" Sep 11 08:46:22.258632 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.257148 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/83409be8-1bb2-43dc-af76-d3d74e569101-host-kubelet\") pod \"ovnkube-node-lhkkf\" (UID: \"83409be8-1bb2-43dc-af76-d3d74e569101\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhkkf" Sep 11 08:46:22.258632 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.257177 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/83409be8-1bb2-43dc-af76-d3d74e569101-run-openvswitch\") pod \"ovnkube-node-lhkkf\" (UID: \"83409be8-1bb2-43dc-af76-d3d74e569101\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhkkf" Sep 11 08:46:22.258632 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.257456 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0c0c64bd-73f4-4cb4-b660-c7232ff8e573-tuning-conf-dir\") pod \"multus-additional-cni-plugins-k599z\" (UID: \"0c0c64bd-73f4-4cb4-b660-c7232ff8e573\") " pod="openshift-multus/multus-additional-cni-plugins-k599z" Sep 11 08:46:22.258632 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.257465 2572 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 11 08:46:22.258632 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.257501 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/83409be8-1bb2-43dc-af76-d3d74e569101-host-cni-netd\") pod \"ovnkube-node-lhkkf\" (UID: \"83409be8-1bb2-43dc-af76-d3d74e569101\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhkkf" Sep 11 08:46:22.258632 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.257528 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/83409be8-1bb2-43dc-af76-d3d74e569101-etc-openvswitch\") pod \"ovnkube-node-lhkkf\" (UID: \"83409be8-1bb2-43dc-af76-d3d74e569101\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhkkf" Sep 11 08:46:22.258632 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.257548 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/83409be8-1bb2-43dc-af76-d3d74e569101-host-slash\") pod \"ovnkube-node-lhkkf\" (UID: \"83409be8-1bb2-43dc-af76-d3d74e569101\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhkkf" Sep 11 08:46:22.258632 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.257588 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/83409be8-1bb2-43dc-af76-d3d74e569101-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lhkkf\" (UID: \"83409be8-1bb2-43dc-af76-d3d74e569101\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhkkf" Sep 11 08:46:22.258632 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.257591 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/46676d4c-0012-4617-9fe5-73d33015eb8c-etc-kubernetes\") pod \"tuned-n56xz\" (UID: \"46676d4c-0012-4617-9fe5-73d33015eb8c\") " pod="openshift-cluster-node-tuning-operator/tuned-n56xz" Sep 11 08:46:22.258632 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.257626 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/eb6eba26-53e8-4946-9491-cf154a8286f5-host\") pod \"node-ca-67p7z\" (UID: \"eb6eba26-53e8-4946-9491-cf154a8286f5\") " pod="openshift-image-registry/node-ca-67p7z" Sep 11 08:46:22.258632 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.257725 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/46676d4c-0012-4617-9fe5-73d33015eb8c-etc-modprobe-d\") pod \"tuned-n56xz\" (UID: \"46676d4c-0012-4617-9fe5-73d33015eb8c\") " pod="openshift-cluster-node-tuning-operator/tuned-n56xz" Sep 11 08:46:22.258632 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.257799 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/46676d4c-0012-4617-9fe5-73d33015eb8c-lib-modules\") pod \"tuned-n56xz\" (UID: \"46676d4c-0012-4617-9fe5-73d33015eb8c\") " pod="openshift-cluster-node-tuning-operator/tuned-n56xz" Sep 11 08:46:22.258632 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.257820 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e78a69db-9d98-4923-b082-ccb30b77cc9e-metrics-certs\") pod \"network-metrics-daemon-ntkhj\" (UID: \"e78a69db-9d98-4923-b082-ccb30b77cc9e\") " pod="openshift-multus/network-metrics-daemon-ntkhj" Sep 11 08:46:22.258632 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.257848 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/46676d4c-0012-4617-9fe5-73d33015eb8c-etc-sysconfig\") pod \"tuned-n56xz\" (UID: \"46676d4c-0012-4617-9fe5-73d33015eb8c\") " pod="openshift-cluster-node-tuning-operator/tuned-n56xz" Sep 11 08:46:22.259512 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.257864 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0c0c64bd-73f4-4cb4-b660-c7232ff8e573-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-k599z\" (UID: \"0c0c64bd-73f4-4cb4-b660-c7232ff8e573\") " pod="openshift-multus/multus-additional-cni-plugins-k599z" Sep 11 08:46:22.259512 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.257935 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/82687de7-ba90-4516-99cc-43f545df75bd-hosts-file\") pod \"node-resolver-4vdzw\" (UID: \"82687de7-ba90-4516-99cc-43f545df75bd\") " pod="openshift-dns/node-resolver-4vdzw" Sep 11 08:46:22.259512 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.257961 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bb4b69fd-d8d3-4134-94c5-16faabe2ec30-host-run-k8s-cni-cncf-io\") pod \"multus-4w5tl\" (UID: \"bb4b69fd-d8d3-4134-94c5-16faabe2ec30\") " pod="openshift-multus/multus-4w5tl" Sep 11 08:46:22.259512 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.257978 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/46676d4c-0012-4617-9fe5-73d33015eb8c-run\") pod \"tuned-n56xz\" (UID: \"46676d4c-0012-4617-9fe5-73d33015eb8c\") " pod="openshift-cluster-node-tuning-operator/tuned-n56xz" Sep 11 08:46:22.259512 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.257987 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q96vm\" (UniqueName: \"kubernetes.io/projected/bb4b69fd-d8d3-4134-94c5-16faabe2ec30-kube-api-access-q96vm\") pod \"multus-4w5tl\" (UID: \"bb4b69fd-d8d3-4134-94c5-16faabe2ec30\") " pod="openshift-multus/multus-4w5tl" Sep 11 08:46:22.259512 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.257892 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0c0c64bd-73f4-4cb4-b660-c7232ff8e573-system-cni-dir\") pod \"multus-additional-cni-plugins-k599z\" (UID: \"0c0c64bd-73f4-4cb4-b660-c7232ff8e573\") " pod="openshift-multus/multus-additional-cni-plugins-k599z" Sep 11 08:46:22.259512 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.258121 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/46676d4c-0012-4617-9fe5-73d33015eb8c-host\") pod \"tuned-n56xz\" (UID: \"46676d4c-0012-4617-9fe5-73d33015eb8c\") " pod="openshift-cluster-node-tuning-operator/tuned-n56xz" Sep 11 08:46:22.259512 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.258149 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/510eab66-1dcf-4ffc-983d-d570fb9e9fda-iptables-alerter-script\") pod \"iptables-alerter-v5rw5\" (UID: \"510eab66-1dcf-4ffc-983d-d570fb9e9fda\") " pod="openshift-network-operator/iptables-alerter-v5rw5" Sep 11 08:46:22.259512 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.258183 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/83409be8-1bb2-43dc-af76-d3d74e569101-run-systemd\") pod \"ovnkube-node-lhkkf\" (UID: \"83409be8-1bb2-43dc-af76-d3d74e569101\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhkkf" Sep 11 08:46:22.259512 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.258193 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bb4b69fd-d8d3-4134-94c5-16faabe2ec30-host-var-lib-cni-bin\") pod \"multus-4w5tl\" (UID: \"bb4b69fd-d8d3-4134-94c5-16faabe2ec30\") " pod="openshift-multus/multus-4w5tl" Sep 11 08:46:22.259512 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:22.258246 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 08:46:22.259512 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.258261 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/82dd9d02-85c9-415b-9db4-d4e8e189ed66-registration-dir\") pod \"aws-ebs-csi-driver-node-f7m87\" (UID: \"82dd9d02-85c9-415b-9db4-d4e8e189ed66\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7m87" Sep 11 08:46:22.259512 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:22.258343 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e78a69db-9d98-4923-b082-ccb30b77cc9e-metrics-certs podName:e78a69db-9d98-4923-b082-ccb30b77cc9e nodeName:}" failed. No retries permitted until 2026-09-11 08:46:22.758305555 +0000 UTC m=+3.032881249 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e78a69db-9d98-4923-b082-ccb30b77cc9e-metrics-certs") pod "network-metrics-daemon-ntkhj" (UID: "e78a69db-9d98-4923-b082-ccb30b77cc9e") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 08:46:22.259512 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.258361 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/83409be8-1bb2-43dc-af76-d3d74e569101-run-openvswitch\") pod \"ovnkube-node-lhkkf\" (UID: \"83409be8-1bb2-43dc-af76-d3d74e569101\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhkkf" Sep 11 08:46:22.259512 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.258403 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/83409be8-1bb2-43dc-af76-d3d74e569101-systemd-units\") pod \"ovnkube-node-lhkkf\" (UID: \"83409be8-1bb2-43dc-af76-d3d74e569101\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhkkf" Sep 11 08:46:22.259512 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.258519 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/83409be8-1bb2-43dc-af76-d3d74e569101-run-ovn\") pod \"ovnkube-node-lhkkf\" (UID: \"83409be8-1bb2-43dc-af76-d3d74e569101\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhkkf" Sep 11 08:46:22.259512 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.258572 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/46676d4c-0012-4617-9fe5-73d33015eb8c-host\") pod \"tuned-n56xz\" (UID: \"46676d4c-0012-4617-9fe5-73d33015eb8c\") " pod="openshift-cluster-node-tuning-operator/tuned-n56xz" Sep 11 08:46:22.260165 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.258608 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/83409be8-1bb2-43dc-af76-d3d74e569101-node-log\") pod \"ovnkube-node-lhkkf\" (UID: \"83409be8-1bb2-43dc-af76-d3d74e569101\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhkkf" Sep 11 08:46:22.260165 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.259096 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/510eab66-1dcf-4ffc-983d-d570fb9e9fda-iptables-alerter-script\") pod \"iptables-alerter-v5rw5\" (UID: \"510eab66-1dcf-4ffc-983d-d570fb9e9fda\") " pod="openshift-network-operator/iptables-alerter-v5rw5" Sep 11 08:46:22.260165 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.259212 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0c0c64bd-73f4-4cb4-b660-c7232ff8e573-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-k599z\" (UID: \"0c0c64bd-73f4-4cb4-b660-c7232ff8e573\") " pod="openshift-multus/multus-additional-cni-plugins-k599z" Sep 11 08:46:22.260165 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.259251 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/83409be8-1bb2-43dc-af76-d3d74e569101-log-socket\") pod \"ovnkube-node-lhkkf\" (UID: \"83409be8-1bb2-43dc-af76-d3d74e569101\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhkkf" Sep 11 08:46:22.261777 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.261747 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/46676d4c-0012-4617-9fe5-73d33015eb8c-tmp\") pod \"tuned-n56xz\" (UID: \"46676d4c-0012-4617-9fe5-73d33015eb8c\") " pod="openshift-cluster-node-tuning-operator/tuned-n56xz" Sep 11 08:46:22.265084 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.262461 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/46676d4c-0012-4617-9fe5-73d33015eb8c-etc-tuned\") pod \"tuned-n56xz\" (UID: \"46676d4c-0012-4617-9fe5-73d33015eb8c\") " pod="openshift-cluster-node-tuning-operator/tuned-n56xz" Sep 11 08:46:22.265084 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.263447 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/83409be8-1bb2-43dc-af76-d3d74e569101-ovn-node-metrics-cert\") pod \"ovnkube-node-lhkkf\" (UID: \"83409be8-1bb2-43dc-af76-d3d74e569101\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhkkf" Sep 11 08:46:22.265084 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.264995 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2c0c8738-0dd8-441e-8c6e-3ff6009337ca-agent-certs\") pod \"konnectivity-agent-p5sg8\" (UID: \"2c0c8738-0dd8-441e-8c6e-3ff6009337ca\") " pod="kube-system/konnectivity-agent-p5sg8" Sep 11 08:46:22.266997 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.266974 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpcqf\" (UniqueName: \"kubernetes.io/projected/83409be8-1bb2-43dc-af76-d3d74e569101-kube-api-access-rpcqf\") pod \"ovnkube-node-lhkkf\" (UID: \"83409be8-1bb2-43dc-af76-d3d74e569101\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhkkf" Sep 11 08:46:22.267761 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:22.267742 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 08:46:22.267859 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:22.267769 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 08:46:22.267859 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:22.267784 2572 projected.go:196] Error preparing data for projected volume kube-api-access-zwc96 for pod openshift-network-diagnostics/network-check-target-82xp9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 08:46:22.267859 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:22.267843 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/12ac768c-2c6a-43e7-9bd3-44679781ebd7-kube-api-access-zwc96 podName:12ac768c-2c6a-43e7-9bd3-44679781ebd7 nodeName:}" failed. No retries permitted until 2026-09-11 08:46:22.767826604 +0000 UTC m=+3.042402284 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-zwc96" (UniqueName: "kubernetes.io/projected/12ac768c-2c6a-43e7-9bd3-44679781ebd7-kube-api-access-zwc96") pod "network-check-target-82xp9" (UID: "12ac768c-2c6a-43e7-9bd3-44679781ebd7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 08:46:22.269439 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.269414 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lx99\" (UniqueName: \"kubernetes.io/projected/510eab66-1dcf-4ffc-983d-d570fb9e9fda-kube-api-access-6lx99\") pod \"iptables-alerter-v5rw5\" (UID: \"510eab66-1dcf-4ffc-983d-d570fb9e9fda\") " pod="openshift-network-operator/iptables-alerter-v5rw5" Sep 11 08:46:22.270741 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.270721 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcc27\" (UniqueName: \"kubernetes.io/projected/eb6eba26-53e8-4946-9491-cf154a8286f5-kube-api-access-lcc27\") pod \"node-ca-67p7z\" (UID: \"eb6eba26-53e8-4946-9491-cf154a8286f5\") " pod="openshift-image-registry/node-ca-67p7z" Sep 11 08:46:22.271080 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.271061 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9b7p\" (UniqueName: \"kubernetes.io/projected/e78a69db-9d98-4923-b082-ccb30b77cc9e-kube-api-access-v9b7p\") pod \"network-metrics-daemon-ntkhj\" (UID: \"e78a69db-9d98-4923-b082-ccb30b77cc9e\") " pod="openshift-multus/network-metrics-daemon-ntkhj" Sep 11 08:46:22.271654 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.271619 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-clk2v\" (UniqueName: \"kubernetes.io/projected/46676d4c-0012-4617-9fe5-73d33015eb8c-kube-api-access-clk2v\") pod \"tuned-n56xz\" (UID: \"46676d4c-0012-4617-9fe5-73d33015eb8c\") " pod="openshift-cluster-node-tuning-operator/tuned-n56xz" Sep 11 08:46:22.272293 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.272269 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwpjw\" (UniqueName: \"kubernetes.io/projected/0c0c64bd-73f4-4cb4-b660-c7232ff8e573-kube-api-access-qwpjw\") pod \"multus-additional-cni-plugins-k599z\" (UID: \"0c0c64bd-73f4-4cb4-b660-c7232ff8e573\") " pod="openshift-multus/multus-additional-cni-plugins-k599z" Sep 11 08:46:22.359318 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.359288 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bb4b69fd-d8d3-4134-94c5-16faabe2ec30-hostroot\") pod \"multus-4w5tl\" (UID: \"bb4b69fd-d8d3-4134-94c5-16faabe2ec30\") " pod="openshift-multus/multus-4w5tl" Sep 11 08:46:22.359496 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.359326 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzvn7\" (UniqueName: \"kubernetes.io/projected/82dd9d02-85c9-415b-9db4-d4e8e189ed66-kube-api-access-fzvn7\") pod \"aws-ebs-csi-driver-node-f7m87\" (UID: \"82dd9d02-85c9-415b-9db4-d4e8e189ed66\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7m87" Sep 11 08:46:22.359496 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.359350 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a370ef3a-98cb-4251-86b9-cb3155c587a5-dbus\") pod \"global-pull-secret-syncer-v7fx2\" (UID: \"a370ef3a-98cb-4251-86b9-cb3155c587a5\") " pod="kube-system/global-pull-secret-syncer-v7fx2" Sep 11 08:46:22.359496 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.359380 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/82687de7-ba90-4516-99cc-43f545df75bd-hosts-file\") pod \"node-resolver-4vdzw\" (UID: \"82687de7-ba90-4516-99cc-43f545df75bd\") " pod="openshift-dns/node-resolver-4vdzw" Sep 11 08:46:22.359496 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.359402 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bb4b69fd-d8d3-4134-94c5-16faabe2ec30-host-run-k8s-cni-cncf-io\") pod \"multus-4w5tl\" (UID: \"bb4b69fd-d8d3-4134-94c5-16faabe2ec30\") " pod="openshift-multus/multus-4w5tl" Sep 11 08:46:22.359496 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.359423 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q96vm\" (UniqueName: \"kubernetes.io/projected/bb4b69fd-d8d3-4134-94c5-16faabe2ec30-kube-api-access-q96vm\") pod \"multus-4w5tl\" (UID: \"bb4b69fd-d8d3-4134-94c5-16faabe2ec30\") " pod="openshift-multus/multus-4w5tl" Sep 11 08:46:22.359496 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.359445 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bb4b69fd-d8d3-4134-94c5-16faabe2ec30-host-var-lib-cni-bin\") pod \"multus-4w5tl\" (UID: \"bb4b69fd-d8d3-4134-94c5-16faabe2ec30\") " pod="openshift-multus/multus-4w5tl" Sep 11 08:46:22.359496 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.359466 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/82dd9d02-85c9-415b-9db4-d4e8e189ed66-registration-dir\") pod \"aws-ebs-csi-driver-node-f7m87\" (UID: \"82dd9d02-85c9-415b-9db4-d4e8e189ed66\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7m87" Sep 11 08:46:22.359496 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.359488 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a370ef3a-98cb-4251-86b9-cb3155c587a5-original-pull-secret\") pod \"global-pull-secret-syncer-v7fx2\" (UID: \"a370ef3a-98cb-4251-86b9-cb3155c587a5\") " pod="kube-system/global-pull-secret-syncer-v7fx2" Sep 11 08:46:22.359893 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.359512 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bb4b69fd-d8d3-4134-94c5-16faabe2ec30-system-cni-dir\") pod \"multus-4w5tl\" (UID: \"bb4b69fd-d8d3-4134-94c5-16faabe2ec30\") " pod="openshift-multus/multus-4w5tl" Sep 11 08:46:22.359893 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.359537 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bb4b69fd-d8d3-4134-94c5-16faabe2ec30-host-var-lib-kubelet\") pod \"multus-4w5tl\" (UID: \"bb4b69fd-d8d3-4134-94c5-16faabe2ec30\") " pod="openshift-multus/multus-4w5tl" Sep 11 08:46:22.359893 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.359555 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a370ef3a-98cb-4251-86b9-cb3155c587a5-dbus\") pod \"global-pull-secret-syncer-v7fx2\" (UID: \"a370ef3a-98cb-4251-86b9-cb3155c587a5\") " pod="kube-system/global-pull-secret-syncer-v7fx2" Sep 11 08:46:22.359893 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.359560 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bb4b69fd-d8d3-4134-94c5-16faabe2ec30-host-run-multus-certs\") pod \"multus-4w5tl\" (UID: \"bb4b69fd-d8d3-4134-94c5-16faabe2ec30\") " pod="openshift-multus/multus-4w5tl" Sep 11 08:46:22.359893 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.359609 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bb4b69fd-d8d3-4134-94c5-16faabe2ec30-host-run-multus-certs\") pod \"multus-4w5tl\" (UID: \"bb4b69fd-d8d3-4134-94c5-16faabe2ec30\") " pod="openshift-multus/multus-4w5tl" Sep 11 08:46:22.359893 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.359611 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bb4b69fd-d8d3-4134-94c5-16faabe2ec30-etc-kubernetes\") pod \"multus-4w5tl\" (UID: \"bb4b69fd-d8d3-4134-94c5-16faabe2ec30\") " pod="openshift-multus/multus-4w5tl" Sep 11 08:46:22.359893 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.359648 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bb4b69fd-d8d3-4134-94c5-16faabe2ec30-etc-kubernetes\") pod \"multus-4w5tl\" (UID: \"bb4b69fd-d8d3-4134-94c5-16faabe2ec30\") " pod="openshift-multus/multus-4w5tl" Sep 11 08:46:22.359893 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.359652 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bb4b69fd-d8d3-4134-94c5-16faabe2ec30-host-var-lib-cni-multus\") pod \"multus-4w5tl\" (UID: \"bb4b69fd-d8d3-4134-94c5-16faabe2ec30\") " pod="openshift-multus/multus-4w5tl" Sep 11 08:46:22.359893 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.359683 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bb4b69fd-d8d3-4134-94c5-16faabe2ec30-hostroot\") pod \"multus-4w5tl\" (UID: \"bb4b69fd-d8d3-4134-94c5-16faabe2ec30\") " pod="openshift-multus/multus-4w5tl" Sep 11 08:46:22.359893 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.359687 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcbdh\" (UniqueName: \"kubernetes.io/projected/82687de7-ba90-4516-99cc-43f545df75bd-kube-api-access-tcbdh\") pod \"node-resolver-4vdzw\" (UID: \"82687de7-ba90-4516-99cc-43f545df75bd\") " pod="openshift-dns/node-resolver-4vdzw" Sep 11 08:46:22.359893 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.359720 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bb4b69fd-d8d3-4134-94c5-16faabe2ec30-host-run-netns\") pod \"multus-4w5tl\" (UID: \"bb4b69fd-d8d3-4134-94c5-16faabe2ec30\") " pod="openshift-multus/multus-4w5tl" Sep 11 08:46:22.359893 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.359746 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/82dd9d02-85c9-415b-9db4-d4e8e189ed66-device-dir\") pod \"aws-ebs-csi-driver-node-f7m87\" (UID: \"82dd9d02-85c9-415b-9db4-d4e8e189ed66\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7m87" Sep 11 08:46:22.359893 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.359769 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a370ef3a-98cb-4251-86b9-cb3155c587a5-kubelet-config\") pod \"global-pull-secret-syncer-v7fx2\" (UID: \"a370ef3a-98cb-4251-86b9-cb3155c587a5\") " pod="kube-system/global-pull-secret-syncer-v7fx2" Sep 11 08:46:22.359893 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.359795 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82687de7-ba90-4516-99cc-43f545df75bd-tmp-dir\") pod \"node-resolver-4vdzw\" (UID: \"82687de7-ba90-4516-99cc-43f545df75bd\") " pod="openshift-dns/node-resolver-4vdzw" Sep 11 08:46:22.359893 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.359820 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bb4b69fd-d8d3-4134-94c5-16faabe2ec30-multus-socket-dir-parent\") pod \"multus-4w5tl\" (UID: \"bb4b69fd-d8d3-4134-94c5-16faabe2ec30\") " pod="openshift-multus/multus-4w5tl" Sep 11 08:46:22.359893 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.359850 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/82dd9d02-85c9-415b-9db4-d4e8e189ed66-socket-dir\") pod \"aws-ebs-csi-driver-node-f7m87\" (UID: \"82dd9d02-85c9-415b-9db4-d4e8e189ed66\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7m87" Sep 11 08:46:22.359893 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.359877 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bb4b69fd-d8d3-4134-94c5-16faabe2ec30-multus-cni-dir\") pod \"multus-4w5tl\" (UID: \"bb4b69fd-d8d3-4134-94c5-16faabe2ec30\") " pod="openshift-multus/multus-4w5tl" Sep 11 08:46:22.359893 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.359902 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bb4b69fd-d8d3-4134-94c5-16faabe2ec30-multus-conf-dir\") pod \"multus-4w5tl\" (UID: \"bb4b69fd-d8d3-4134-94c5-16faabe2ec30\") " pod="openshift-multus/multus-4w5tl" Sep 11 08:46:22.360778 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.359929 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bb4b69fd-d8d3-4134-94c5-16faabe2ec30-multus-daemon-config\") pod \"multus-4w5tl\" (UID: \"bb4b69fd-d8d3-4134-94c5-16faabe2ec30\") " pod="openshift-multus/multus-4w5tl" Sep 11 08:46:22.360778 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.359956 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/82dd9d02-85c9-415b-9db4-d4e8e189ed66-kubelet-dir\") pod \"aws-ebs-csi-driver-node-f7m87\" (UID: \"82dd9d02-85c9-415b-9db4-d4e8e189ed66\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7m87" Sep 11 08:46:22.360778 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.359980 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/82dd9d02-85c9-415b-9db4-d4e8e189ed66-sys-fs\") pod \"aws-ebs-csi-driver-node-f7m87\" (UID: \"82dd9d02-85c9-415b-9db4-d4e8e189ed66\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7m87" Sep 11 08:46:22.360778 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.360021 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bb4b69fd-d8d3-4134-94c5-16faabe2ec30-cnibin\") pod \"multus-4w5tl\" (UID: \"bb4b69fd-d8d3-4134-94c5-16faabe2ec30\") " pod="openshift-multus/multus-4w5tl" Sep 11 08:46:22.360778 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.360046 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bb4b69fd-d8d3-4134-94c5-16faabe2ec30-cni-binary-copy\") pod \"multus-4w5tl\" (UID: \"bb4b69fd-d8d3-4134-94c5-16faabe2ec30\") " pod="openshift-multus/multus-4w5tl" Sep 11 08:46:22.360778 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.360070 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/82dd9d02-85c9-415b-9db4-d4e8e189ed66-etc-selinux\") pod \"aws-ebs-csi-driver-node-f7m87\" (UID: \"82dd9d02-85c9-415b-9db4-d4e8e189ed66\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7m87" Sep 11 08:46:22.360778 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.360203 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/82dd9d02-85c9-415b-9db4-d4e8e189ed66-etc-selinux\") pod \"aws-ebs-csi-driver-node-f7m87\" (UID: \"82dd9d02-85c9-415b-9db4-d4e8e189ed66\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7m87" Sep 11 08:46:22.360778 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.360273 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/82687de7-ba90-4516-99cc-43f545df75bd-hosts-file\") pod \"node-resolver-4vdzw\" (UID: \"82687de7-ba90-4516-99cc-43f545df75bd\") " pod="openshift-dns/node-resolver-4vdzw" Sep 11 08:46:22.360778 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.360327 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bb4b69fd-d8d3-4134-94c5-16faabe2ec30-host-run-k8s-cni-cncf-io\") pod \"multus-4w5tl\" (UID: \"bb4b69fd-d8d3-4134-94c5-16faabe2ec30\") " pod="openshift-multus/multus-4w5tl" Sep 11 08:46:22.360778 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.360412 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a370ef3a-98cb-4251-86b9-cb3155c587a5-kubelet-config\") pod \"global-pull-secret-syncer-v7fx2\" (UID: \"a370ef3a-98cb-4251-86b9-cb3155c587a5\") " pod="kube-system/global-pull-secret-syncer-v7fx2" Sep 11 08:46:22.360778 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.360446 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bb4b69fd-d8d3-4134-94c5-16faabe2ec30-host-var-lib-cni-bin\") pod \"multus-4w5tl\" (UID: \"bb4b69fd-d8d3-4134-94c5-16faabe2ec30\") " pod="openshift-multus/multus-4w5tl" Sep 11 08:46:22.360778 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.360479 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/82dd9d02-85c9-415b-9db4-d4e8e189ed66-registration-dir\") pod \"aws-ebs-csi-driver-node-f7m87\" (UID: \"82dd9d02-85c9-415b-9db4-d4e8e189ed66\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7m87" Sep 11 08:46:22.360778 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:22.360551 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 08:46:22.360778 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:22.360602 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a370ef3a-98cb-4251-86b9-cb3155c587a5-original-pull-secret podName:a370ef3a-98cb-4251-86b9-cb3155c587a5 nodeName:}" failed. No retries permitted until 2026-09-11 08:46:22.860587669 +0000 UTC m=+3.135163343 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a370ef3a-98cb-4251-86b9-cb3155c587a5-original-pull-secret") pod "global-pull-secret-syncer-v7fx2" (UID: "a370ef3a-98cb-4251-86b9-cb3155c587a5") : object "kube-system"/"original-pull-secret" not registered Sep 11 08:46:22.360778 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.360785 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bb4b69fd-d8d3-4134-94c5-16faabe2ec30-system-cni-dir\") pod \"multus-4w5tl\" (UID: \"bb4b69fd-d8d3-4134-94c5-16faabe2ec30\") " pod="openshift-multus/multus-4w5tl" Sep 11 08:46:22.361523 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.360825 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bb4b69fd-d8d3-4134-94c5-16faabe2ec30-host-var-lib-kubelet\") pod \"multus-4w5tl\" (UID: \"bb4b69fd-d8d3-4134-94c5-16faabe2ec30\") " pod="openshift-multus/multus-4w5tl" Sep 11 08:46:22.361523 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.360836 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82687de7-ba90-4516-99cc-43f545df75bd-tmp-dir\") pod \"node-resolver-4vdzw\" (UID: \"82687de7-ba90-4516-99cc-43f545df75bd\") " pod="openshift-dns/node-resolver-4vdzw" Sep 11 08:46:22.361523 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.360855 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bb4b69fd-d8d3-4134-94c5-16faabe2ec30-host-var-lib-cni-multus\") pod \"multus-4w5tl\" (UID: \"bb4b69fd-d8d3-4134-94c5-16faabe2ec30\") " pod="openshift-multus/multus-4w5tl" Sep 11 08:46:22.361523 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.360910 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bb4b69fd-d8d3-4134-94c5-16faabe2ec30-multus-socket-dir-parent\") pod \"multus-4w5tl\" (UID: \"bb4b69fd-d8d3-4134-94c5-16faabe2ec30\") " pod="openshift-multus/multus-4w5tl" Sep 11 08:46:22.361523 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.361006 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/82dd9d02-85c9-415b-9db4-d4e8e189ed66-socket-dir\") pod \"aws-ebs-csi-driver-node-f7m87\" (UID: \"82dd9d02-85c9-415b-9db4-d4e8e189ed66\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7m87" Sep 11 08:46:22.361523 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.361027 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bb4b69fd-d8d3-4134-94c5-16faabe2ec30-host-run-netns\") pod \"multus-4w5tl\" (UID: \"bb4b69fd-d8d3-4134-94c5-16faabe2ec30\") " pod="openshift-multus/multus-4w5tl" Sep 11 08:46:22.361523 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.361067 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bb4b69fd-d8d3-4134-94c5-16faabe2ec30-multus-cni-dir\") pod \"multus-4w5tl\" (UID: \"bb4b69fd-d8d3-4134-94c5-16faabe2ec30\") " pod="openshift-multus/multus-4w5tl" Sep 11 08:46:22.361523 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.361068 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/82dd9d02-85c9-415b-9db4-d4e8e189ed66-device-dir\") pod \"aws-ebs-csi-driver-node-f7m87\" (UID: \"82dd9d02-85c9-415b-9db4-d4e8e189ed66\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7m87" Sep 11 08:46:22.361523 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.361122 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/82dd9d02-85c9-415b-9db4-d4e8e189ed66-kubelet-dir\") pod \"aws-ebs-csi-driver-node-f7m87\" (UID: \"82dd9d02-85c9-415b-9db4-d4e8e189ed66\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7m87" Sep 11 08:46:22.361523 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.361157 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bb4b69fd-d8d3-4134-94c5-16faabe2ec30-multus-conf-dir\") pod \"multus-4w5tl\" (UID: \"bb4b69fd-d8d3-4134-94c5-16faabe2ec30\") " pod="openshift-multus/multus-4w5tl" Sep 11 08:46:22.361523 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.361207 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/82dd9d02-85c9-415b-9db4-d4e8e189ed66-sys-fs\") pod \"aws-ebs-csi-driver-node-f7m87\" (UID: \"82dd9d02-85c9-415b-9db4-d4e8e189ed66\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7m87" Sep 11 08:46:22.361523 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.361278 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bb4b69fd-d8d3-4134-94c5-16faabe2ec30-cnibin\") pod \"multus-4w5tl\" (UID: \"bb4b69fd-d8d3-4134-94c5-16faabe2ec30\") " pod="openshift-multus/multus-4w5tl" Sep 11 08:46:22.361980 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.361589 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bb4b69fd-d8d3-4134-94c5-16faabe2ec30-multus-daemon-config\") pod \"multus-4w5tl\" (UID: \"bb4b69fd-d8d3-4134-94c5-16faabe2ec30\") " pod="openshift-multus/multus-4w5tl" Sep 11 08:46:22.361980 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.361747 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bb4b69fd-d8d3-4134-94c5-16faabe2ec30-cni-binary-copy\") pod \"multus-4w5tl\" (UID: \"bb4b69fd-d8d3-4134-94c5-16faabe2ec30\") " pod="openshift-multus/multus-4w5tl" Sep 11 08:46:22.369513 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.369492 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzvn7\" (UniqueName: \"kubernetes.io/projected/82dd9d02-85c9-415b-9db4-d4e8e189ed66-kube-api-access-fzvn7\") pod \"aws-ebs-csi-driver-node-f7m87\" (UID: \"82dd9d02-85c9-415b-9db4-d4e8e189ed66\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7m87" Sep 11 08:46:22.370643 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.370589 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q96vm\" (UniqueName: \"kubernetes.io/projected/bb4b69fd-d8d3-4134-94c5-16faabe2ec30-kube-api-access-q96vm\") pod \"multus-4w5tl\" (UID: \"bb4b69fd-d8d3-4134-94c5-16faabe2ec30\") " pod="openshift-multus/multus-4w5tl" Sep 11 08:46:22.370829 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.370809 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcbdh\" (UniqueName: \"kubernetes.io/projected/82687de7-ba90-4516-99cc-43f545df75bd-kube-api-access-tcbdh\") pod \"node-resolver-4vdzw\" (UID: \"82687de7-ba90-4516-99cc-43f545df75bd\") " pod="openshift-dns/node-resolver-4vdzw" Sep 11 08:46:22.439886 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.439851 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-n56xz" Sep 11 08:46:22.447263 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.447244 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-67p7z" Sep 11 08:46:22.457995 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.457974 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-v5rw5" Sep 11 08:46:22.463727 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.463706 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7m87" Sep 11 08:46:22.473318 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.473296 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-k599z" Sep 11 08:46:22.480896 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.480879 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-p5sg8" Sep 11 08:46:22.488488 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.488472 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-lhkkf" Sep 11 08:46:22.495032 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.495016 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-4vdzw" Sep 11 08:46:22.500557 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.500540 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4w5tl" Sep 11 08:46:22.762814 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.762777 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e78a69db-9d98-4923-b082-ccb30b77cc9e-metrics-certs\") pod \"network-metrics-daemon-ntkhj\" (UID: \"e78a69db-9d98-4923-b082-ccb30b77cc9e\") " pod="openshift-multus/network-metrics-daemon-ntkhj" Sep 11 08:46:22.763002 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:22.762925 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 08:46:22.763002 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:22.762993 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e78a69db-9d98-4923-b082-ccb30b77cc9e-metrics-certs podName:e78a69db-9d98-4923-b082-ccb30b77cc9e nodeName:}" failed. No retries permitted until 2026-09-11 08:46:23.762976946 +0000 UTC m=+4.037552620 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e78a69db-9d98-4923-b082-ccb30b77cc9e-metrics-certs") pod "network-metrics-daemon-ntkhj" (UID: "e78a69db-9d98-4923-b082-ccb30b77cc9e") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 08:46:22.863351 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.863307 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwc96\" (UniqueName: \"kubernetes.io/projected/12ac768c-2c6a-43e7-9bd3-44679781ebd7-kube-api-access-zwc96\") pod \"network-check-target-82xp9\" (UID: \"12ac768c-2c6a-43e7-9bd3-44679781ebd7\") " pod="openshift-network-diagnostics/network-check-target-82xp9" Sep 11 08:46:22.863564 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:22.863374 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a370ef3a-98cb-4251-86b9-cb3155c587a5-original-pull-secret\") pod \"global-pull-secret-syncer-v7fx2\" (UID: \"a370ef3a-98cb-4251-86b9-cb3155c587a5\") " pod="kube-system/global-pull-secret-syncer-v7fx2" Sep 11 08:46:22.863564 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:22.863482 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 08:46:22.863564 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:22.863514 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 08:46:22.863564 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:22.863543 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 08:46:22.863564 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:22.863536 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a370ef3a-98cb-4251-86b9-cb3155c587a5-original-pull-secret podName:a370ef3a-98cb-4251-86b9-cb3155c587a5 nodeName:}" failed. No retries permitted until 2026-09-11 08:46:23.863523089 +0000 UTC m=+4.138098763 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a370ef3a-98cb-4251-86b9-cb3155c587a5-original-pull-secret") pod "global-pull-secret-syncer-v7fx2" (UID: "a370ef3a-98cb-4251-86b9-cb3155c587a5") : object "kube-system"/"original-pull-secret" not registered Sep 11 08:46:22.863564 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:22.863559 2572 projected.go:196] Error preparing data for projected volume kube-api-access-zwc96 for pod openshift-network-diagnostics/network-check-target-82xp9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 08:46:22.863837 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:22.863611 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/12ac768c-2c6a-43e7-9bd3-44679781ebd7-kube-api-access-zwc96 podName:12ac768c-2c6a-43e7-9bd3-44679781ebd7 nodeName:}" failed. No retries permitted until 2026-09-11 08:46:23.863595042 +0000 UTC m=+4.138170737 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-zwc96" (UniqueName: "kubernetes.io/projected/12ac768c-2c6a-43e7-9bd3-44679781ebd7-kube-api-access-zwc96") pod "network-check-target-82xp9" (UID: "12ac768c-2c6a-43e7-9bd3-44679781ebd7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 08:46:23.195182 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:23.195146 2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 08:41:22 +0000 UTC" deadline="2028-05-11 17:36:18.761647641 +0000 UTC" Sep 11 08:46:23.195182 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:23.195175 2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14600h49m55.566476487s" Sep 11 08:46:23.236170 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:23.236138 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb6eba26_53e8_4946_9491_cf154a8286f5.slice/crio-02c6505bc59e1f88bb393ded943f62ad17cb3b5ba59c1017a30055eadbecd3ed WatchSource:0}: Error finding container 02c6505bc59e1f88bb393ded943f62ad17cb3b5ba59c1017a30055eadbecd3ed: Status 404 returned error can't find the container with id 02c6505bc59e1f88bb393ded943f62ad17cb3b5ba59c1017a30055eadbecd3ed Sep 11 08:46:23.244402 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:23.244376 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c0c8738_0dd8_441e_8c6e_3ff6009337ca.slice/crio-c7e393fdee835f0ee9cd84e01fa3cc49d401197fad3d12750d0f52dff69c8196 WatchSource:0}: Error finding container c7e393fdee835f0ee9cd84e01fa3cc49d401197fad3d12750d0f52dff69c8196: Status 404 returned error can't find the container with id c7e393fdee835f0ee9cd84e01fa3cc49d401197fad3d12750d0f52dff69c8196 Sep 11 08:46:23.245440 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:23.245407 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82dd9d02_85c9_415b_9db4_d4e8e189ed66.slice/crio-769bbcd3420666cbead50eea36a1c271c3a2e191324ca6021d1420a3452f3632 WatchSource:0}: Error finding container 769bbcd3420666cbead50eea36a1c271c3a2e191324ca6021d1420a3452f3632: Status 404 returned error can't find the container with id 769bbcd3420666cbead50eea36a1c271c3a2e191324ca6021d1420a3452f3632 Sep 11 08:46:23.246184 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:23.246151 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb4b69fd_d8d3_4134_94c5_16faabe2ec30.slice/crio-66dad10995c2c7d4814818cd83b5ac7a80d515eef94c1cd46ff90a9cc2d350fa WatchSource:0}: Error finding container 66dad10995c2c7d4814818cd83b5ac7a80d515eef94c1cd46ff90a9cc2d350fa: Status 404 returned error can't find the container with id 66dad10995c2c7d4814818cd83b5ac7a80d515eef94c1cd46ff90a9cc2d350fa Sep 11 08:46:23.246663 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:23.246501 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ntkhj" Sep 11 08:46:23.246663 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:23.246617 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ntkhj" podUID="e78a69db-9d98-4923-b082-ccb30b77cc9e" Sep 11 08:46:23.248107 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:23.248083 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46676d4c_0012_4617_9fe5_73d33015eb8c.slice/crio-b9f46ec303c4878025ffd7f479b03e9578acb4423251d8c4a06d33fa2906e982 WatchSource:0}: Error finding container b9f46ec303c4878025ffd7f479b03e9578acb4423251d8c4a06d33fa2906e982: Status 404 returned error can't find the container with id b9f46ec303c4878025ffd7f479b03e9578acb4423251d8c4a06d33fa2906e982 Sep 11 08:46:23.249989 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:23.249951 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83409be8_1bb2_43dc_af76_d3d74e569101.slice/crio-578d615967fbf7f760fd358f8478fd4e611c31640a57d5525ffd26b8e5c83917 WatchSource:0}: Error finding container 578d615967fbf7f760fd358f8478fd4e611c31640a57d5525ffd26b8e5c83917: Status 404 returned error can't find the container with id 578d615967fbf7f760fd358f8478fd4e611c31640a57d5525ffd26b8e5c83917 Sep 11 08:46:23.251463 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:23.251436 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod510eab66_1dcf_4ffc_983d_d570fb9e9fda.slice/crio-45c6cc9fdd6d0e9a26e799eeaab9645eef90a5a392ae437c3ef599a8d07f8398 WatchSource:0}: Error finding container 45c6cc9fdd6d0e9a26e799eeaab9645eef90a5a392ae437c3ef599a8d07f8398: Status 404 returned error can't find the container with id 45c6cc9fdd6d0e9a26e799eeaab9645eef90a5a392ae437c3ef599a8d07f8398 Sep 11 08:46:23.253581 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:23.253558 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c0c64bd_73f4_4cb4_b660_c7232ff8e573.slice/crio-6e5b814fa1ac75264b9af555dfbc75f998e923cf621cd606ca10167500bc4c4c WatchSource:0}: Error finding container 6e5b814fa1ac75264b9af555dfbc75f998e923cf621cd606ca10167500bc4c4c: Status 404 returned error can't find the container with id 6e5b814fa1ac75264b9af555dfbc75f998e923cf621cd606ca10167500bc4c4c Sep 11 08:46:23.258162 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:23.258134 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-p5sg8" event={"ID":"2c0c8738-0dd8-441e-8c6e-3ff6009337ca","Type":"ContainerStarted","Data":"c7e393fdee835f0ee9cd84e01fa3cc49d401197fad3d12750d0f52dff69c8196"} Sep 11 08:46:23.259392 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:23.259364 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k599z" event={"ID":"0c0c64bd-73f4-4cb4-b660-c7232ff8e573","Type":"ContainerStarted","Data":"6e5b814fa1ac75264b9af555dfbc75f998e923cf621cd606ca10167500bc4c4c"} Sep 11 08:46:23.260591 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:23.260570 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-v5rw5" event={"ID":"510eab66-1dcf-4ffc-983d-d570fb9e9fda","Type":"ContainerStarted","Data":"45c6cc9fdd6d0e9a26e799eeaab9645eef90a5a392ae437c3ef599a8d07f8398"} Sep 11 08:46:23.261863 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:23.261842 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lhkkf" event={"ID":"83409be8-1bb2-43dc-af76-d3d74e569101","Type":"ContainerStarted","Data":"578d615967fbf7f760fd358f8478fd4e611c31640a57d5525ffd26b8e5c83917"} Sep 11 08:46:23.263112 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:23.263091 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-4vdzw" event={"ID":"82687de7-ba90-4516-99cc-43f545df75bd","Type":"ContainerStarted","Data":"23f67172e7b7fce70380374e5008b6965042081b91e0835249f092bbed0becc3"} Sep 11 08:46:23.264528 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:23.264502 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4w5tl" event={"ID":"bb4b69fd-d8d3-4134-94c5-16faabe2ec30","Type":"ContainerStarted","Data":"66dad10995c2c7d4814818cd83b5ac7a80d515eef94c1cd46ff90a9cc2d350fa"} Sep 11 08:46:23.265662 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:23.265642 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-67p7z" event={"ID":"eb6eba26-53e8-4946-9491-cf154a8286f5","Type":"ContainerStarted","Data":"02c6505bc59e1f88bb393ded943f62ad17cb3b5ba59c1017a30055eadbecd3ed"} Sep 11 08:46:23.266899 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:23.266878 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-n56xz" event={"ID":"46676d4c-0012-4617-9fe5-73d33015eb8c","Type":"ContainerStarted","Data":"b9f46ec303c4878025ffd7f479b03e9578acb4423251d8c4a06d33fa2906e982"} Sep 11 08:46:23.268208 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:23.268188 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7m87" event={"ID":"82dd9d02-85c9-415b-9db4-d4e8e189ed66","Type":"ContainerStarted","Data":"769bbcd3420666cbead50eea36a1c271c3a2e191324ca6021d1420a3452f3632"} Sep 11 08:46:23.768727 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:23.768610 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e78a69db-9d98-4923-b082-ccb30b77cc9e-metrics-certs\") pod \"network-metrics-daemon-ntkhj\" (UID: \"e78a69db-9d98-4923-b082-ccb30b77cc9e\") " pod="openshift-multus/network-metrics-daemon-ntkhj" Sep 11 08:46:23.768870 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:23.768785 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 08:46:23.768870 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:23.768850 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e78a69db-9d98-4923-b082-ccb30b77cc9e-metrics-certs podName:e78a69db-9d98-4923-b082-ccb30b77cc9e nodeName:}" failed. No retries permitted until 2026-09-11 08:46:25.768831474 +0000 UTC m=+6.043407152 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e78a69db-9d98-4923-b082-ccb30b77cc9e-metrics-certs") pod "network-metrics-daemon-ntkhj" (UID: "e78a69db-9d98-4923-b082-ccb30b77cc9e") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 08:46:23.869558 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:23.869399 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwc96\" (UniqueName: \"kubernetes.io/projected/12ac768c-2c6a-43e7-9bd3-44679781ebd7-kube-api-access-zwc96\") pod \"network-check-target-82xp9\" (UID: \"12ac768c-2c6a-43e7-9bd3-44679781ebd7\") " pod="openshift-network-diagnostics/network-check-target-82xp9" Sep 11 08:46:23.869558 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:23.869470 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a370ef3a-98cb-4251-86b9-cb3155c587a5-original-pull-secret\") pod \"global-pull-secret-syncer-v7fx2\" (UID: \"a370ef3a-98cb-4251-86b9-cb3155c587a5\") " pod="kube-system/global-pull-secret-syncer-v7fx2" Sep 11 08:46:23.869774 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:23.869624 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 08:46:23.869774 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:23.869689 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a370ef3a-98cb-4251-86b9-cb3155c587a5-original-pull-secret podName:a370ef3a-98cb-4251-86b9-cb3155c587a5 nodeName:}" failed. No retries permitted until 2026-09-11 08:46:25.869669751 +0000 UTC m=+6.144245427 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a370ef3a-98cb-4251-86b9-cb3155c587a5-original-pull-secret") pod "global-pull-secret-syncer-v7fx2" (UID: "a370ef3a-98cb-4251-86b9-cb3155c587a5") : object "kube-system"/"original-pull-secret" not registered Sep 11 08:46:23.870154 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:23.870134 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 08:46:23.870252 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:23.870163 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 08:46:23.870252 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:23.870177 2572 projected.go:196] Error preparing data for projected volume kube-api-access-zwc96 for pod openshift-network-diagnostics/network-check-target-82xp9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 08:46:23.870252 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:23.870210 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/12ac768c-2c6a-43e7-9bd3-44679781ebd7-kube-api-access-zwc96 podName:12ac768c-2c6a-43e7-9bd3-44679781ebd7 nodeName:}" failed. No retries permitted until 2026-09-11 08:46:25.870199821 +0000 UTC m=+6.144775494 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-zwc96" (UniqueName: "kubernetes.io/projected/12ac768c-2c6a-43e7-9bd3-44679781ebd7-kube-api-access-zwc96") pod "network-check-target-82xp9" (UID: "12ac768c-2c6a-43e7-9bd3-44679781ebd7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 08:46:24.195639 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:24.195575 2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 08:41:22 +0000 UTC" deadline="2028-06-28 18:28:05.662792378 +0000 UTC" Sep 11 08:46:24.195639 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:24.195638 2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15753h41m41.467158699s" Sep 11 08:46:24.247075 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:24.247040 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-82xp9" Sep 11 08:46:24.247289 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:24.247176 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-82xp9" podUID="12ac768c-2c6a-43e7-9bd3-44679781ebd7" Sep 11 08:46:24.247621 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:24.247601 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v7fx2" Sep 11 08:46:24.247727 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:24.247706 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v7fx2" podUID="a370ef3a-98cb-4251-86b9-cb3155c587a5" Sep 11 08:46:24.284959 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:24.284924 2572 generic.go:356] "Generic (PLEG): container finished" podID="3e81cc2e86491c3258cb0c986cc8b166" containerID="73668fd5de81120cc7345920625aac9d4bc120daffc4c5954711ec66ef522e7e" exitCode=0 Sep 11 08:46:24.285117 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:24.285016 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-166.ec2.internal" event={"ID":"3e81cc2e86491c3258cb0c986cc8b166","Type":"ContainerDied","Data":"73668fd5de81120cc7345920625aac9d4bc120daffc4c5954711ec66ef522e7e"} Sep 11 08:46:24.292871 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:24.292582 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-166.ec2.internal" event={"ID":"1f3dec62f93374a659aa01a42bf55935","Type":"ContainerStarted","Data":"7855d92e34a9b6823d3aa460d3ff196aea20e1443b8577e14c3126b6e189a605"} Sep 11 08:46:25.248029 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:25.247519 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ntkhj" Sep 11 08:46:25.248029 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:25.247675 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ntkhj" podUID="e78a69db-9d98-4923-b082-ccb30b77cc9e" Sep 11 08:46:25.310602 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:25.310518 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-166.ec2.internal" event={"ID":"3e81cc2e86491c3258cb0c986cc8b166","Type":"ContainerStarted","Data":"3f1e59b16c63c59dce2b8c48f5db57000857cce2979916f86db21021a40d4448"} Sep 11 08:46:25.325498 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:25.324458 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-166.ec2.internal" podStartSLOduration=4.324438699 podStartE2EDuration="4.324438699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 08:46:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 08:46:24.317662546 +0000 UTC m=+4.592238240" watchObservedRunningTime="2026-09-11 08:46:25.324438699 +0000 UTC m=+5.599014399" Sep 11 08:46:25.787383 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:25.787345 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e78a69db-9d98-4923-b082-ccb30b77cc9e-metrics-certs\") pod \"network-metrics-daemon-ntkhj\" (UID: \"e78a69db-9d98-4923-b082-ccb30b77cc9e\") " pod="openshift-multus/network-metrics-daemon-ntkhj" Sep 11 08:46:25.787595 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:25.787558 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 08:46:25.787653 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:25.787623 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e78a69db-9d98-4923-b082-ccb30b77cc9e-metrics-certs podName:e78a69db-9d98-4923-b082-ccb30b77cc9e nodeName:}" failed. No retries permitted until 2026-09-11 08:46:29.7876041 +0000 UTC m=+10.062179791 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e78a69db-9d98-4923-b082-ccb30b77cc9e-metrics-certs") pod "network-metrics-daemon-ntkhj" (UID: "e78a69db-9d98-4923-b082-ccb30b77cc9e") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 08:46:25.889139 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:25.888324 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a370ef3a-98cb-4251-86b9-cb3155c587a5-original-pull-secret\") pod \"global-pull-secret-syncer-v7fx2\" (UID: \"a370ef3a-98cb-4251-86b9-cb3155c587a5\") " pod="kube-system/global-pull-secret-syncer-v7fx2" Sep 11 08:46:25.889139 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:25.888409 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwc96\" (UniqueName: \"kubernetes.io/projected/12ac768c-2c6a-43e7-9bd3-44679781ebd7-kube-api-access-zwc96\") pod \"network-check-target-82xp9\" (UID: \"12ac768c-2c6a-43e7-9bd3-44679781ebd7\") " pod="openshift-network-diagnostics/network-check-target-82xp9" Sep 11 08:46:25.889139 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:25.888565 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 08:46:25.889139 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:25.888583 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 08:46:25.889139 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:25.888598 2572 projected.go:196] Error preparing data for projected volume kube-api-access-zwc96 for pod openshift-network-diagnostics/network-check-target-82xp9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 08:46:25.889139 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:25.888656 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/12ac768c-2c6a-43e7-9bd3-44679781ebd7-kube-api-access-zwc96 podName:12ac768c-2c6a-43e7-9bd3-44679781ebd7 nodeName:}" failed. No retries permitted until 2026-09-11 08:46:29.888637327 +0000 UTC m=+10.163213004 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-zwc96" (UniqueName: "kubernetes.io/projected/12ac768c-2c6a-43e7-9bd3-44679781ebd7-kube-api-access-zwc96") pod "network-check-target-82xp9" (UID: "12ac768c-2c6a-43e7-9bd3-44679781ebd7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 08:46:25.889139 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:25.889039 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 08:46:25.889139 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:25.889089 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a370ef3a-98cb-4251-86b9-cb3155c587a5-original-pull-secret podName:a370ef3a-98cb-4251-86b9-cb3155c587a5 nodeName:}" failed. No retries permitted until 2026-09-11 08:46:29.889074593 +0000 UTC m=+10.163650268 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a370ef3a-98cb-4251-86b9-cb3155c587a5-original-pull-secret") pod "global-pull-secret-syncer-v7fx2" (UID: "a370ef3a-98cb-4251-86b9-cb3155c587a5") : object "kube-system"/"original-pull-secret" not registered Sep 11 08:46:26.247450 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:26.247417 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-82xp9" Sep 11 08:46:26.247627 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:26.247541 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-82xp9" podUID="12ac768c-2c6a-43e7-9bd3-44679781ebd7" Sep 11 08:46:26.247627 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:26.247604 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v7fx2" Sep 11 08:46:26.247737 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:26.247676 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v7fx2" podUID="a370ef3a-98cb-4251-86b9-cb3155c587a5" Sep 11 08:46:27.247449 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:27.247415 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ntkhj" Sep 11 08:46:27.247838 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:27.247552 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ntkhj" podUID="e78a69db-9d98-4923-b082-ccb30b77cc9e" Sep 11 08:46:28.247241 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:28.247190 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v7fx2" Sep 11 08:46:28.247433 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:28.247190 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-82xp9" Sep 11 08:46:28.247433 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:28.247355 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v7fx2" podUID="a370ef3a-98cb-4251-86b9-cb3155c587a5" Sep 11 08:46:28.247760 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:28.247424 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-82xp9" podUID="12ac768c-2c6a-43e7-9bd3-44679781ebd7" Sep 11 08:46:29.247119 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:29.247082 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ntkhj" Sep 11 08:46:29.247328 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:29.247208 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ntkhj" podUID="e78a69db-9d98-4923-b082-ccb30b77cc9e" Sep 11 08:46:29.820493 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:29.820448 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e78a69db-9d98-4923-b082-ccb30b77cc9e-metrics-certs\") pod \"network-metrics-daemon-ntkhj\" (UID: \"e78a69db-9d98-4923-b082-ccb30b77cc9e\") " pod="openshift-multus/network-metrics-daemon-ntkhj" Sep 11 08:46:29.820957 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:29.820630 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 08:46:29.820957 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:29.820695 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e78a69db-9d98-4923-b082-ccb30b77cc9e-metrics-certs podName:e78a69db-9d98-4923-b082-ccb30b77cc9e nodeName:}" failed. No retries permitted until 2026-09-11 08:46:37.820674467 +0000 UTC m=+18.095250147 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e78a69db-9d98-4923-b082-ccb30b77cc9e-metrics-certs") pod "network-metrics-daemon-ntkhj" (UID: "e78a69db-9d98-4923-b082-ccb30b77cc9e") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 08:46:29.921020 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:29.920980 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwc96\" (UniqueName: \"kubernetes.io/projected/12ac768c-2c6a-43e7-9bd3-44679781ebd7-kube-api-access-zwc96\") pod \"network-check-target-82xp9\" (UID: \"12ac768c-2c6a-43e7-9bd3-44679781ebd7\") " pod="openshift-network-diagnostics/network-check-target-82xp9" Sep 11 08:46:29.921180 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:29.921042 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a370ef3a-98cb-4251-86b9-cb3155c587a5-original-pull-secret\") pod \"global-pull-secret-syncer-v7fx2\" (UID: \"a370ef3a-98cb-4251-86b9-cb3155c587a5\") " pod="kube-system/global-pull-secret-syncer-v7fx2" Sep 11 08:46:29.921180 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:29.921141 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 08:46:29.921371 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:29.921189 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a370ef3a-98cb-4251-86b9-cb3155c587a5-original-pull-secret podName:a370ef3a-98cb-4251-86b9-cb3155c587a5 nodeName:}" failed. No retries permitted until 2026-09-11 08:46:37.921176533 +0000 UTC m=+18.195752221 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a370ef3a-98cb-4251-86b9-cb3155c587a5-original-pull-secret") pod "global-pull-secret-syncer-v7fx2" (UID: "a370ef3a-98cb-4251-86b9-cb3155c587a5") : object "kube-system"/"original-pull-secret" not registered Sep 11 08:46:29.921546 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:29.921530 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 08:46:29.921603 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:29.921551 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 08:46:29.921603 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:29.921560 2572 projected.go:196] Error preparing data for projected volume kube-api-access-zwc96 for pod openshift-network-diagnostics/network-check-target-82xp9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 08:46:29.921741 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:29.921602 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/12ac768c-2c6a-43e7-9bd3-44679781ebd7-kube-api-access-zwc96 podName:12ac768c-2c6a-43e7-9bd3-44679781ebd7 nodeName:}" failed. No retries permitted until 2026-09-11 08:46:37.921588277 +0000 UTC m=+18.196163969 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-zwc96" (UniqueName: "kubernetes.io/projected/12ac768c-2c6a-43e7-9bd3-44679781ebd7-kube-api-access-zwc96") pod "network-check-target-82xp9" (UID: "12ac768c-2c6a-43e7-9bd3-44679781ebd7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 08:46:30.248042 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:30.248013 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v7fx2" Sep 11 08:46:30.248204 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:30.248170 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v7fx2" podUID="a370ef3a-98cb-4251-86b9-cb3155c587a5" Sep 11 08:46:30.248298 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:30.248237 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-82xp9" Sep 11 08:46:30.248372 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:30.248352 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-82xp9" podUID="12ac768c-2c6a-43e7-9bd3-44679781ebd7" Sep 11 08:46:31.247894 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:31.247373 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ntkhj" Sep 11 08:46:31.247894 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:31.247518 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ntkhj" podUID="e78a69db-9d98-4923-b082-ccb30b77cc9e" Sep 11 08:46:32.247030 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:32.246998 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v7fx2" Sep 11 08:46:32.247193 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:32.247130 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v7fx2" podUID="a370ef3a-98cb-4251-86b9-cb3155c587a5" Sep 11 08:46:32.247193 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:32.247162 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-82xp9" Sep 11 08:46:32.247350 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:32.247300 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-82xp9" podUID="12ac768c-2c6a-43e7-9bd3-44679781ebd7" Sep 11 08:46:33.247055 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:33.247021 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ntkhj" Sep 11 08:46:33.247517 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:33.247159 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ntkhj" podUID="e78a69db-9d98-4923-b082-ccb30b77cc9e" Sep 11 08:46:34.247253 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:34.247213 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v7fx2" Sep 11 08:46:34.247253 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:34.247241 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-82xp9" Sep 11 08:46:34.247743 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:34.247329 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v7fx2" podUID="a370ef3a-98cb-4251-86b9-cb3155c587a5" Sep 11 08:46:34.247743 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:34.247457 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-82xp9" podUID="12ac768c-2c6a-43e7-9bd3-44679781ebd7" Sep 11 08:46:35.246707 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:35.246663 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ntkhj" Sep 11 08:46:35.246859 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:35.246791 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ntkhj" podUID="e78a69db-9d98-4923-b082-ccb30b77cc9e" Sep 11 08:46:36.247283 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:36.247245 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-82xp9" Sep 11 08:46:36.247727 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:36.247386 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-82xp9" podUID="12ac768c-2c6a-43e7-9bd3-44679781ebd7" Sep 11 08:46:36.247727 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:36.247428 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v7fx2" Sep 11 08:46:36.247727 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:36.247499 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v7fx2" podUID="a370ef3a-98cb-4251-86b9-cb3155c587a5" Sep 11 08:46:37.246663 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:37.246627 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ntkhj" Sep 11 08:46:37.246909 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:37.246756 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ntkhj" podUID="e78a69db-9d98-4923-b082-ccb30b77cc9e" Sep 11 08:46:37.879671 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:37.879634 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e78a69db-9d98-4923-b082-ccb30b77cc9e-metrics-certs\") pod \"network-metrics-daemon-ntkhj\" (UID: \"e78a69db-9d98-4923-b082-ccb30b77cc9e\") " pod="openshift-multus/network-metrics-daemon-ntkhj" Sep 11 08:46:37.880028 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:37.879776 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 08:46:37.880028 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:37.879850 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e78a69db-9d98-4923-b082-ccb30b77cc9e-metrics-certs podName:e78a69db-9d98-4923-b082-ccb30b77cc9e nodeName:}" failed. No retries permitted until 2026-09-11 08:46:53.879827014 +0000 UTC m=+34.154402690 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e78a69db-9d98-4923-b082-ccb30b77cc9e-metrics-certs") pod "network-metrics-daemon-ntkhj" (UID: "e78a69db-9d98-4923-b082-ccb30b77cc9e") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 08:46:37.981020 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:37.980942 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a370ef3a-98cb-4251-86b9-cb3155c587a5-original-pull-secret\") pod \"global-pull-secret-syncer-v7fx2\" (UID: \"a370ef3a-98cb-4251-86b9-cb3155c587a5\") " pod="kube-system/global-pull-secret-syncer-v7fx2" Sep 11 08:46:37.981020 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:37.981005 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwc96\" (UniqueName: \"kubernetes.io/projected/12ac768c-2c6a-43e7-9bd3-44679781ebd7-kube-api-access-zwc96\") pod \"network-check-target-82xp9\" (UID: \"12ac768c-2c6a-43e7-9bd3-44679781ebd7\") " pod="openshift-network-diagnostics/network-check-target-82xp9" Sep 11 08:46:37.981250 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:37.981110 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 08:46:37.981250 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:37.981132 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 08:46:37.981250 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:37.981155 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 08:46:37.981250 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:37.981168 2572 projected.go:196] Error preparing data for projected volume kube-api-access-zwc96 for pod openshift-network-diagnostics/network-check-target-82xp9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 08:46:37.981250 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:37.981191 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a370ef3a-98cb-4251-86b9-cb3155c587a5-original-pull-secret podName:a370ef3a-98cb-4251-86b9-cb3155c587a5 nodeName:}" failed. No retries permitted until 2026-09-11 08:46:53.981174452 +0000 UTC m=+34.255750139 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a370ef3a-98cb-4251-86b9-cb3155c587a5-original-pull-secret") pod "global-pull-secret-syncer-v7fx2" (UID: "a370ef3a-98cb-4251-86b9-cb3155c587a5") : object "kube-system"/"original-pull-secret" not registered Sep 11 08:46:37.981250 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:37.981218 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/12ac768c-2c6a-43e7-9bd3-44679781ebd7-kube-api-access-zwc96 podName:12ac768c-2c6a-43e7-9bd3-44679781ebd7 nodeName:}" failed. No retries permitted until 2026-09-11 08:46:53.981202077 +0000 UTC m=+34.255777771 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-zwc96" (UniqueName: "kubernetes.io/projected/12ac768c-2c6a-43e7-9bd3-44679781ebd7-kube-api-access-zwc96") pod "network-check-target-82xp9" (UID: "12ac768c-2c6a-43e7-9bd3-44679781ebd7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 08:46:38.250246 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:38.250159 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v7fx2" Sep 11 08:46:38.250389 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:38.250292 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v7fx2" podUID="a370ef3a-98cb-4251-86b9-cb3155c587a5" Sep 11 08:46:38.250389 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:38.250342 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-82xp9" Sep 11 08:46:38.250483 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:38.250432 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-82xp9" podUID="12ac768c-2c6a-43e7-9bd3-44679781ebd7" Sep 11 08:46:39.246980 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:39.246945 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ntkhj" Sep 11 08:46:39.247414 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:39.247086 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ntkhj" podUID="e78a69db-9d98-4923-b082-ccb30b77cc9e" Sep 11 08:46:40.250416 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:40.250380 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v7fx2" Sep 11 08:46:40.250844 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:40.250477 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v7fx2" podUID="a370ef3a-98cb-4251-86b9-cb3155c587a5" Sep 11 08:46:40.250844 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:40.250539 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-82xp9" Sep 11 08:46:40.250844 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:40.250648 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-82xp9" podUID="12ac768c-2c6a-43e7-9bd3-44679781ebd7" Sep 11 08:46:41.247477 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:41.247445 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ntkhj" Sep 11 08:46:41.247683 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:41.247592 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ntkhj" podUID="e78a69db-9d98-4923-b082-ccb30b77cc9e" Sep 11 08:46:42.250652 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:42.250445 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v7fx2" Sep 11 08:46:42.251417 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:42.250515 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-82xp9" Sep 11 08:46:42.251417 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:42.250752 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v7fx2" podUID="a370ef3a-98cb-4251-86b9-cb3155c587a5" Sep 11 08:46:42.251417 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:42.250808 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-82xp9" podUID="12ac768c-2c6a-43e7-9bd3-44679781ebd7" Sep 11 08:46:42.344657 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:42.344573 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lhkkf" event={"ID":"83409be8-1bb2-43dc-af76-d3d74e569101","Type":"ContainerStarted","Data":"b65d6a08dbdc81e75172cfa807ef67ff366b3a409ad2938b0d15df60b4ed8758"} Sep 11 08:46:42.344795 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:42.344656 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lhkkf" event={"ID":"83409be8-1bb2-43dc-af76-d3d74e569101","Type":"ContainerStarted","Data":"d1e621cc22dcd1bacaf9ce519528ae07b4dc6acab192444b36bdc28d11e5b3c8"} Sep 11 08:46:42.344795 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:42.344703 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lhkkf" event={"ID":"83409be8-1bb2-43dc-af76-d3d74e569101","Type":"ContainerStarted","Data":"863922dca5ef17f87349c573bf2c724bd59f4db88f7b67e4270d31cfb78c1cb1"} Sep 11 08:46:42.344795 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:42.344738 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lhkkf" event={"ID":"83409be8-1bb2-43dc-af76-d3d74e569101","Type":"ContainerStarted","Data":"f1513d6af6bfefaeb62896c1616985af44912240cd74c620a132c17639b76f15"} Sep 11 08:46:42.344795 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:42.344785 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lhkkf" event={"ID":"83409be8-1bb2-43dc-af76-d3d74e569101","Type":"ContainerStarted","Data":"05a3aab499ec6b59b79cd039123e951d1c8ce0b8c813b8dd93b8ff5fd0396ab3"} Sep 11 08:46:42.344985 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:42.344831 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lhkkf" event={"ID":"83409be8-1bb2-43dc-af76-d3d74e569101","Type":"ContainerStarted","Data":"86ce477750339b20e87d07f8a1ed778665e422dd9f964ce3cda18c77b86465af"} Sep 11 08:46:42.345894 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:42.345853 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-4vdzw" event={"ID":"82687de7-ba90-4516-99cc-43f545df75bd","Type":"ContainerStarted","Data":"9a3ff6f8e3136c3739ce46159e968f8c3f718c9ffb95e042824694e0dc167293"} Sep 11 08:46:42.347328 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:42.347305 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4w5tl" event={"ID":"bb4b69fd-d8d3-4134-94c5-16faabe2ec30","Type":"ContainerStarted","Data":"a404766c7f0cdc1e0f40137108f8d466987c4486150ce9c4f074d4c31663579e"} Sep 11 08:46:42.348767 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:42.348741 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-67p7z" event={"ID":"eb6eba26-53e8-4946-9491-cf154a8286f5","Type":"ContainerStarted","Data":"f9baf33fa219fa3a9f0baf5d8669e67f5007e1a236d9e1d078f4e97fbd305ba0"} Sep 11 08:46:42.350249 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:42.350209 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-n56xz" event={"ID":"46676d4c-0012-4617-9fe5-73d33015eb8c","Type":"ContainerStarted","Data":"cb212b7aa8480ead4bddf2f10d7180ebf3e08f6098f1ce518713e0c91f63aee7"} Sep 11 08:46:42.351595 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:42.351572 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7m87" event={"ID":"82dd9d02-85c9-415b-9db4-d4e8e189ed66","Type":"ContainerStarted","Data":"691fe8166b7f40f7f1a505819c27791ded662639073c5703b00b52617d3736cf"} Sep 11 08:46:42.352956 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:42.352939 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-p5sg8" event={"ID":"2c0c8738-0dd8-441e-8c6e-3ff6009337ca","Type":"ContainerStarted","Data":"6a7894874409b799d771a37bc23b8053ec74e4c58a1777e6322dee87c1cb4895"} Sep 11 08:46:42.354420 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:42.354399 2572 generic.go:356] "Generic (PLEG): container finished" podID="0c0c64bd-73f4-4cb4-b660-c7232ff8e573" containerID="c5ce9d6371dd6800d22adc9e72f0b85f36189a3bc2f5c58db5f0a3f3d09502db" exitCode=0 Sep 11 08:46:42.354499 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:42.354426 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k599z" event={"ID":"0c0c64bd-73f4-4cb4-b660-c7232ff8e573","Type":"ContainerDied","Data":"c5ce9d6371dd6800d22adc9e72f0b85f36189a3bc2f5c58db5f0a3f3d09502db"} Sep 11 08:46:42.359890 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:42.359853 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-166.ec2.internal" podStartSLOduration=21.359842375 podStartE2EDuration="21.359842375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 08:46:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 08:46:25.325484414 +0000 UTC m=+5.600060113" watchObservedRunningTime="2026-09-11 08:46:42.359842375 +0000 UTC m=+22.634418071" Sep 11 08:46:42.360249 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:42.360208 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-4vdzw" podStartSLOduration=4.157347903 podStartE2EDuration="22.360202385s" totalImagesPullingTime="18.202854482s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 08:46:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 08:46:42.359534433 +0000 UTC m=+22.634110128" watchObservedRunningTime="2026-09-11 08:46:42.360202385 +0000 UTC m=+22.634778080" Sep 11 08:46:42.375628 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:42.375596 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-4w5tl" podStartSLOduration=4.122854138 podStartE2EDuration="22.375584395s" totalImagesPullingTime="18.252730257s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 08:46:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 08:46:42.375457022 +0000 UTC m=+22.650032718" watchObservedRunningTime="2026-09-11 08:46:42.375584395 +0000 UTC m=+22.650160090" Sep 11 08:46:42.391623 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:42.391588 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-67p7z" podStartSLOduration=4.176930083 podStartE2EDuration="22.391576054s" totalImagesPullingTime="18.214645971s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 08:46:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 08:46:42.391150111 +0000 UTC m=+22.665725806" watchObservedRunningTime="2026-09-11 08:46:42.391576054 +0000 UTC m=+22.666151750" Sep 11 08:46:42.456235 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:42.456169 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-p5sg8" podStartSLOduration=7.974356004 podStartE2EDuration="22.45615493s" totalImagesPullingTime="14.481798926s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 08:46:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 08:46:42.438085204 +0000 UTC m=+22.712660900" watchObservedRunningTime="2026-09-11 08:46:42.45615493 +0000 UTC m=+22.730730626" Sep 11 08:46:42.456379 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:42.456310 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-n56xz" podStartSLOduration=4.242370175 podStartE2EDuration="22.456299394s" totalImagesPullingTime="18.213929219s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 08:46:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 08:46:42.456145093 +0000 UTC m=+22.730720790" watchObservedRunningTime="2026-09-11 08:46:42.456299394 +0000 UTC m=+22.730875090" Sep 11 08:46:42.990862 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:42.990839 2572 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 11 08:46:43.206608 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:43.206509 2572 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-11T08:46:42.990857923Z","UUID":"f5ff6416-0e69-4fdf-ab44-20cd84106bd4","Handler":null,"Name":"","Endpoint":""} Sep 11 08:46:43.208241 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:43.208202 2572 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 11 08:46:43.208388 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:43.208257 2572 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 11 08:46:43.247077 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:43.247051 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ntkhj" Sep 11 08:46:43.247207 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:43.247166 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ntkhj" podUID="e78a69db-9d98-4923-b082-ccb30b77cc9e" Sep 11 08:46:43.359007 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:43.358964 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7m87" event={"ID":"82dd9d02-85c9-415b-9db4-d4e8e189ed66","Type":"ContainerStarted","Data":"e9b10dc577515e342d032ffdf7c4940c727decf9a6a6db23d989054f190a29b7"} Sep 11 08:46:43.360755 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:43.360664 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-v5rw5" event={"ID":"510eab66-1dcf-4ffc-983d-d570fb9e9fda","Type":"ContainerStarted","Data":"1beccc9b42f2f8f2a9a7186ce04cc8759629b9deeff3f9637820f27b4fd22129"} Sep 11 08:46:43.375575 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:43.375537 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-v5rw5" podStartSLOduration=5.173136682 podStartE2EDuration="23.375523941s" totalImagesPullingTime="18.202387259s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 08:46:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 08:46:43.375370898 +0000 UTC m=+23.649946617" watchObservedRunningTime="2026-09-11 08:46:43.375523941 +0000 UTC m=+23.650099636" Sep 11 08:46:44.247463 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:44.247437 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-82xp9" Sep 11 08:46:44.247582 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:44.247559 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-82xp9" podUID="12ac768c-2c6a-43e7-9bd3-44679781ebd7" Sep 11 08:46:44.247640 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:44.247614 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v7fx2" Sep 11 08:46:44.247738 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:44.247720 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v7fx2" podUID="a370ef3a-98cb-4251-86b9-cb3155c587a5" Sep 11 08:46:44.366219 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:44.366143 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lhkkf" event={"ID":"83409be8-1bb2-43dc-af76-d3d74e569101","Type":"ContainerStarted","Data":"20bd762ab18fe76d45ba6dfcc6e3e42da910d186cdd637b622b0b5b24ec50224"} Sep 11 08:46:44.368307 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:44.368282 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7m87" event={"ID":"82dd9d02-85c9-415b-9db4-d4e8e189ed66","Type":"ContainerStarted","Data":"67e557b43bd2b0d219f2c4cf929e31e13bd0cf456815cf17009f4077cc97dd1b"} Sep 11 08:46:44.384284 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:44.384243 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7m87" podStartSLOduration=3.975935249 podStartE2EDuration="24.384208214s" totalImagesPullingTime="20.408272965s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 08:46:20 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 08:46:44.383737947 +0000 UTC m=+24.658313643" watchObservedRunningTime="2026-09-11 08:46:44.384208214 +0000 UTC m=+24.658783940" Sep 11 08:46:45.246976 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:45.246939 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ntkhj" Sep 11 08:46:45.247158 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:45.247082 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ntkhj" podUID="e78a69db-9d98-4923-b082-ccb30b77cc9e" Sep 11 08:46:46.251307 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:46.251270 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-82xp9" Sep 11 08:46:46.251770 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:46.251388 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-82xp9" podUID="12ac768c-2c6a-43e7-9bd3-44679781ebd7" Sep 11 08:46:46.251770 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:46.251739 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v7fx2" Sep 11 08:46:46.251889 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:46.251822 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v7fx2" podUID="a370ef3a-98cb-4251-86b9-cb3155c587a5" Sep 11 08:46:46.960847 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:46.960645 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-p5sg8" Sep 11 08:46:46.961449 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:46.961424 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-p5sg8" Sep 11 08:46:47.247038 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:47.246966 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ntkhj" Sep 11 08:46:47.247200 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:47.247094 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ntkhj" podUID="e78a69db-9d98-4923-b082-ccb30b77cc9e" Sep 11 08:46:47.375652 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:47.375617 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-p5sg8" Sep 11 08:46:47.376345 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:47.376326 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-p5sg8" Sep 11 08:46:48.247066 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:48.247038 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v7fx2" Sep 11 08:46:48.247218 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:48.247135 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v7fx2" podUID="a370ef3a-98cb-4251-86b9-cb3155c587a5" Sep 11 08:46:48.247218 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:48.247195 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-82xp9" Sep 11 08:46:48.247369 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:48.247324 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-82xp9" podUID="12ac768c-2c6a-43e7-9bd3-44679781ebd7" Sep 11 08:46:48.379401 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:48.379358 2572 generic.go:356] "Generic (PLEG): container finished" podID="0c0c64bd-73f4-4cb4-b660-c7232ff8e573" containerID="a88096b66ec22a3a536846b4a101b94e24d906f5f5b1290e9bd7fc2646a534cb" exitCode=0 Sep 11 08:46:48.380276 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:48.379436 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k599z" event={"ID":"0c0c64bd-73f4-4cb4-b660-c7232ff8e573","Type":"ContainerDied","Data":"a88096b66ec22a3a536846b4a101b94e24d906f5f5b1290e9bd7fc2646a534cb"} Sep 11 08:46:48.383495 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:48.383441 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lhkkf" event={"ID":"83409be8-1bb2-43dc-af76-d3d74e569101","Type":"ContainerStarted","Data":"c7afe256f3cc6b22426ffd9f3a12bba685784634e3341543a2c0393f1fe2aea1"} Sep 11 08:46:48.383622 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:48.383604 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lhkkf" Sep 11 08:46:49.246730 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:49.246559 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ntkhj" Sep 11 08:46:49.246904 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:49.246815 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ntkhj" podUID="e78a69db-9d98-4923-b082-ccb30b77cc9e" Sep 11 08:46:49.386115 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:49.386090 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lhkkf" Sep 11 08:46:49.386440 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:49.386180 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lhkkf" Sep 11 08:46:49.403691 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:49.403667 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lhkkf" Sep 11 08:46:49.404380 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:49.404357 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lhkkf" Sep 11 08:46:49.446849 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:49.446759 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-lhkkf" podStartSLOduration=11.183261104 podStartE2EDuration="29.446739991s" totalImagesPullingTime="18.263478887s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 08:46:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 08:46:48.427420027 +0000 UTC m=+28.701995724" watchObservedRunningTime="2026-09-11 08:46:49.446739991 +0000 UTC m=+29.721315688" Sep 11 08:46:49.528824 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:49.528798 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-82xp9"] Sep 11 08:46:49.528932 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:49.528910 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-82xp9" Sep 11 08:46:49.529004 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:49.528986 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-82xp9" podUID="12ac768c-2c6a-43e7-9bd3-44679781ebd7" Sep 11 08:46:49.546266 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:49.546218 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-v7fx2"] Sep 11 08:46:49.546365 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:49.546333 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v7fx2" Sep 11 08:46:49.546421 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:49.546405 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v7fx2" podUID="a370ef3a-98cb-4251-86b9-cb3155c587a5" Sep 11 08:46:49.549484 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:49.549464 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-ntkhj"] Sep 11 08:46:49.549571 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:49.549533 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ntkhj" Sep 11 08:46:49.549613 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:49.549599 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ntkhj" podUID="e78a69db-9d98-4923-b082-ccb30b77cc9e" Sep 11 08:46:50.389348 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:50.389320 2572 generic.go:356] "Generic (PLEG): container finished" podID="0c0c64bd-73f4-4cb4-b660-c7232ff8e573" containerID="018df64607d5103df71305c689b32bae8bc9627b0d7489a00f187e08e0c54a58" exitCode=0 Sep 11 08:46:50.389635 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:50.389402 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k599z" event={"ID":"0c0c64bd-73f4-4cb4-b660-c7232ff8e573","Type":"ContainerDied","Data":"018df64607d5103df71305c689b32bae8bc9627b0d7489a00f187e08e0c54a58"} Sep 11 08:46:51.246993 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:51.246967 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v7fx2" Sep 11 08:46:51.247132 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:51.247057 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-82xp9" Sep 11 08:46:51.247132 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:51.247061 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v7fx2" podUID="a370ef3a-98cb-4251-86b9-cb3155c587a5" Sep 11 08:46:51.247132 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:51.247086 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ntkhj" Sep 11 08:46:51.247320 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:51.247161 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-82xp9" podUID="12ac768c-2c6a-43e7-9bd3-44679781ebd7" Sep 11 08:46:51.247320 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:51.247248 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ntkhj" podUID="e78a69db-9d98-4923-b082-ccb30b77cc9e" Sep 11 08:46:52.395900 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:52.395724 2572 generic.go:356] "Generic (PLEG): container finished" podID="0c0c64bd-73f4-4cb4-b660-c7232ff8e573" containerID="185cd9f512f231b28793be69c5920762ef3b9f7f8d6eff85700e8d3b0c6b06a9" exitCode=0 Sep 11 08:46:52.396258 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:52.395806 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k599z" event={"ID":"0c0c64bd-73f4-4cb4-b660-c7232ff8e573","Type":"ContainerDied","Data":"185cd9f512f231b28793be69c5920762ef3b9f7f8d6eff85700e8d3b0c6b06a9"} Sep 11 08:46:53.247082 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:53.247052 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v7fx2" Sep 11 08:46:53.247261 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:53.247139 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-82xp9" Sep 11 08:46:53.247261 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:53.247149 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ntkhj" Sep 11 08:46:53.247261 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:53.247146 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v7fx2" podUID="a370ef3a-98cb-4251-86b9-cb3155c587a5" Sep 11 08:46:53.247261 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:53.247214 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-82xp9" podUID="12ac768c-2c6a-43e7-9bd3-44679781ebd7" Sep 11 08:46:53.247414 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:53.247332 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ntkhj" podUID="e78a69db-9d98-4923-b082-ccb30b77cc9e" Sep 11 08:46:53.894648 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:53.894616 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e78a69db-9d98-4923-b082-ccb30b77cc9e-metrics-certs\") pod \"network-metrics-daemon-ntkhj\" (UID: \"e78a69db-9d98-4923-b082-ccb30b77cc9e\") " pod="openshift-multus/network-metrics-daemon-ntkhj" Sep 11 08:46:53.895040 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:53.894759 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 08:46:53.895040 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:53.894822 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e78a69db-9d98-4923-b082-ccb30b77cc9e-metrics-certs podName:e78a69db-9d98-4923-b082-ccb30b77cc9e nodeName:}" failed. No retries permitted until 2026-09-11 08:47:25.894806071 +0000 UTC m=+66.169381764 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e78a69db-9d98-4923-b082-ccb30b77cc9e-metrics-certs") pod "network-metrics-daemon-ntkhj" (UID: "e78a69db-9d98-4923-b082-ccb30b77cc9e") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 08:46:53.995404 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:53.995381 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwc96\" (UniqueName: \"kubernetes.io/projected/12ac768c-2c6a-43e7-9bd3-44679781ebd7-kube-api-access-zwc96\") pod \"network-check-target-82xp9\" (UID: \"12ac768c-2c6a-43e7-9bd3-44679781ebd7\") " pod="openshift-network-diagnostics/network-check-target-82xp9" Sep 11 08:46:53.995501 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:53.995425 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a370ef3a-98cb-4251-86b9-cb3155c587a5-original-pull-secret\") pod \"global-pull-secret-syncer-v7fx2\" (UID: \"a370ef3a-98cb-4251-86b9-cb3155c587a5\") " pod="kube-system/global-pull-secret-syncer-v7fx2" Sep 11 08:46:53.995541 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:53.995518 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 08:46:53.995578 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:53.995537 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 08:46:53.995578 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:53.995556 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 08:46:53.995578 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:53.995566 2572 projected.go:196] Error preparing data for projected volume kube-api-access-zwc96 for pod openshift-network-diagnostics/network-check-target-82xp9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 08:46:53.995676 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:53.995567 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a370ef3a-98cb-4251-86b9-cb3155c587a5-original-pull-secret podName:a370ef3a-98cb-4251-86b9-cb3155c587a5 nodeName:}" failed. No retries permitted until 2026-09-11 08:47:25.995554809 +0000 UTC m=+66.270130495 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a370ef3a-98cb-4251-86b9-cb3155c587a5-original-pull-secret") pod "global-pull-secret-syncer-v7fx2" (UID: "a370ef3a-98cb-4251-86b9-cb3155c587a5") : object "kube-system"/"original-pull-secret" not registered Sep 11 08:46:53.995676 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:53.995612 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/12ac768c-2c6a-43e7-9bd3-44679781ebd7-kube-api-access-zwc96 podName:12ac768c-2c6a-43e7-9bd3-44679781ebd7 nodeName:}" failed. No retries permitted until 2026-09-11 08:47:25.995601306 +0000 UTC m=+66.270176980 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-zwc96" (UniqueName: "kubernetes.io/projected/12ac768c-2c6a-43e7-9bd3-44679781ebd7-kube-api-access-zwc96") pod "network-check-target-82xp9" (UID: "12ac768c-2c6a-43e7-9bd3-44679781ebd7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 08:46:54.021038 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:54.019835 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-166.ec2.internal" event="NodeReady" Sep 11 08:46:54.021038 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:54.019972 2572 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 11 08:46:54.057584 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:54.057559 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6fdf7ff479-xlfkd"] Sep 11 08:46:54.069479 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:54.069460 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6fdf7ff479-xlfkd" Sep 11 08:46:54.071828 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:54.071800 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-l57fw\"" Sep 11 08:46:54.071933 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:54.071875 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 11 08:46:54.071933 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:54.071899 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 11 08:46:54.072078 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:54.072061 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 11 08:46:54.072294 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:54.072276 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-d67t9"] Sep 11 08:46:54.079821 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:54.079804 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 11 08:46:54.081374 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:54.081357 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-2tlm8"] Sep 11 08:46:54.081497 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:54.081472 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-d67t9" Sep 11 08:46:54.084004 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:54.083985 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 11 08:46:54.084256 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:54.084217 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 11 08:46:54.084341 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:54.084264 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-7fqkq\"" Sep 11 08:46:54.091904 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:54.091883 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6fdf7ff479-xlfkd"] Sep 11 08:46:54.092012 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:54.091919 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-d67t9"] Sep 11 08:46:54.092012 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:54.091946 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-2tlm8"] Sep 11 08:46:54.092012 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:54.091989 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-2tlm8" Sep 11 08:46:54.093995 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:54.093974 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 11 08:46:54.094128 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:54.093998 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 11 08:46:54.094128 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:54.093980 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 11 08:46:54.094128 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:54.093977 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-q7bhf\"" Sep 11 08:46:54.197406 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:54.197379 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/64b9c16d-d0c5-423f-98e3-67d3fcff6128-registry-certificates\") pod \"image-registry-6fdf7ff479-xlfkd\" (UID: \"64b9c16d-d0c5-423f-98e3-67d3fcff6128\") " pod="openshift-image-registry/image-registry-6fdf7ff479-xlfkd" Sep 11 08:46:54.197614 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:54.197413 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8ea2d42c-f484-483d-8714-603b8f05cc54-config-volume\") pod \"dns-default-d67t9\" (UID: \"8ea2d42c-f484-483d-8714-603b8f05cc54\") " pod="openshift-dns/dns-default-d67t9" Sep 11 08:46:54.197614 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:54.197430 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ea2d42c-f484-483d-8714-603b8f05cc54-tmp-dir\") pod \"dns-default-d67t9\" (UID: \"8ea2d42c-f484-483d-8714-603b8f05cc54\") " pod="openshift-dns/dns-default-d67t9" Sep 11 08:46:54.197614 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:54.197453 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsj8n\" (UniqueName: \"kubernetes.io/projected/a9917159-f37b-4842-ae12-10c24123dff1-kube-api-access-nsj8n\") pod \"ingress-canary-2tlm8\" (UID: \"a9917159-f37b-4842-ae12-10c24123dff1\") " pod="openshift-ingress-canary/ingress-canary-2tlm8" Sep 11 08:46:54.197614 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:54.197523 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/64b9c16d-d0c5-423f-98e3-67d3fcff6128-registry-tls\") pod \"image-registry-6fdf7ff479-xlfkd\" (UID: \"64b9c16d-d0c5-423f-98e3-67d3fcff6128\") " pod="openshift-image-registry/image-registry-6fdf7ff479-xlfkd" Sep 11 08:46:54.197614 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:54.197553 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/64b9c16d-d0c5-423f-98e3-67d3fcff6128-ca-trust-extracted\") pod \"image-registry-6fdf7ff479-xlfkd\" (UID: \"64b9c16d-d0c5-423f-98e3-67d3fcff6128\") " pod="openshift-image-registry/image-registry-6fdf7ff479-xlfkd" Sep 11 08:46:54.197614 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:54.197575 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a9917159-f37b-4842-ae12-10c24123dff1-cert\") pod \"ingress-canary-2tlm8\" (UID: \"a9917159-f37b-4842-ae12-10c24123dff1\") " pod="openshift-ingress-canary/ingress-canary-2tlm8" Sep 11 08:46:54.197614 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:54.197597 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/64b9c16d-d0c5-423f-98e3-67d3fcff6128-image-registry-private-configuration\") pod \"image-registry-6fdf7ff479-xlfkd\" (UID: \"64b9c16d-d0c5-423f-98e3-67d3fcff6128\") " pod="openshift-image-registry/image-registry-6fdf7ff479-xlfkd" Sep 11 08:46:54.197903 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:54.197622 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8w4l9\" (UniqueName: \"kubernetes.io/projected/64b9c16d-d0c5-423f-98e3-67d3fcff6128-kube-api-access-8w4l9\") pod \"image-registry-6fdf7ff479-xlfkd\" (UID: \"64b9c16d-d0c5-423f-98e3-67d3fcff6128\") " pod="openshift-image-registry/image-registry-6fdf7ff479-xlfkd" Sep 11 08:46:54.197903 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:54.197663 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mz4qk\" (UniqueName: \"kubernetes.io/projected/8ea2d42c-f484-483d-8714-603b8f05cc54-kube-api-access-mz4qk\") pod \"dns-default-d67t9\" (UID: \"8ea2d42c-f484-483d-8714-603b8f05cc54\") " pod="openshift-dns/dns-default-d67t9" Sep 11 08:46:54.197903 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:54.197685 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/64b9c16d-d0c5-423f-98e3-67d3fcff6128-bound-sa-token\") pod \"image-registry-6fdf7ff479-xlfkd\" (UID: \"64b9c16d-d0c5-423f-98e3-67d3fcff6128\") " pod="openshift-image-registry/image-registry-6fdf7ff479-xlfkd" Sep 11 08:46:54.197903 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:54.197776 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8ea2d42c-f484-483d-8714-603b8f05cc54-metrics-tls\") pod \"dns-default-d67t9\" (UID: \"8ea2d42c-f484-483d-8714-603b8f05cc54\") " pod="openshift-dns/dns-default-d67t9" Sep 11 08:46:54.197903 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:54.197805 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/64b9c16d-d0c5-423f-98e3-67d3fcff6128-installation-pull-secrets\") pod \"image-registry-6fdf7ff479-xlfkd\" (UID: \"64b9c16d-d0c5-423f-98e3-67d3fcff6128\") " pod="openshift-image-registry/image-registry-6fdf7ff479-xlfkd" Sep 11 08:46:54.197903 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:54.197824 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/64b9c16d-d0c5-423f-98e3-67d3fcff6128-trusted-ca\") pod \"image-registry-6fdf7ff479-xlfkd\" (UID: \"64b9c16d-d0c5-423f-98e3-67d3fcff6128\") " pod="openshift-image-registry/image-registry-6fdf7ff479-xlfkd" Sep 11 08:46:54.235274 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:54.235213 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-p5sg8_2c0c8738-0dd8-441e-8c6e-3ff6009337ca/konnectivity-agent/0.log" limit=1024 Sep 11 08:46:54.298918 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:54.298879 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mz4qk\" (UniqueName: \"kubernetes.io/projected/8ea2d42c-f484-483d-8714-603b8f05cc54-kube-api-access-mz4qk\") pod \"dns-default-d67t9\" (UID: \"8ea2d42c-f484-483d-8714-603b8f05cc54\") " pod="openshift-dns/dns-default-d67t9" Sep 11 08:46:54.299072 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:54.298932 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/64b9c16d-d0c5-423f-98e3-67d3fcff6128-bound-sa-token\") pod \"image-registry-6fdf7ff479-xlfkd\" (UID: \"64b9c16d-d0c5-423f-98e3-67d3fcff6128\") " pod="openshift-image-registry/image-registry-6fdf7ff479-xlfkd" Sep 11 08:46:54.299072 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:54.298992 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8ea2d42c-f484-483d-8714-603b8f05cc54-metrics-tls\") pod \"dns-default-d67t9\" (UID: \"8ea2d42c-f484-483d-8714-603b8f05cc54\") " pod="openshift-dns/dns-default-d67t9" Sep 11 08:46:54.299072 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:54.299020 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/64b9c16d-d0c5-423f-98e3-67d3fcff6128-installation-pull-secrets\") pod \"image-registry-6fdf7ff479-xlfkd\" (UID: \"64b9c16d-d0c5-423f-98e3-67d3fcff6128\") " pod="openshift-image-registry/image-registry-6fdf7ff479-xlfkd" Sep 11 08:46:54.299268 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:54.299129 2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 08:46:54.299268 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:54.299199 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ea2d42c-f484-483d-8714-603b8f05cc54-metrics-tls podName:8ea2d42c-f484-483d-8714-603b8f05cc54 nodeName:}" failed. No retries permitted until 2026-09-11 08:46:54.799176608 +0000 UTC m=+35.073752296 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8ea2d42c-f484-483d-8714-603b8f05cc54-metrics-tls") pod "dns-default-d67t9" (UID: "8ea2d42c-f484-483d-8714-603b8f05cc54") : secret "dns-default-metrics-tls" not found Sep 11 08:46:54.299268 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:54.299132 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/64b9c16d-d0c5-423f-98e3-67d3fcff6128-trusted-ca\") pod \"image-registry-6fdf7ff479-xlfkd\" (UID: \"64b9c16d-d0c5-423f-98e3-67d3fcff6128\") " pod="openshift-image-registry/image-registry-6fdf7ff479-xlfkd" Sep 11 08:46:54.299457 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:54.299327 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/64b9c16d-d0c5-423f-98e3-67d3fcff6128-registry-certificates\") pod \"image-registry-6fdf7ff479-xlfkd\" (UID: \"64b9c16d-d0c5-423f-98e3-67d3fcff6128\") " pod="openshift-image-registry/image-registry-6fdf7ff479-xlfkd" Sep 11 08:46:54.299457 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:54.299360 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8ea2d42c-f484-483d-8714-603b8f05cc54-config-volume\") pod \"dns-default-d67t9\" (UID: \"8ea2d42c-f484-483d-8714-603b8f05cc54\") " pod="openshift-dns/dns-default-d67t9" Sep 11 08:46:54.299457 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:54.299381 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ea2d42c-f484-483d-8714-603b8f05cc54-tmp-dir\") pod \"dns-default-d67t9\" (UID: \"8ea2d42c-f484-483d-8714-603b8f05cc54\") " pod="openshift-dns/dns-default-d67t9" Sep 11 08:46:54.299457 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:54.299404 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsj8n\" (UniqueName: \"kubernetes.io/projected/a9917159-f37b-4842-ae12-10c24123dff1-kube-api-access-nsj8n\") pod \"ingress-canary-2tlm8\" (UID: \"a9917159-f37b-4842-ae12-10c24123dff1\") " pod="openshift-ingress-canary/ingress-canary-2tlm8" Sep 11 08:46:54.299654 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:54.299631 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/64b9c16d-d0c5-423f-98e3-67d3fcff6128-registry-tls\") pod \"image-registry-6fdf7ff479-xlfkd\" (UID: \"64b9c16d-d0c5-423f-98e3-67d3fcff6128\") " pod="openshift-image-registry/image-registry-6fdf7ff479-xlfkd" Sep 11 08:46:54.299706 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:54.299667 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/64b9c16d-d0c5-423f-98e3-67d3fcff6128-ca-trust-extracted\") pod \"image-registry-6fdf7ff479-xlfkd\" (UID: \"64b9c16d-d0c5-423f-98e3-67d3fcff6128\") " pod="openshift-image-registry/image-registry-6fdf7ff479-xlfkd" Sep 11 08:46:54.299757 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:54.299704 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a9917159-f37b-4842-ae12-10c24123dff1-cert\") pod \"ingress-canary-2tlm8\" (UID: \"a9917159-f37b-4842-ae12-10c24123dff1\") " pod="openshift-ingress-canary/ingress-canary-2tlm8" Sep 11 08:46:54.299757 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:54.299730 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/64b9c16d-d0c5-423f-98e3-67d3fcff6128-image-registry-private-configuration\") pod \"image-registry-6fdf7ff479-xlfkd\" (UID: \"64b9c16d-d0c5-423f-98e3-67d3fcff6128\") " pod="openshift-image-registry/image-registry-6fdf7ff479-xlfkd" Sep 11 08:46:54.299886 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:54.299765 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8w4l9\" (UniqueName: \"kubernetes.io/projected/64b9c16d-d0c5-423f-98e3-67d3fcff6128-kube-api-access-8w4l9\") pod \"image-registry-6fdf7ff479-xlfkd\" (UID: \"64b9c16d-d0c5-423f-98e3-67d3fcff6128\") " pod="openshift-image-registry/image-registry-6fdf7ff479-xlfkd" Sep 11 08:46:54.299886 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:54.299792 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ea2d42c-f484-483d-8714-603b8f05cc54-tmp-dir\") pod \"dns-default-d67t9\" (UID: \"8ea2d42c-f484-483d-8714-603b8f05cc54\") " pod="openshift-dns/dns-default-d67t9" Sep 11 08:46:54.299886 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:54.299869 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/64b9c16d-d0c5-423f-98e3-67d3fcff6128-registry-certificates\") pod \"image-registry-6fdf7ff479-xlfkd\" (UID: \"64b9c16d-d0c5-423f-98e3-67d3fcff6128\") " pod="openshift-image-registry/image-registry-6fdf7ff479-xlfkd" Sep 11 08:46:54.300011 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:54.299978 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8ea2d42c-f484-483d-8714-603b8f05cc54-config-volume\") pod \"dns-default-d67t9\" (UID: \"8ea2d42c-f484-483d-8714-603b8f05cc54\") " pod="openshift-dns/dns-default-d67t9" Sep 11 08:46:54.300106 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:54.300087 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/64b9c16d-d0c5-423f-98e3-67d3fcff6128-ca-trust-extracted\") pod \"image-registry-6fdf7ff479-xlfkd\" (UID: \"64b9c16d-d0c5-423f-98e3-67d3fcff6128\") " pod="openshift-image-registry/image-registry-6fdf7ff479-xlfkd" Sep 11 08:46:54.300165 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:54.300100 2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 11 08:46:54.300165 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:54.300106 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/64b9c16d-d0c5-423f-98e3-67d3fcff6128-trusted-ca\") pod \"image-registry-6fdf7ff479-xlfkd\" (UID: \"64b9c16d-d0c5-423f-98e3-67d3fcff6128\") " pod="openshift-image-registry/image-registry-6fdf7ff479-xlfkd" Sep 11 08:46:54.300165 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:54.300119 2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6fdf7ff479-xlfkd: secret "image-registry-tls" not found Sep 11 08:46:54.300322 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:54.300191 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/64b9c16d-d0c5-423f-98e3-67d3fcff6128-registry-tls podName:64b9c16d-d0c5-423f-98e3-67d3fcff6128 nodeName:}" failed. No retries permitted until 2026-09-11 08:46:54.80017111 +0000 UTC m=+35.074746807 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/64b9c16d-d0c5-423f-98e3-67d3fcff6128-registry-tls") pod "image-registry-6fdf7ff479-xlfkd" (UID: "64b9c16d-d0c5-423f-98e3-67d3fcff6128") : secret "image-registry-tls" not found Sep 11 08:46:54.303841 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:54.303809 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/64b9c16d-d0c5-423f-98e3-67d3fcff6128-image-registry-private-configuration\") pod \"image-registry-6fdf7ff479-xlfkd\" (UID: \"64b9c16d-d0c5-423f-98e3-67d3fcff6128\") " pod="openshift-image-registry/image-registry-6fdf7ff479-xlfkd" Sep 11 08:46:54.303935 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:54.303845 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a9917159-f37b-4842-ae12-10c24123dff1-cert\") pod \"ingress-canary-2tlm8\" (UID: \"a9917159-f37b-4842-ae12-10c24123dff1\") " pod="openshift-ingress-canary/ingress-canary-2tlm8" Sep 11 08:46:54.303935 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:54.303820 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/64b9c16d-d0c5-423f-98e3-67d3fcff6128-installation-pull-secrets\") pod \"image-registry-6fdf7ff479-xlfkd\" (UID: \"64b9c16d-d0c5-423f-98e3-67d3fcff6128\") " pod="openshift-image-registry/image-registry-6fdf7ff479-xlfkd" Sep 11 08:46:54.307638 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:54.307617 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/64b9c16d-d0c5-423f-98e3-67d3fcff6128-bound-sa-token\") pod \"image-registry-6fdf7ff479-xlfkd\" (UID: \"64b9c16d-d0c5-423f-98e3-67d3fcff6128\") " pod="openshift-image-registry/image-registry-6fdf7ff479-xlfkd" Sep 11 08:46:54.311412 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:54.311389 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsj8n\" (UniqueName: \"kubernetes.io/projected/a9917159-f37b-4842-ae12-10c24123dff1-kube-api-access-nsj8n\") pod \"ingress-canary-2tlm8\" (UID: \"a9917159-f37b-4842-ae12-10c24123dff1\") " pod="openshift-ingress-canary/ingress-canary-2tlm8" Sep 11 08:46:54.311667 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:54.311649 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mz4qk\" (UniqueName: \"kubernetes.io/projected/8ea2d42c-f484-483d-8714-603b8f05cc54-kube-api-access-mz4qk\") pod \"dns-default-d67t9\" (UID: \"8ea2d42c-f484-483d-8714-603b8f05cc54\") " pod="openshift-dns/dns-default-d67t9" Sep 11 08:46:54.312133 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:54.312108 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8w4l9\" (UniqueName: \"kubernetes.io/projected/64b9c16d-d0c5-423f-98e3-67d3fcff6128-kube-api-access-8w4l9\") pod \"image-registry-6fdf7ff479-xlfkd\" (UID: \"64b9c16d-d0c5-423f-98e3-67d3fcff6128\") " pod="openshift-image-registry/image-registry-6fdf7ff479-xlfkd" Sep 11 08:46:54.358240 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:54.358204 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-p5sg8_2c0c8738-0dd8-441e-8c6e-3ff6009337ca/konnectivity-agent/0.log" limit=1024 Sep 11 08:46:54.401739 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:54.401718 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-2tlm8" Sep 11 08:46:54.720354 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:54.720186 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-2tlm8"] Sep 11 08:46:54.723174 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:46:54.723151 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9917159_f37b_4842_ae12_10c24123dff1.slice/crio-e90d6223a9ac8ef6d49071c4591a847ff21d009cc827b8c60022e1d69bd74610 WatchSource:0}: Error finding container e90d6223a9ac8ef6d49071c4591a847ff21d009cc827b8c60022e1d69bd74610: Status 404 returned error can't find the container with id e90d6223a9ac8ef6d49071c4591a847ff21d009cc827b8c60022e1d69bd74610 Sep 11 08:46:54.802368 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:54.802290 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8ea2d42c-f484-483d-8714-603b8f05cc54-metrics-tls\") pod \"dns-default-d67t9\" (UID: \"8ea2d42c-f484-483d-8714-603b8f05cc54\") " pod="openshift-dns/dns-default-d67t9" Sep 11 08:46:54.802368 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:54.802341 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/64b9c16d-d0c5-423f-98e3-67d3fcff6128-registry-tls\") pod \"image-registry-6fdf7ff479-xlfkd\" (UID: \"64b9c16d-d0c5-423f-98e3-67d3fcff6128\") " pod="openshift-image-registry/image-registry-6fdf7ff479-xlfkd" Sep 11 08:46:54.802573 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:54.802459 2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 11 08:46:54.802573 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:54.802469 2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6fdf7ff479-xlfkd: secret "image-registry-tls" not found Sep 11 08:46:54.802573 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:54.802532 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/64b9c16d-d0c5-423f-98e3-67d3fcff6128-registry-tls podName:64b9c16d-d0c5-423f-98e3-67d3fcff6128 nodeName:}" failed. No retries permitted until 2026-09-11 08:46:55.802512268 +0000 UTC m=+36.077087946 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/64b9c16d-d0c5-423f-98e3-67d3fcff6128-registry-tls") pod "image-registry-6fdf7ff479-xlfkd" (UID: "64b9c16d-d0c5-423f-98e3-67d3fcff6128") : secret "image-registry-tls" not found Sep 11 08:46:54.802855 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:54.802827 2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 08:46:54.802965 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:54.802869 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ea2d42c-f484-483d-8714-603b8f05cc54-metrics-tls podName:8ea2d42c-f484-483d-8714-603b8f05cc54 nodeName:}" failed. No retries permitted until 2026-09-11 08:46:55.802855716 +0000 UTC m=+36.077431403 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8ea2d42c-f484-483d-8714-603b8f05cc54-metrics-tls") pod "dns-default-d67t9" (UID: "8ea2d42c-f484-483d-8714-603b8f05cc54") : secret "dns-default-metrics-tls" not found Sep 11 08:46:55.247122 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:55.247091 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ntkhj" Sep 11 08:46:55.247122 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:55.247126 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-82xp9" Sep 11 08:46:55.248215 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:55.247140 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v7fx2" Sep 11 08:46:55.249318 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:55.249296 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 11 08:46:55.249438 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:55.249378 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 11 08:46:55.249839 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:55.249810 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 11 08:46:55.249839 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:55.249811 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 11 08:46:55.250006 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:55.249855 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-wbz7z\"" Sep 11 08:46:55.250172 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:55.250131 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xm9k4\"" Sep 11 08:46:55.404067 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:55.404029 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-2tlm8" event={"ID":"a9917159-f37b-4842-ae12-10c24123dff1","Type":"ContainerStarted","Data":"e90d6223a9ac8ef6d49071c4591a847ff21d009cc827b8c60022e1d69bd74610"} Sep 11 08:46:55.810063 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:55.810026 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8ea2d42c-f484-483d-8714-603b8f05cc54-metrics-tls\") pod \"dns-default-d67t9\" (UID: \"8ea2d42c-f484-483d-8714-603b8f05cc54\") " pod="openshift-dns/dns-default-d67t9" Sep 11 08:46:55.810268 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:55.810090 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/64b9c16d-d0c5-423f-98e3-67d3fcff6128-registry-tls\") pod \"image-registry-6fdf7ff479-xlfkd\" (UID: \"64b9c16d-d0c5-423f-98e3-67d3fcff6128\") " pod="openshift-image-registry/image-registry-6fdf7ff479-xlfkd" Sep 11 08:46:55.810268 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:55.810194 2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 11 08:46:55.810268 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:55.810193 2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 08:46:55.810268 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:55.810207 2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6fdf7ff479-xlfkd: secret "image-registry-tls" not found Sep 11 08:46:55.810474 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:55.810293 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/64b9c16d-d0c5-423f-98e3-67d3fcff6128-registry-tls podName:64b9c16d-d0c5-423f-98e3-67d3fcff6128 nodeName:}" failed. No retries permitted until 2026-09-11 08:46:57.810273576 +0000 UTC m=+38.084849255 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/64b9c16d-d0c5-423f-98e3-67d3fcff6128-registry-tls") pod "image-registry-6fdf7ff479-xlfkd" (UID: "64b9c16d-d0c5-423f-98e3-67d3fcff6128") : secret "image-registry-tls" not found Sep 11 08:46:55.810474 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:55.810310 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ea2d42c-f484-483d-8714-603b8f05cc54-metrics-tls podName:8ea2d42c-f484-483d-8714-603b8f05cc54 nodeName:}" failed. No retries permitted until 2026-09-11 08:46:57.810301456 +0000 UTC m=+38.084877144 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8ea2d42c-f484-483d-8714-603b8f05cc54-metrics-tls") pod "dns-default-d67t9" (UID: "8ea2d42c-f484-483d-8714-603b8f05cc54") : secret "dns-default-metrics-tls" not found Sep 11 08:46:57.825301 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:57.825071 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8ea2d42c-f484-483d-8714-603b8f05cc54-metrics-tls\") pod \"dns-default-d67t9\" (UID: \"8ea2d42c-f484-483d-8714-603b8f05cc54\") " pod="openshift-dns/dns-default-d67t9" Sep 11 08:46:57.825766 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:57.825355 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/64b9c16d-d0c5-423f-98e3-67d3fcff6128-registry-tls\") pod \"image-registry-6fdf7ff479-xlfkd\" (UID: \"64b9c16d-d0c5-423f-98e3-67d3fcff6128\") " pod="openshift-image-registry/image-registry-6fdf7ff479-xlfkd" Sep 11 08:46:57.825766 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:57.825276 2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 08:46:57.825766 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:57.825452 2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 11 08:46:57.825766 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:57.825463 2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6fdf7ff479-xlfkd: secret "image-registry-tls" not found Sep 11 08:46:57.825766 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:57.825464 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ea2d42c-f484-483d-8714-603b8f05cc54-metrics-tls podName:8ea2d42c-f484-483d-8714-603b8f05cc54 nodeName:}" failed. No retries permitted until 2026-09-11 08:47:01.825439798 +0000 UTC m=+42.100015483 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8ea2d42c-f484-483d-8714-603b8f05cc54-metrics-tls") pod "dns-default-d67t9" (UID: "8ea2d42c-f484-483d-8714-603b8f05cc54") : secret "dns-default-metrics-tls" not found Sep 11 08:46:57.825766 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:46:57.825525 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/64b9c16d-d0c5-423f-98e3-67d3fcff6128-registry-tls podName:64b9c16d-d0c5-423f-98e3-67d3fcff6128 nodeName:}" failed. No retries permitted until 2026-09-11 08:47:01.825509619 +0000 UTC m=+42.100085294 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/64b9c16d-d0c5-423f-98e3-67d3fcff6128-registry-tls") pod "image-registry-6fdf7ff479-xlfkd" (UID: "64b9c16d-d0c5-423f-98e3-67d3fcff6128") : secret "image-registry-tls" not found Sep 11 08:46:59.421217 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:59.421185 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k599z" event={"ID":"0c0c64bd-73f4-4cb4-b660-c7232ff8e573","Type":"ContainerStarted","Data":"0a91dd03cf28ed891194e533b44a4eec543f32ae9c3052549f43489f4e75eebe"} Sep 11 08:46:59.423269 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:59.423200 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-2tlm8" event={"ID":"a9917159-f37b-4842-ae12-10c24123dff1","Type":"ContainerStarted","Data":"554b55df88264906edf3c3c493fab83906e220c6727beb04b10a4e213c7b0df1"} Sep 11 08:46:59.438977 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:46:59.438909 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-2tlm8" podStartSLOduration=0.900440015 podStartE2EDuration="5.438889927s" totalImagesPullingTime="4.538449912s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 08:46:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 08:46:59.438005105 +0000 UTC m=+39.712580800" watchObservedRunningTime="2026-09-11 08:46:59.438889927 +0000 UTC m=+39.713465624" Sep 11 08:47:00.431210 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:00.431168 2572 generic.go:356] "Generic (PLEG): container finished" podID="0c0c64bd-73f4-4cb4-b660-c7232ff8e573" containerID="0a91dd03cf28ed891194e533b44a4eec543f32ae9c3052549f43489f4e75eebe" exitCode=0 Sep 11 08:47:00.431616 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:00.431259 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k599z" event={"ID":"0c0c64bd-73f4-4cb4-b660-c7232ff8e573","Type":"ContainerDied","Data":"0a91dd03cf28ed891194e533b44a4eec543f32ae9c3052549f43489f4e75eebe"} Sep 11 08:47:00.798767 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:00.798694 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-rsjdv"] Sep 11 08:47:00.816434 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:00.816403 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-rsjdv"] Sep 11 08:47:00.816574 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:00.816536 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-rsjdv" Sep 11 08:47:00.818745 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:00.818724 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Sep 11 08:47:00.819030 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:00.819013 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Sep 11 08:47:00.819094 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:00.819048 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-hkctm\"" Sep 11 08:47:00.819327 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:00.819315 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Sep 11 08:47:00.900303 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:00.900275 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-wdwcs"] Sep 11 08:47:00.922271 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:00.922251 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-z8sdf"] Sep 11 08:47:00.922385 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:00.922318 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-wdwcs" Sep 11 08:47:00.924496 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:00.924479 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Sep 11 08:47:00.924496 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:00.924490 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Sep 11 08:47:00.924819 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:00.924803 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Sep 11 08:47:00.924915 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:00.924802 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Sep 11 08:47:00.925239 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:00.925208 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-6tqcv\"" Sep 11 08:47:00.943238 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:00.943206 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-blrfj"] Sep 11 08:47:00.943335 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:00.943319 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-z8sdf" Sep 11 08:47:00.947307 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:00.947292 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Sep 11 08:47:00.947395 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:00.947317 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Sep 11 08:47:00.947552 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:00.947533 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Sep 11 08:47:00.947642 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:00.947630 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pl9b4\" (UniqueName: \"kubernetes.io/projected/7dae9305-821b-4929-900e-2d7bf09d5654-kube-api-access-pl9b4\") pod \"cluster-samples-operator-86499c89c4-rsjdv\" (UID: \"7dae9305-821b-4929-900e-2d7bf09d5654\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-rsjdv" Sep 11 08:47:00.947774 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:00.947755 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7dae9305-821b-4929-900e-2d7bf09d5654-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-rsjdv\" (UID: \"7dae9305-821b-4929-900e-2d7bf09d5654\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-rsjdv" Sep 11 08:47:00.948061 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:00.948047 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Sep 11 08:47:00.948290 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:00.948274 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-w29t9\"" Sep 11 08:47:00.963349 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:00.963333 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-57979bfb5b-9fr79"] Sep 11 08:47:00.963453 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:00.963439 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-blrfj" Sep 11 08:47:00.969876 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:00.969854 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-xwrd9\"" Sep 11 08:47:00.969979 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:00.969890 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Sep 11 08:47:00.970127 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:00.970111 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 11 08:47:00.970634 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:00.970615 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Sep 11 08:47:00.970725 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:00.970649 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 11 08:47:00.989572 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:00.989551 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5844b856db-xjhvj"] Sep 11 08:47:00.989682 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:00.989665 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-57979bfb5b-9fr79" Sep 11 08:47:00.992652 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:00.992636 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Sep 11 08:47:00.992895 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:00.992682 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Sep 11 08:47:00.992895 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:00.992759 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Sep 11 08:47:00.993353 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:00.993339 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 11 08:47:00.993419 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:00.993374 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Sep 11 08:47:00.993591 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:00.993576 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-dwngz\"" Sep 11 08:47:00.993668 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:00.993578 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 11 08:47:01.011912 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:01.011893 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-wdwcs"] Sep 11 08:47:01.012019 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:01.011929 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-z8sdf"] Sep 11 08:47:01.012019 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:01.011949 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-blrfj"] Sep 11 08:47:01.012019 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:01.011973 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-57979bfb5b-9fr79"] Sep 11 08:47:01.012019 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:01.012019 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-xjhvj" Sep 11 08:47:01.012208 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:01.012062 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-xjhvj"] Sep 11 08:47:01.012208 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:01.012110 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-f6twl"] Sep 11 08:47:01.014870 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:01.014852 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Sep 11 08:47:01.015100 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:01.015087 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 11 08:47:01.015147 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:01.015135 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-nr4cj\"" Sep 11 08:47:01.015704 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:01.015683 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Sep 11 08:47:01.015845 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:01.015688 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 11 08:47:01.022050 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:01.022033 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Sep 11 08:47:01.033930 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:01.033911 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-xjs9g"] Sep 11 08:47:01.034057 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:01.034038 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-f6twl" Sep 11 08:47:01.036018 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:01.036000 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-xpmpj\"" Sep 11 08:47:01.048826 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:01.048777 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1d7a379e-1344-48f2-9540-7ac11ee14eae-serving-cert\") pod \"service-ca-operator-cf7c776b7-z8sdf\" (UID: \"1d7a379e-1344-48f2-9540-7ac11ee14eae\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-z8sdf" Sep 11 08:47:01.048826 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:01.048807 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1d7a379e-1344-48f2-9540-7ac11ee14eae-config\") pod \"service-ca-operator-cf7c776b7-z8sdf\" (UID: \"1d7a379e-1344-48f2-9540-7ac11ee14eae\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-z8sdf" Sep 11 08:47:01.048954 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:01.048849 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/aa036e63-1ef3-481a-9786-90b2f397ab9a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-blrfj\" (UID: \"aa036e63-1ef3-481a-9786-90b2f397ab9a\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-blrfj" Sep 11 08:47:01.048954 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:01.048902 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nc2d4\" (UniqueName: \"kubernetes.io/projected/aa036e63-1ef3-481a-9786-90b2f397ab9a-kube-api-access-nc2d4\") pod \"cluster-monitoring-operator-6cd954b897-blrfj\" (UID: \"aa036e63-1ef3-481a-9786-90b2f397ab9a\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-blrfj" Sep 11 08:47:01.048954 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:01.048950 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/aa036e63-1ef3-481a-9786-90b2f397ab9a-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-blrfj\" (UID: \"aa036e63-1ef3-481a-9786-90b2f397ab9a\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-blrfj" Sep 11 08:47:01.049067 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:01.048996 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7dae9305-821b-4929-900e-2d7bf09d5654-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-rsjdv\" (UID: \"7dae9305-821b-4929-900e-2d7bf09d5654\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-rsjdv" Sep 11 08:47:01.049067 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:01.049047 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9222c5e5-755c-460e-8568-db40848b8123-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-wdwcs\" (UID: \"9222c5e5-755c-460e-8568-db40848b8123\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-wdwcs" Sep 11 08:47:01.049067 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:01.049064 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbrcb\" (UniqueName: \"kubernetes.io/projected/9222c5e5-755c-460e-8568-db40848b8123-kube-api-access-nbrcb\") pod \"kube-storage-version-migrator-operator-6c8f84f897-wdwcs\" (UID: \"9222c5e5-755c-460e-8568-db40848b8123\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-wdwcs" Sep 11 08:47:01.049181 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:01.049100 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9222c5e5-755c-460e-8568-db40848b8123-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-wdwcs\" (UID: \"9222c5e5-755c-460e-8568-db40848b8123\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-wdwcs" Sep 11 08:47:01.049181 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:01.049164 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pl9b4\" (UniqueName: \"kubernetes.io/projected/7dae9305-821b-4929-900e-2d7bf09d5654-kube-api-access-pl9b4\") pod \"cluster-samples-operator-86499c89c4-rsjdv\" (UID: \"7dae9305-821b-4929-900e-2d7bf09d5654\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-rsjdv" Sep 11 08:47:01.049281 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:47:01.049194 2572 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 11 08:47:01.049436 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:47:01.049421 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7dae9305-821b-4929-900e-2d7bf09d5654-samples-operator-tls podName:7dae9305-821b-4929-900e-2d7bf09d5654 nodeName:}" failed. No retries permitted until 2026-09-11 08:47:01.549249561 +0000 UTC m=+41.823825235 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7dae9305-821b-4929-900e-2d7bf09d5654-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-rsjdv" (UID: "7dae9305-821b-4929-900e-2d7bf09d5654") : secret "samples-operator-tls" not found Sep 11 08:47:01.049566 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:01.049442 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkd6m\" (UniqueName: \"kubernetes.io/projected/1d7a379e-1344-48f2-9540-7ac11ee14eae-kube-api-access-gkd6m\") pod \"service-ca-operator-cf7c776b7-z8sdf\" (UID: \"1d7a379e-1344-48f2-9540-7ac11ee14eae\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-z8sdf" Sep 11 08:47:01.058306 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:01.058079 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-xp8k6"] Sep 11 08:47:01.058404 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:01.058350 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-xjs9g" Sep 11 08:47:01.060702 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:01.060679 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Sep 11 08:47:01.060856 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:01.060776 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Sep 11 08:47:01.060856 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:01.060831 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Sep 11 08:47:01.060966 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:01.060945 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-9gxnb\"" Sep 11 08:47:01.061505 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:01.061471 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Sep 11 08:47:01.065205 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:01.065188 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pl9b4\" (UniqueName: \"kubernetes.io/projected/7dae9305-821b-4929-900e-2d7bf09d5654-kube-api-access-pl9b4\") pod \"cluster-samples-operator-86499c89c4-rsjdv\" (UID: \"7dae9305-821b-4929-900e-2d7bf09d5654\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-rsjdv" Sep 11 08:47:01.067750 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:01.067732 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Sep 11 08:47:01.077156 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:01.077139 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-f6twl"] Sep 11 08:47:01.077245 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:01.077165 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-xjs9g"] Sep 11 08:47:01.077245 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:01.077179 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-xp8k6"] Sep 11 08:47:01.077316 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:01.077252 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-xp8k6" Sep 11 08:47:01.080280 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:01.080263 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Sep 11 08:47:01.080600 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:01.080579 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Sep 11 08:47:01.080869 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:01.080847 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-fhp2r\"" Sep 11 08:47:01.150642 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:01.150614 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6wv2\" (UniqueName: \"kubernetes.io/projected/a882da04-e40f-472b-b823-dd7650334c17-kube-api-access-v6wv2\") pod \"router-default-57979bfb5b-9fr79\" (UID: \"a882da04-e40f-472b-b823-dd7650334c17\") " pod="openshift-ingress/router-default-57979bfb5b-9fr79" Sep 11 08:47:01.150757 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:01.150654 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9222c5e5-755c-460e-8568-db40848b8123-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-wdwcs\" (UID: \"9222c5e5-755c-460e-8568-db40848b8123\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-wdwcs" Sep 11 08:47:01.150757 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:01.150680 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/a882da04-e40f-472b-b823-dd7650334c17-default-certificate\") pod \"router-default-57979bfb5b-9fr79\" (UID: \"a882da04-e40f-472b-b823-dd7650334c17\") " pod="openshift-ingress/router-default-57979bfb5b-9fr79" Sep 11 08:47:01.150757 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:01.150730 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9222c5e5-755c-460e-8568-db40848b8123-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-wdwcs\" (UID: \"9222c5e5-755c-460e-8568-db40848b8123\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-wdwcs" Sep 11 08:47:01.150757 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:01.150747 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkd6m\" (UniqueName: \"kubernetes.io/projected/1d7a379e-1344-48f2-9540-7ac11ee14eae-kube-api-access-gkd6m\") pod \"service-ca-operator-cf7c776b7-z8sdf\" (UID: \"1d7a379e-1344-48f2-9540-7ac11ee14eae\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-z8sdf" Sep 11 08:47:01.150966 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:01.150763 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/8a3bbadc-80e8-4436-9c8e-077a0dd03df3-snapshots\") pod \"insights-operator-5844b856db-xjhvj\" (UID: \"8a3bbadc-80e8-4436-9c8e-077a0dd03df3\") " pod="openshift-insights/insights-operator-5844b856db-xjhvj" Sep 11 08:47:01.150966 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:01.150786 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a882da04-e40f-472b-b823-dd7650334c17-service-ca-bundle\") pod \"router-default-57979bfb5b-9fr79\" (UID: \"a882da04-e40f-472b-b823-dd7650334c17\") " pod="openshift-ingress/router-default-57979bfb5b-9fr79" Sep 11 08:47:01.150966 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:01.150816 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8a3bbadc-80e8-4436-9c8e-077a0dd03df3-tmp\") pod \"insights-operator-5844b856db-xjhvj\" (UID: \"8a3bbadc-80e8-4436-9c8e-077a0dd03df3\") " pod="openshift-insights/insights-operator-5844b856db-xjhvj" Sep 11 08:47:01.150966 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:01.150856 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a3bbadc-80e8-4436-9c8e-077a0dd03df3-trusted-ca-bundle\") pod \"insights-operator-5844b856db-xjhvj\" (UID: \"8a3bbadc-80e8-4436-9c8e-077a0dd03df3\") " pod="openshift-insights/insights-operator-5844b856db-xjhvj" Sep 11 08:47:01.150966 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:01.150897 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1d7a379e-1344-48f2-9540-7ac11ee14eae-serving-cert\") pod \"service-ca-operator-cf7c776b7-z8sdf\" (UID: \"1d7a379e-1344-48f2-9540-7ac11ee14eae\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-z8sdf" Sep 11 08:47:01.150966 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:01.150960 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1d7a379e-1344-48f2-9540-7ac11ee14eae-config\") pod \"service-ca-operator-cf7c776b7-z8sdf\" (UID: \"1d7a379e-1344-48f2-9540-7ac11ee14eae\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-z8sdf" Sep 11 08:47:01.151309 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:01.151270 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nc2d4\" (UniqueName: \"kubernetes.io/projected/aa036e63-1ef3-481a-9786-90b2f397ab9a-kube-api-access-nc2d4\") pod \"cluster-monitoring-operator-6cd954b897-blrfj\" (UID: \"aa036e63-1ef3-481a-9786-90b2f397ab9a\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-blrfj" Sep 11 08:47:01.151366 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:01.151310 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bftzf\" (UniqueName: \"kubernetes.io/projected/546705a6-1aaf-49fe-9f7f-3fc9e566437b-kube-api-access-bftzf\") pod \"console-operator-696884fb9c-xjs9g\" (UID: \"546705a6-1aaf-49fe-9f7f-3fc9e566437b\") " pod="openshift-console-operator/console-operator-696884fb9c-xjs9g" Sep 11 08:47:01.151366 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:01.151354 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qn6vk\" (UniqueName: \"kubernetes.io/projected/8a3bbadc-80e8-4436-9c8e-077a0dd03df3-kube-api-access-qn6vk\") pod \"insights-operator-5844b856db-xjhvj\" (UID: \"8a3bbadc-80e8-4436-9c8e-077a0dd03df3\") " pod="openshift-insights/insights-operator-5844b856db-xjhvj" Sep 11 08:47:01.151462 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:01.151342 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9222c5e5-755c-460e-8568-db40848b8123-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-wdwcs\" (UID: \"9222c5e5-755c-460e-8568-db40848b8123\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-wdwcs" Sep 11 08:47:01.151462 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:01.151394 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/aa036e63-1ef3-481a-9786-90b2f397ab9a-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-blrfj\" (UID: \"aa036e63-1ef3-481a-9786-90b2f397ab9a\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-blrfj" Sep 11 08:47:01.151564 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:01.151503 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1d7a379e-1344-48f2-9540-7ac11ee14eae-config\") pod \"service-ca-operator-cf7c776b7-z8sdf\" (UID: \"1d7a379e-1344-48f2-9540-7ac11ee14eae\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-z8sdf" Sep 11 08:47:01.151564 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:01.151543 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/546705a6-1aaf-49fe-9f7f-3fc9e566437b-config\") pod \"console-operator-696884fb9c-xjs9g\" (UID: \"546705a6-1aaf-49fe-9f7f-3fc9e566437b\") " pod="openshift-console-operator/console-operator-696884fb9c-xjs9g" Sep 11 08:47:01.151667 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:01.151613 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbrcb\" (UniqueName: \"kubernetes.io/projected/9222c5e5-755c-460e-8568-db40848b8123-kube-api-access-nbrcb\") pod \"kube-storage-version-migrator-operator-6c8f84f897-wdwcs\" (UID: \"9222c5e5-755c-460e-8568-db40848b8123\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-wdwcs" Sep 11 08:47:01.151667 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:01.151641 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/546705a6-1aaf-49fe-9f7f-3fc9e566437b-trusted-ca\") pod \"console-operator-696884fb9c-xjs9g\" (UID: \"546705a6-1aaf-49fe-9f7f-3fc9e566437b\") " pod="openshift-console-operator/console-operator-696884fb9c-xjs9g" Sep 11 08:47:01.151773 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:01.151667 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9h55\" (UniqueName: \"kubernetes.io/projected/356c3904-2228-43a3-9045-b34e49f0c527-kube-api-access-m9h55\") pod \"network-check-source-6f8b768455-f6twl\" (UID: \"356c3904-2228-43a3-9045-b34e49f0c527\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-f6twl" Sep 11 08:47:01.151773 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:01.151703 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/aa036e63-1ef3-481a-9786-90b2f397ab9a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-blrfj\" (UID: \"aa036e63-1ef3-481a-9786-90b2f397ab9a\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-blrfj" Sep 11 08:47:01.151773 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:01.151725 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/a882da04-e40f-472b-b823-dd7650334c17-stats-auth\") pod \"router-default-57979bfb5b-9fr79\" (UID: \"a882da04-e40f-472b-b823-dd7650334c17\") " pod="openshift-ingress/router-default-57979bfb5b-9fr79" Sep 11 08:47:01.151773 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:01.151745 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8a3bbadc-80e8-4436-9c8e-077a0dd03df3-serving-cert\") pod \"insights-operator-5844b856db-xjhvj\" (UID: \"8a3bbadc-80e8-4436-9c8e-077a0dd03df3\") " pod="openshift-insights/insights-operator-5844b856db-xjhvj" Sep 11 08:47:01.151773 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:01.151770 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a3bbadc-80e8-4436-9c8e-077a0dd03df3-service-ca-bundle\") pod \"insights-operator-5844b856db-xjhvj\" (UID: \"8a3bbadc-80e8-4436-9c8e-077a0dd03df3\") " pod="openshift-insights/insights-operator-5844b856db-xjhvj" Sep 11 08:47:01.152047 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:01.151804 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a882da04-e40f-472b-b823-dd7650334c17-metrics-certs\") pod \"router-default-57979bfb5b-9fr79\" (UID: \"a882da04-e40f-472b-b823-dd7650334c17\") " pod="openshift-ingress/router-default-57979bfb5b-9fr79" Sep 11 08:47:01.152047 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:47:01.151821 2572 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 11 08:47:01.152047 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:01.151827 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/546705a6-1aaf-49fe-9f7f-3fc9e566437b-serving-cert\") pod \"console-operator-696884fb9c-xjs9g\" (UID: \"546705a6-1aaf-49fe-9f7f-3fc9e566437b\") " pod="openshift-console-operator/console-operator-696884fb9c-xjs9g" Sep 11 08:47:01.152047 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:47:01.151877 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aa036e63-1ef3-481a-9786-90b2f397ab9a-cluster-monitoring-operator-tls podName:aa036e63-1ef3-481a-9786-90b2f397ab9a nodeName:}" failed. No retries permitted until 2026-09-11 08:47:01.65186181 +0000 UTC m=+41.926437484 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/aa036e63-1ef3-481a-9786-90b2f397ab9a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-blrfj" (UID: "aa036e63-1ef3-481a-9786-90b2f397ab9a") : secret "cluster-monitoring-operator-tls" not found Sep 11 08:47:01.152295 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:01.152273 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/aa036e63-1ef3-481a-9786-90b2f397ab9a-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-blrfj\" (UID: \"aa036e63-1ef3-481a-9786-90b2f397ab9a\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-blrfj" Sep 11 08:47:01.153771 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:01.153746 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9222c5e5-755c-460e-8568-db40848b8123-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-wdwcs\" (UID: \"9222c5e5-755c-460e-8568-db40848b8123\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-wdwcs" Sep 11 08:47:01.153771 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:01.153756 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1d7a379e-1344-48f2-9540-7ac11ee14eae-serving-cert\") pod \"service-ca-operator-cf7c776b7-z8sdf\" (UID: \"1d7a379e-1344-48f2-9540-7ac11ee14eae\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-z8sdf" Sep 11 08:47:01.165525 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:01.165504 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nc2d4\" (UniqueName: \"kubernetes.io/projected/aa036e63-1ef3-481a-9786-90b2f397ab9a-kube-api-access-nc2d4\") pod \"cluster-monitoring-operator-6cd954b897-blrfj\" (UID: \"aa036e63-1ef3-481a-9786-90b2f397ab9a\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-blrfj" Sep 11 08:47:01.165643 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:01.165623 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkd6m\" (UniqueName: \"kubernetes.io/projected/1d7a379e-1344-48f2-9540-7ac11ee14eae-kube-api-access-gkd6m\") pod \"service-ca-operator-cf7c776b7-z8sdf\" (UID: \"1d7a379e-1344-48f2-9540-7ac11ee14eae\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-z8sdf" Sep 11 08:47:01.166421 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:01.166404 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbrcb\" (UniqueName: \"kubernetes.io/projected/9222c5e5-755c-460e-8568-db40848b8123-kube-api-access-nbrcb\") pod \"kube-storage-version-migrator-operator-6c8f84f897-wdwcs\" (UID: \"9222c5e5-755c-460e-8568-db40848b8123\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-wdwcs" Sep 11 08:47:01.230639 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:01.230612 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-wdwcs" Sep 11 08:47:01.251615 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:01.251590 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-z8sdf" Sep 11 08:47:01.252907 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:01.252880 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwnkn\" (UniqueName: \"kubernetes.io/projected/8382b5d8-4142-449a-93e9-02c5666f9848-kube-api-access-fwnkn\") pod \"volume-data-source-validator-86695b7cc7-xp8k6\" (UID: \"8382b5d8-4142-449a-93e9-02c5666f9848\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-xp8k6" Sep 11 08:47:01.253012 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:01.252965 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/a882da04-e40f-472b-b823-dd7650334c17-stats-auth\") pod \"router-default-57979bfb5b-9fr79\" (UID: \"a882da04-e40f-472b-b823-dd7650334c17\") " pod="openshift-ingress/router-default-57979bfb5b-9fr79" Sep 11 08:47:01.253012 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:01.253001 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8a3bbadc-80e8-4436-9c8e-077a0dd03df3-serving-cert\") pod \"insights-operator-5844b856db-xjhvj\" (UID: \"8a3bbadc-80e8-4436-9c8e-077a0dd03df3\") " pod="openshift-insights/insights-operator-5844b856db-xjhvj" Sep 11 08:47:01.253075 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:01.253029 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a3bbadc-80e8-4436-9c8e-077a0dd03df3-service-ca-bundle\") pod \"insights-operator-5844b856db-xjhvj\" (UID: \"8a3bbadc-80e8-4436-9c8e-077a0dd03df3\") " pod="openshift-insights/insights-operator-5844b856db-xjhvj" Sep 11 08:47:01.253075 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:01.253058 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a882da04-e40f-472b-b823-dd7650334c17-metrics-certs\") pod \"router-default-57979bfb5b-9fr79\" (UID: \"a882da04-e40f-472b-b823-dd7650334c17\") " pod="openshift-ingress/router-default-57979bfb5b-9fr79" Sep 11 08:47:01.253152 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:01.253081 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/546705a6-1aaf-49fe-9f7f-3fc9e566437b-serving-cert\") pod \"console-operator-696884fb9c-xjs9g\" (UID: \"546705a6-1aaf-49fe-9f7f-3fc9e566437b\") " pod="openshift-console-operator/console-operator-696884fb9c-xjs9g" Sep 11 08:47:01.253204 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:47:01.253182 2572 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 11 08:47:01.253268 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:01.253218 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6wv2\" (UniqueName: \"kubernetes.io/projected/a882da04-e40f-472b-b823-dd7650334c17-kube-api-access-v6wv2\") pod \"router-default-57979bfb5b-9fr79\" (UID: \"a882da04-e40f-472b-b823-dd7650334c17\") " pod="openshift-ingress/router-default-57979bfb5b-9fr79" Sep 11 08:47:01.253304 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:47:01.253275 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a882da04-e40f-472b-b823-dd7650334c17-metrics-certs podName:a882da04-e40f-472b-b823-dd7650334c17 nodeName:}" failed. No retries permitted until 2026-09-11 08:47:01.753252566 +0000 UTC m=+42.027828256 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a882da04-e40f-472b-b823-dd7650334c17-metrics-certs") pod "router-default-57979bfb5b-9fr79" (UID: "a882da04-e40f-472b-b823-dd7650334c17") : secret "router-metrics-certs-default" not found Sep 11 08:47:01.253357 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:01.253317 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/a882da04-e40f-472b-b823-dd7650334c17-default-certificate\") pod \"router-default-57979bfb5b-9fr79\" (UID: \"a882da04-e40f-472b-b823-dd7650334c17\") " pod="openshift-ingress/router-default-57979bfb5b-9fr79" Sep 11 08:47:01.253406 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:01.253377 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/8a3bbadc-80e8-4436-9c8e-077a0dd03df3-snapshots\") pod \"insights-operator-5844b856db-xjhvj\" (UID: \"8a3bbadc-80e8-4436-9c8e-077a0dd03df3\") " pod="openshift-insights/insights-operator-5844b856db-xjhvj" Sep 11 08:47:01.253459 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:01.253405 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a882da04-e40f-472b-b823-dd7650334c17-service-ca-bundle\") pod \"router-default-57979bfb5b-9fr79\" (UID: \"a882da04-e40f-472b-b823-dd7650334c17\") " pod="openshift-ingress/router-default-57979bfb5b-9fr79" Sep 11 08:47:01.253459 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:01.253441 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8a3bbadc-80e8-4436-9c8e-077a0dd03df3-tmp\") pod \"insights-operator-5844b856db-xjhvj\" (UID: \"8a3bbadc-80e8-4436-9c8e-077a0dd03df3\") " pod="openshift-insights/insights-operator-5844b856db-xjhvj" Sep 11 08:47:01.253596 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:01.253467 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a3bbadc-80e8-4436-9c8e-077a0dd03df3-trusted-ca-bundle\") pod \"insights-operator-5844b856db-xjhvj\" (UID: \"8a3bbadc-80e8-4436-9c8e-077a0dd03df3\") " pod="openshift-insights/insights-operator-5844b856db-xjhvj" Sep 11 08:47:01.253596 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:01.253504 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bftzf\" (UniqueName: \"kubernetes.io/projected/546705a6-1aaf-49fe-9f7f-3fc9e566437b-kube-api-access-bftzf\") pod \"console-operator-696884fb9c-xjs9g\" (UID: \"546705a6-1aaf-49fe-9f7f-3fc9e566437b\") " pod="openshift-console-operator/console-operator-696884fb9c-xjs9g" Sep 11 08:47:01.253596 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:01.253540 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qn6vk\" (UniqueName: \"kubernetes.io/projected/8a3bbadc-80e8-4436-9c8e-077a0dd03df3-kube-api-access-qn6vk\") pod \"insights-operator-5844b856db-xjhvj\" (UID: \"8a3bbadc-80e8-4436-9c8e-077a0dd03df3\") " pod="openshift-insights/insights-operator-5844b856db-xjhvj" Sep 11 08:47:01.253596 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:01.253576 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/546705a6-1aaf-49fe-9f7f-3fc9e566437b-config\") pod \"console-operator-696884fb9c-xjs9g\" (UID: \"546705a6-1aaf-49fe-9f7f-3fc9e566437b\") " pod="openshift-console-operator/console-operator-696884fb9c-xjs9g" Sep 11 08:47:01.253825 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:01.253640 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/546705a6-1aaf-49fe-9f7f-3fc9e566437b-trusted-ca\") pod \"console-operator-696884fb9c-xjs9g\" (UID: \"546705a6-1aaf-49fe-9f7f-3fc9e566437b\") " pod="openshift-console-operator/console-operator-696884fb9c-xjs9g" Sep 11 08:47:01.253825 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:01.253687 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a3bbadc-80e8-4436-9c8e-077a0dd03df3-service-ca-bundle\") pod \"insights-operator-5844b856db-xjhvj\" (UID: \"8a3bbadc-80e8-4436-9c8e-077a0dd03df3\") " pod="openshift-insights/insights-operator-5844b856db-xjhvj" Sep 11 08:47:01.253825 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:01.253687 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9h55\" (UniqueName: \"kubernetes.io/projected/356c3904-2228-43a3-9045-b34e49f0c527-kube-api-access-m9h55\") pod \"network-check-source-6f8b768455-f6twl\" (UID: \"356c3904-2228-43a3-9045-b34e49f0c527\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-f6twl" Sep 11 08:47:01.254341 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:47:01.254322 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a882da04-e40f-472b-b823-dd7650334c17-service-ca-bundle podName:a882da04-e40f-472b-b823-dd7650334c17 nodeName:}" failed. No retries permitted until 2026-09-11 08:47:01.75430416 +0000 UTC m=+42.028879851 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/a882da04-e40f-472b-b823-dd7650334c17-service-ca-bundle") pod "router-default-57979bfb5b-9fr79" (UID: "a882da04-e40f-472b-b823-dd7650334c17") : configmap references non-existent config key: service-ca.crt Sep 11 08:47:01.254487 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:01.254466 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8a3bbadc-80e8-4436-9c8e-077a0dd03df3-tmp\") pod \"insights-operator-5844b856db-xjhvj\" (UID: \"8a3bbadc-80e8-4436-9c8e-077a0dd03df3\") " pod="openshift-insights/insights-operator-5844b856db-xjhvj" Sep 11 08:47:01.254600 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:01.254577 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/546705a6-1aaf-49fe-9f7f-3fc9e566437b-config\") pod \"console-operator-696884fb9c-xjs9g\" (UID: \"546705a6-1aaf-49fe-9f7f-3fc9e566437b\") " pod="openshift-console-operator/console-operator-696884fb9c-xjs9g" Sep 11 08:47:01.255070 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:01.255050 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/8a3bbadc-80e8-4436-9c8e-077a0dd03df3-snapshots\") pod \"insights-operator-5844b856db-xjhvj\" (UID: \"8a3bbadc-80e8-4436-9c8e-077a0dd03df3\") " pod="openshift-insights/insights-operator-5844b856db-xjhvj" Sep 11 08:47:01.255175 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:01.255127 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/546705a6-1aaf-49fe-9f7f-3fc9e566437b-trusted-ca\") pod \"console-operator-696884fb9c-xjs9g\" (UID: \"546705a6-1aaf-49fe-9f7f-3fc9e566437b\") " pod="openshift-console-operator/console-operator-696884fb9c-xjs9g" Sep 11 08:47:01.255175 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:01.255164 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a3bbadc-80e8-4436-9c8e-077a0dd03df3-trusted-ca-bundle\") pod \"insights-operator-5844b856db-xjhvj\" (UID: \"8a3bbadc-80e8-4436-9c8e-077a0dd03df3\") " pod="openshift-insights/insights-operator-5844b856db-xjhvj" Sep 11 08:47:01.255962 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:01.255868 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/546705a6-1aaf-49fe-9f7f-3fc9e566437b-serving-cert\") pod \"console-operator-696884fb9c-xjs9g\" (UID: \"546705a6-1aaf-49fe-9f7f-3fc9e566437b\") " pod="openshift-console-operator/console-operator-696884fb9c-xjs9g" Sep 11 08:47:01.256195 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:01.256172 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/a882da04-e40f-472b-b823-dd7650334c17-default-certificate\") pod \"router-default-57979bfb5b-9fr79\" (UID: \"a882da04-e40f-472b-b823-dd7650334c17\") " pod="openshift-ingress/router-default-57979bfb5b-9fr79" Sep 11 08:47:01.256331 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:01.256315 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/a882da04-e40f-472b-b823-dd7650334c17-stats-auth\") pod \"router-default-57979bfb5b-9fr79\" (UID: \"a882da04-e40f-472b-b823-dd7650334c17\") " pod="openshift-ingress/router-default-57979bfb5b-9fr79" Sep 11 08:47:01.257015 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:01.256996 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8a3bbadc-80e8-4436-9c8e-077a0dd03df3-serving-cert\") pod \"insights-operator-5844b856db-xjhvj\" (UID: \"8a3bbadc-80e8-4436-9c8e-077a0dd03df3\") " pod="openshift-insights/insights-operator-5844b856db-xjhvj" Sep 11 08:47:01.262714 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:01.262696 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6wv2\" (UniqueName: \"kubernetes.io/projected/a882da04-e40f-472b-b823-dd7650334c17-kube-api-access-v6wv2\") pod \"router-default-57979bfb5b-9fr79\" (UID: \"a882da04-e40f-472b-b823-dd7650334c17\") " pod="openshift-ingress/router-default-57979bfb5b-9fr79" Sep 11 08:47:01.270323 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:01.270300 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qn6vk\" (UniqueName: \"kubernetes.io/projected/8a3bbadc-80e8-4436-9c8e-077a0dd03df3-kube-api-access-qn6vk\") pod \"insights-operator-5844b856db-xjhvj\" (UID: \"8a3bbadc-80e8-4436-9c8e-077a0dd03df3\") " pod="openshift-insights/insights-operator-5844b856db-xjhvj" Sep 11 08:47:01.270416 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:01.270325 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bftzf\" (UniqueName: \"kubernetes.io/projected/546705a6-1aaf-49fe-9f7f-3fc9e566437b-kube-api-access-bftzf\") pod \"console-operator-696884fb9c-xjs9g\" (UID: \"546705a6-1aaf-49fe-9f7f-3fc9e566437b\") " pod="openshift-console-operator/console-operator-696884fb9c-xjs9g" Sep 11 08:47:01.270416 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:01.270401 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9h55\" (UniqueName: \"kubernetes.io/projected/356c3904-2228-43a3-9045-b34e49f0c527-kube-api-access-m9h55\") pod \"network-check-source-6f8b768455-f6twl\" (UID: \"356c3904-2228-43a3-9045-b34e49f0c527\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-f6twl" Sep 11 08:47:01.320520 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:01.320489 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-xjhvj" Sep 11 08:47:01.341818 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:01.341798 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-f6twl" Sep 11 08:47:01.355161 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:01.355065 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwnkn\" (UniqueName: \"kubernetes.io/projected/8382b5d8-4142-449a-93e9-02c5666f9848-kube-api-access-fwnkn\") pod \"volume-data-source-validator-86695b7cc7-xp8k6\" (UID: \"8382b5d8-4142-449a-93e9-02c5666f9848\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-xp8k6" Sep 11 08:47:01.368384 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:01.368358 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwnkn\" (UniqueName: \"kubernetes.io/projected/8382b5d8-4142-449a-93e9-02c5666f9848-kube-api-access-fwnkn\") pod \"volume-data-source-validator-86695b7cc7-xp8k6\" (UID: \"8382b5d8-4142-449a-93e9-02c5666f9848\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-xp8k6" Sep 11 08:47:01.371302 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:01.371276 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-xjs9g" Sep 11 08:47:01.387435 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:01.387396 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-xp8k6" Sep 11 08:47:01.439811 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:01.439738 2572 generic.go:356] "Generic (PLEG): container finished" podID="0c0c64bd-73f4-4cb4-b660-c7232ff8e573" containerID="d4a170ecb2419a454bf29ee216fdf8f4781ddfd52d7d7d57ba28e8511d595ffc" exitCode=0 Sep 11 08:47:01.439811 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:01.439808 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k599z" event={"ID":"0c0c64bd-73f4-4cb4-b660-c7232ff8e573","Type":"ContainerDied","Data":"d4a170ecb2419a454bf29ee216fdf8f4781ddfd52d7d7d57ba28e8511d595ffc"} Sep 11 08:47:01.541164 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:01.541075 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-wdwcs"] Sep 11 08:47:01.544503 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:47:01.544478 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9222c5e5_755c_460e_8568_db40848b8123.slice/crio-5cb3e161934a6994ae9ce82a2d309c2033d1c3fc2c1cebacce4a524f56f5a3cd WatchSource:0}: Error finding container 5cb3e161934a6994ae9ce82a2d309c2033d1c3fc2c1cebacce4a524f56f5a3cd: Status 404 returned error can't find the container with id 5cb3e161934a6994ae9ce82a2d309c2033d1c3fc2c1cebacce4a524f56f5a3cd Sep 11 08:47:01.556784 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:01.556765 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7dae9305-821b-4929-900e-2d7bf09d5654-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-rsjdv\" (UID: \"7dae9305-821b-4929-900e-2d7bf09d5654\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-rsjdv" Sep 11 08:47:01.556936 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:47:01.556920 2572 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 11 08:47:01.557001 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:47:01.556989 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7dae9305-821b-4929-900e-2d7bf09d5654-samples-operator-tls podName:7dae9305-821b-4929-900e-2d7bf09d5654 nodeName:}" failed. No retries permitted until 2026-09-11 08:47:02.556969828 +0000 UTC m=+42.831545506 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7dae9305-821b-4929-900e-2d7bf09d5654-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-rsjdv" (UID: "7dae9305-821b-4929-900e-2d7bf09d5654") : secret "samples-operator-tls" not found Sep 11 08:47:01.568361 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:01.568337 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-z8sdf"] Sep 11 08:47:01.571658 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:47:01.571633 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d7a379e_1344_48f2_9540_7ac11ee14eae.slice/crio-4709988d7e02f619e7c20e9222e961b12af1a20bc3e557994e438c7e3f05267e WatchSource:0}: Error finding container 4709988d7e02f619e7c20e9222e961b12af1a20bc3e557994e438c7e3f05267e: Status 404 returned error can't find the container with id 4709988d7e02f619e7c20e9222e961b12af1a20bc3e557994e438c7e3f05267e Sep 11 08:47:01.619637 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:01.619619 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-xjhvj"] Sep 11 08:47:01.622572 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:47:01.622547 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a3bbadc_80e8_4436_9c8e_077a0dd03df3.slice/crio-c7d8879b98a18cfd8e9e93c028dde291f9a89018d0e0793c02858763ad39d264 WatchSource:0}: Error finding container c7d8879b98a18cfd8e9e93c028dde291f9a89018d0e0793c02858763ad39d264: Status 404 returned error can't find the container with id c7d8879b98a18cfd8e9e93c028dde291f9a89018d0e0793c02858763ad39d264 Sep 11 08:47:01.637917 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:01.637893 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-f6twl"] Sep 11 08:47:01.639586 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:47:01.639565 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod356c3904_2228_43a3_9045_b34e49f0c527.slice/crio-93b008bd5ae07c3cfdefe8ce3135f7836346199f8497b58bc8fb46f9ee69f6ce WatchSource:0}: Error finding container 93b008bd5ae07c3cfdefe8ce3135f7836346199f8497b58bc8fb46f9ee69f6ce: Status 404 returned error can't find the container with id 93b008bd5ae07c3cfdefe8ce3135f7836346199f8497b58bc8fb46f9ee69f6ce Sep 11 08:47:01.658331 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:01.658311 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/aa036e63-1ef3-481a-9786-90b2f397ab9a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-blrfj\" (UID: \"aa036e63-1ef3-481a-9786-90b2f397ab9a\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-blrfj" Sep 11 08:47:01.658504 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:47:01.658473 2572 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 11 08:47:01.658558 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:47:01.658548 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aa036e63-1ef3-481a-9786-90b2f397ab9a-cluster-monitoring-operator-tls podName:aa036e63-1ef3-481a-9786-90b2f397ab9a nodeName:}" failed. No retries permitted until 2026-09-11 08:47:02.658533097 +0000 UTC m=+42.933108771 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/aa036e63-1ef3-481a-9786-90b2f397ab9a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-blrfj" (UID: "aa036e63-1ef3-481a-9786-90b2f397ab9a") : secret "cluster-monitoring-operator-tls" not found Sep 11 08:47:01.687035 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:01.687018 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-xjs9g"] Sep 11 08:47:01.689084 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:47:01.689066 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod546705a6_1aaf_49fe_9f7f_3fc9e566437b.slice/crio-a7ed45cb4e41ce321adcd550ab154ce0cb03618cc93f9e794308a746edfcc1a9 WatchSource:0}: Error finding container a7ed45cb4e41ce321adcd550ab154ce0cb03618cc93f9e794308a746edfcc1a9: Status 404 returned error can't find the container with id a7ed45cb4e41ce321adcd550ab154ce0cb03618cc93f9e794308a746edfcc1a9 Sep 11 08:47:01.711509 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:01.711490 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-xp8k6"] Sep 11 08:47:01.714049 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:47:01.714028 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8382b5d8_4142_449a_93e9_02c5666f9848.slice/crio-5fb4dfb6e2952b3cbe79210debad9893f7aaefbfb0cf0cd7cd4fbe4ad8ac3b1f WatchSource:0}: Error finding container 5fb4dfb6e2952b3cbe79210debad9893f7aaefbfb0cf0cd7cd4fbe4ad8ac3b1f: Status 404 returned error can't find the container with id 5fb4dfb6e2952b3cbe79210debad9893f7aaefbfb0cf0cd7cd4fbe4ad8ac3b1f Sep 11 08:47:01.758925 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:01.758902 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a882da04-e40f-472b-b823-dd7650334c17-metrics-certs\") pod \"router-default-57979bfb5b-9fr79\" (UID: \"a882da04-e40f-472b-b823-dd7650334c17\") " pod="openshift-ingress/router-default-57979bfb5b-9fr79" Sep 11 08:47:01.759009 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:01.758949 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a882da04-e40f-472b-b823-dd7650334c17-service-ca-bundle\") pod \"router-default-57979bfb5b-9fr79\" (UID: \"a882da04-e40f-472b-b823-dd7650334c17\") " pod="openshift-ingress/router-default-57979bfb5b-9fr79" Sep 11 08:47:01.759061 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:47:01.759052 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a882da04-e40f-472b-b823-dd7650334c17-service-ca-bundle podName:a882da04-e40f-472b-b823-dd7650334c17 nodeName:}" failed. No retries permitted until 2026-09-11 08:47:02.759038991 +0000 UTC m=+43.033614665 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/a882da04-e40f-472b-b823-dd7650334c17-service-ca-bundle") pod "router-default-57979bfb5b-9fr79" (UID: "a882da04-e40f-472b-b823-dd7650334c17") : configmap references non-existent config key: service-ca.crt Sep 11 08:47:01.759116 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:47:01.759058 2572 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 11 08:47:01.759169 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:47:01.759114 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a882da04-e40f-472b-b823-dd7650334c17-metrics-certs podName:a882da04-e40f-472b-b823-dd7650334c17 nodeName:}" failed. No retries permitted until 2026-09-11 08:47:02.759096898 +0000 UTC m=+43.033672586 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a882da04-e40f-472b-b823-dd7650334c17-metrics-certs") pod "router-default-57979bfb5b-9fr79" (UID: "a882da04-e40f-472b-b823-dd7650334c17") : secret "router-metrics-certs-default" not found Sep 11 08:47:01.859777 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:01.859691 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/64b9c16d-d0c5-423f-98e3-67d3fcff6128-registry-tls\") pod \"image-registry-6fdf7ff479-xlfkd\" (UID: \"64b9c16d-d0c5-423f-98e3-67d3fcff6128\") " pod="openshift-image-registry/image-registry-6fdf7ff479-xlfkd" Sep 11 08:47:01.859926 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:47:01.859837 2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 11 08:47:01.859926 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:47:01.859856 2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6fdf7ff479-xlfkd: secret "image-registry-tls" not found Sep 11 08:47:01.859926 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:47:01.859903 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/64b9c16d-d0c5-423f-98e3-67d3fcff6128-registry-tls podName:64b9c16d-d0c5-423f-98e3-67d3fcff6128 nodeName:}" failed. No retries permitted until 2026-09-11 08:47:09.859888235 +0000 UTC m=+50.134463908 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/64b9c16d-d0c5-423f-98e3-67d3fcff6128-registry-tls") pod "image-registry-6fdf7ff479-xlfkd" (UID: "64b9c16d-d0c5-423f-98e3-67d3fcff6128") : secret "image-registry-tls" not found Sep 11 08:47:01.859926 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:01.859841 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8ea2d42c-f484-483d-8714-603b8f05cc54-metrics-tls\") pod \"dns-default-d67t9\" (UID: \"8ea2d42c-f484-483d-8714-603b8f05cc54\") " pod="openshift-dns/dns-default-d67t9" Sep 11 08:47:01.859926 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:47:01.859925 2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 08:47:01.860128 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:47:01.859980 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ea2d42c-f484-483d-8714-603b8f05cc54-metrics-tls podName:8ea2d42c-f484-483d-8714-603b8f05cc54 nodeName:}" failed. No retries permitted until 2026-09-11 08:47:09.859964998 +0000 UTC m=+50.134540690 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8ea2d42c-f484-483d-8714-603b8f05cc54-metrics-tls") pod "dns-default-d67t9" (UID: "8ea2d42c-f484-483d-8714-603b8f05cc54") : secret "dns-default-metrics-tls" not found Sep 11 08:47:02.444962 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:02.444921 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-f6twl" event={"ID":"356c3904-2228-43a3-9045-b34e49f0c527","Type":"ContainerStarted","Data":"93b008bd5ae07c3cfdefe8ce3135f7836346199f8497b58bc8fb46f9ee69f6ce"} Sep 11 08:47:02.446568 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:02.446537 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-xjhvj" event={"ID":"8a3bbadc-80e8-4436-9c8e-077a0dd03df3","Type":"ContainerStarted","Data":"c7d8879b98a18cfd8e9e93c028dde291f9a89018d0e0793c02858763ad39d264"} Sep 11 08:47:02.448668 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:02.448641 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-z8sdf" event={"ID":"1d7a379e-1344-48f2-9540-7ac11ee14eae","Type":"ContainerStarted","Data":"4709988d7e02f619e7c20e9222e961b12af1a20bc3e557994e438c7e3f05267e"} Sep 11 08:47:02.450078 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:02.450051 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-xjs9g" event={"ID":"546705a6-1aaf-49fe-9f7f-3fc9e566437b","Type":"ContainerStarted","Data":"a7ed45cb4e41ce321adcd550ab154ce0cb03618cc93f9e794308a746edfcc1a9"} Sep 11 08:47:02.452386 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:02.452358 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-wdwcs" event={"ID":"9222c5e5-755c-460e-8568-db40848b8123","Type":"ContainerStarted","Data":"5cb3e161934a6994ae9ce82a2d309c2033d1c3fc2c1cebacce4a524f56f5a3cd"} Sep 11 08:47:02.458611 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:02.457752 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k599z" event={"ID":"0c0c64bd-73f4-4cb4-b660-c7232ff8e573","Type":"ContainerStarted","Data":"43fea24307d11cad83580d4a2d265d668763018007ac9dbfa22dc76d8ee98ac6"} Sep 11 08:47:02.460078 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:02.460033 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-xp8k6" event={"ID":"8382b5d8-4142-449a-93e9-02c5666f9848","Type":"ContainerStarted","Data":"5fb4dfb6e2952b3cbe79210debad9893f7aaefbfb0cf0cd7cd4fbe4ad8ac3b1f"} Sep 11 08:47:02.483951 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:02.483896 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-k599z" podStartSLOduration=9.79965023 podStartE2EDuration="42.4838768s" totalImagesPullingTime="32.68422657s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 08:46:20 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 08:47:02.481987267 +0000 UTC m=+42.756562963" watchObservedRunningTime="2026-09-11 08:47:02.4838768 +0000 UTC m=+42.758452498" Sep 11 08:47:02.568424 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:02.567635 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7dae9305-821b-4929-900e-2d7bf09d5654-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-rsjdv\" (UID: \"7dae9305-821b-4929-900e-2d7bf09d5654\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-rsjdv" Sep 11 08:47:02.568424 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:47:02.568336 2572 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 11 08:47:02.568424 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:47:02.568418 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7dae9305-821b-4929-900e-2d7bf09d5654-samples-operator-tls podName:7dae9305-821b-4929-900e-2d7bf09d5654 nodeName:}" failed. No retries permitted until 2026-09-11 08:47:04.568399165 +0000 UTC m=+44.842974858 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7dae9305-821b-4929-900e-2d7bf09d5654-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-rsjdv" (UID: "7dae9305-821b-4929-900e-2d7bf09d5654") : secret "samples-operator-tls" not found Sep 11 08:47:02.668572 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:02.668536 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/aa036e63-1ef3-481a-9786-90b2f397ab9a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-blrfj\" (UID: \"aa036e63-1ef3-481a-9786-90b2f397ab9a\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-blrfj" Sep 11 08:47:02.668768 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:47:02.668716 2572 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 11 08:47:02.668856 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:47:02.668775 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aa036e63-1ef3-481a-9786-90b2f397ab9a-cluster-monitoring-operator-tls podName:aa036e63-1ef3-481a-9786-90b2f397ab9a nodeName:}" failed. No retries permitted until 2026-09-11 08:47:04.668756536 +0000 UTC m=+44.943332223 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/aa036e63-1ef3-481a-9786-90b2f397ab9a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-blrfj" (UID: "aa036e63-1ef3-481a-9786-90b2f397ab9a") : secret "cluster-monitoring-operator-tls" not found Sep 11 08:47:02.769539 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:02.769456 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a882da04-e40f-472b-b823-dd7650334c17-service-ca-bundle\") pod \"router-default-57979bfb5b-9fr79\" (UID: \"a882da04-e40f-472b-b823-dd7650334c17\") " pod="openshift-ingress/router-default-57979bfb5b-9fr79" Sep 11 08:47:02.769681 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:02.769603 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a882da04-e40f-472b-b823-dd7650334c17-metrics-certs\") pod \"router-default-57979bfb5b-9fr79\" (UID: \"a882da04-e40f-472b-b823-dd7650334c17\") " pod="openshift-ingress/router-default-57979bfb5b-9fr79" Sep 11 08:47:02.769776 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:47:02.769758 2572 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 11 08:47:02.769838 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:47:02.769827 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a882da04-e40f-472b-b823-dd7650334c17-metrics-certs podName:a882da04-e40f-472b-b823-dd7650334c17 nodeName:}" failed. No retries permitted until 2026-09-11 08:47:04.769808356 +0000 UTC m=+45.044384047 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a882da04-e40f-472b-b823-dd7650334c17-metrics-certs") pod "router-default-57979bfb5b-9fr79" (UID: "a882da04-e40f-472b-b823-dd7650334c17") : secret "router-metrics-certs-default" not found Sep 11 08:47:02.770278 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:47:02.770260 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a882da04-e40f-472b-b823-dd7650334c17-service-ca-bundle podName:a882da04-e40f-472b-b823-dd7650334c17 nodeName:}" failed. No retries permitted until 2026-09-11 08:47:04.770242515 +0000 UTC m=+45.044818208 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/a882da04-e40f-472b-b823-dd7650334c17-service-ca-bundle") pod "router-default-57979bfb5b-9fr79" (UID: "a882da04-e40f-472b-b823-dd7650334c17") : configmap references non-existent config key: service-ca.crt Sep 11 08:47:04.588573 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:04.588540 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7dae9305-821b-4929-900e-2d7bf09d5654-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-rsjdv\" (UID: \"7dae9305-821b-4929-900e-2d7bf09d5654\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-rsjdv" Sep 11 08:47:04.588969 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:47:04.588699 2572 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 11 08:47:04.588969 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:47:04.588768 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7dae9305-821b-4929-900e-2d7bf09d5654-samples-operator-tls podName:7dae9305-821b-4929-900e-2d7bf09d5654 nodeName:}" failed. No retries permitted until 2026-09-11 08:47:08.588750352 +0000 UTC m=+48.863326037 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7dae9305-821b-4929-900e-2d7bf09d5654-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-rsjdv" (UID: "7dae9305-821b-4929-900e-2d7bf09d5654") : secret "samples-operator-tls" not found Sep 11 08:47:04.689655 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:04.689619 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/aa036e63-1ef3-481a-9786-90b2f397ab9a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-blrfj\" (UID: \"aa036e63-1ef3-481a-9786-90b2f397ab9a\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-blrfj" Sep 11 08:47:04.689818 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:47:04.689762 2572 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 11 08:47:04.689863 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:47:04.689834 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aa036e63-1ef3-481a-9786-90b2f397ab9a-cluster-monitoring-operator-tls podName:aa036e63-1ef3-481a-9786-90b2f397ab9a nodeName:}" failed. No retries permitted until 2026-09-11 08:47:08.689813686 +0000 UTC m=+48.964389375 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/aa036e63-1ef3-481a-9786-90b2f397ab9a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-blrfj" (UID: "aa036e63-1ef3-481a-9786-90b2f397ab9a") : secret "cluster-monitoring-operator-tls" not found Sep 11 08:47:04.790185 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:04.790141 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a882da04-e40f-472b-b823-dd7650334c17-service-ca-bundle\") pod \"router-default-57979bfb5b-9fr79\" (UID: \"a882da04-e40f-472b-b823-dd7650334c17\") " pod="openshift-ingress/router-default-57979bfb5b-9fr79" Sep 11 08:47:04.790362 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:04.790306 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a882da04-e40f-472b-b823-dd7650334c17-metrics-certs\") pod \"router-default-57979bfb5b-9fr79\" (UID: \"a882da04-e40f-472b-b823-dd7650334c17\") " pod="openshift-ingress/router-default-57979bfb5b-9fr79" Sep 11 08:47:04.790362 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:47:04.790323 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a882da04-e40f-472b-b823-dd7650334c17-service-ca-bundle podName:a882da04-e40f-472b-b823-dd7650334c17 nodeName:}" failed. No retries permitted until 2026-09-11 08:47:08.790304405 +0000 UTC m=+49.064880098 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/a882da04-e40f-472b-b823-dd7650334c17-service-ca-bundle") pod "router-default-57979bfb5b-9fr79" (UID: "a882da04-e40f-472b-b823-dd7650334c17") : configmap references non-existent config key: service-ca.crt Sep 11 08:47:04.790482 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:47:04.790387 2572 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 11 08:47:04.790482 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:47:04.790432 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a882da04-e40f-472b-b823-dd7650334c17-metrics-certs podName:a882da04-e40f-472b-b823-dd7650334c17 nodeName:}" failed. No retries permitted until 2026-09-11 08:47:08.790419488 +0000 UTC m=+49.064995162 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a882da04-e40f-472b-b823-dd7650334c17-metrics-certs") pod "router-default-57979bfb5b-9fr79" (UID: "a882da04-e40f-472b-b823-dd7650334c17") : secret "router-metrics-certs-default" not found Sep 11 08:47:08.478871 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:08.478843 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-wdwcs" event={"ID":"9222c5e5-755c-460e-8568-db40848b8123","Type":"ContainerStarted","Data":"3e3b15d35c6ad814d1a61a29f22a86aa7167be0b01231763adad2ce0d8f93aac"} Sep 11 08:47:08.489274 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:08.489016 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-xp8k6" event={"ID":"8382b5d8-4142-449a-93e9-02c5666f9848","Type":"ContainerStarted","Data":"affef451779d0b156887991ff3a6bbc3d67649657069d80ab72f3a47f6ec5483"} Sep 11 08:47:08.490702 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:08.490679 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-xjhvj" event={"ID":"8a3bbadc-80e8-4436-9c8e-077a0dd03df3","Type":"ContainerStarted","Data":"14fde40295bb309abcd85304c4a727a71b17c41cdbda28bf67a17a8ed499f6e5"} Sep 11 08:47:08.492258 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:08.492215 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-z8sdf" event={"ID":"1d7a379e-1344-48f2-9540-7ac11ee14eae","Type":"ContainerStarted","Data":"86f878c7af8a075aeaa91c1f3e43973413a38bcaf80ee60387b4610ff32092f6"} Sep 11 08:47:08.499986 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:08.499630 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-wdwcs" podStartSLOduration=1.9684883119999999 podStartE2EDuration="8.499612673s" totalImagesPullingTime="6.531124361s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 08:47:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 08:47:08.499021831 +0000 UTC m=+48.773597527" watchObservedRunningTime="2026-09-11 08:47:08.499612673 +0000 UTC m=+48.774188370" Sep 11 08:47:08.532595 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:08.532418 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-xp8k6" podStartSLOduration=2.194735456 podStartE2EDuration="8.532381341s" totalImagesPullingTime="6.337645885s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 08:47:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 08:47:08.531868855 +0000 UTC m=+48.806444552" watchObservedRunningTime="2026-09-11 08:47:08.532381341 +0000 UTC m=+48.806957034" Sep 11 08:47:08.550887 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:08.550555 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-z8sdf" podStartSLOduration=2.046809032 podStartE2EDuration="8.550534945s" totalImagesPullingTime="6.503725913s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 08:47:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 08:47:08.549401684 +0000 UTC m=+48.823977380" watchObservedRunningTime="2026-09-11 08:47:08.550534945 +0000 UTC m=+48.825110641" Sep 11 08:47:08.571910 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:08.571736 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5844b856db-xjhvj" podStartSLOduration=2.11992439 podStartE2EDuration="8.571716931s" totalImagesPullingTime="6.451792541s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 08:47:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 08:47:08.569919183 +0000 UTC m=+48.844494881" watchObservedRunningTime="2026-09-11 08:47:08.571716931 +0000 UTC m=+48.846292628" Sep 11 08:47:08.622325 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:08.622295 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7dae9305-821b-4929-900e-2d7bf09d5654-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-rsjdv\" (UID: \"7dae9305-821b-4929-900e-2d7bf09d5654\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-rsjdv" Sep 11 08:47:08.622435 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:47:08.622424 2572 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 11 08:47:08.622498 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:47:08.622473 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7dae9305-821b-4929-900e-2d7bf09d5654-samples-operator-tls podName:7dae9305-821b-4929-900e-2d7bf09d5654 nodeName:}" failed. No retries permitted until 2026-09-11 08:47:16.622457646 +0000 UTC m=+56.897033323 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7dae9305-821b-4929-900e-2d7bf09d5654-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-rsjdv" (UID: "7dae9305-821b-4929-900e-2d7bf09d5654") : secret "samples-operator-tls" not found Sep 11 08:47:08.723379 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:08.723343 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/aa036e63-1ef3-481a-9786-90b2f397ab9a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-blrfj\" (UID: \"aa036e63-1ef3-481a-9786-90b2f397ab9a\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-blrfj" Sep 11 08:47:08.723537 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:47:08.723474 2572 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 11 08:47:08.723537 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:47:08.723536 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aa036e63-1ef3-481a-9786-90b2f397ab9a-cluster-monitoring-operator-tls podName:aa036e63-1ef3-481a-9786-90b2f397ab9a nodeName:}" failed. No retries permitted until 2026-09-11 08:47:16.723517795 +0000 UTC m=+56.998093471 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/aa036e63-1ef3-481a-9786-90b2f397ab9a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-blrfj" (UID: "aa036e63-1ef3-481a-9786-90b2f397ab9a") : secret "cluster-monitoring-operator-tls" not found Sep 11 08:47:08.824424 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:08.824347 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a882da04-e40f-472b-b823-dd7650334c17-metrics-certs\") pod \"router-default-57979bfb5b-9fr79\" (UID: \"a882da04-e40f-472b-b823-dd7650334c17\") " pod="openshift-ingress/router-default-57979bfb5b-9fr79" Sep 11 08:47:08.824551 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:08.824432 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a882da04-e40f-472b-b823-dd7650334c17-service-ca-bundle\") pod \"router-default-57979bfb5b-9fr79\" (UID: \"a882da04-e40f-472b-b823-dd7650334c17\") " pod="openshift-ingress/router-default-57979bfb5b-9fr79" Sep 11 08:47:08.824551 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:47:08.824502 2572 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 11 08:47:08.824662 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:47:08.824573 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a882da04-e40f-472b-b823-dd7650334c17-service-ca-bundle podName:a882da04-e40f-472b-b823-dd7650334c17 nodeName:}" failed. No retries permitted until 2026-09-11 08:47:16.824555873 +0000 UTC m=+57.099131561 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/a882da04-e40f-472b-b823-dd7650334c17-service-ca-bundle") pod "router-default-57979bfb5b-9fr79" (UID: "a882da04-e40f-472b-b823-dd7650334c17") : configmap references non-existent config key: service-ca.crt Sep 11 08:47:08.824662 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:47:08.824595 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a882da04-e40f-472b-b823-dd7650334c17-metrics-certs podName:a882da04-e40f-472b-b823-dd7650334c17 nodeName:}" failed. No retries permitted until 2026-09-11 08:47:16.82458523 +0000 UTC m=+57.099160909 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a882da04-e40f-472b-b823-dd7650334c17-metrics-certs") pod "router-default-57979bfb5b-9fr79" (UID: "a882da04-e40f-472b-b823-dd7650334c17") : secret "router-metrics-certs-default" not found Sep 11 08:47:09.497558 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:09.497522 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-f6twl" event={"ID":"356c3904-2228-43a3-9045-b34e49f0c527","Type":"ContainerStarted","Data":"660ac4f6da1f062f35fb834110e6bf84e6ee9299f9f0d6bbae674ac725945673"} Sep 11 08:47:09.499683 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:09.499663 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-xjs9g_546705a6-1aaf-49fe-9f7f-3fc9e566437b/console-operator/0.log" Sep 11 08:47:09.499803 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:09.499700 2572 generic.go:356] "Generic (PLEG): container finished" podID="546705a6-1aaf-49fe-9f7f-3fc9e566437b" containerID="4e5f26bbdcb29980a9ea3a50823c11a3c32cf415db5112b58baa6e31a5528320" exitCode=255 Sep 11 08:47:09.499867 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:09.499788 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-xjs9g" event={"ID":"546705a6-1aaf-49fe-9f7f-3fc9e566437b","Type":"ContainerDied","Data":"4e5f26bbdcb29980a9ea3a50823c11a3c32cf415db5112b58baa6e31a5528320"} Sep 11 08:47:09.499995 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:09.499979 2572 scope.go:117] "RemoveContainer" containerID="4e5f26bbdcb29980a9ea3a50823c11a3c32cf415db5112b58baa6e31a5528320" Sep 11 08:47:09.511780 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:09.511735 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f8b768455-f6twl" podStartSLOduration=2.677355371 podStartE2EDuration="9.511721724s" totalImagesPullingTime="6.834366353s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 08:47:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 08:47:09.510436043 +0000 UTC m=+49.785011750" watchObservedRunningTime="2026-09-11 08:47:09.511721724 +0000 UTC m=+49.786297420" Sep 11 08:47:09.937022 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:09.936985 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8ea2d42c-f484-483d-8714-603b8f05cc54-metrics-tls\") pod \"dns-default-d67t9\" (UID: \"8ea2d42c-f484-483d-8714-603b8f05cc54\") " pod="openshift-dns/dns-default-d67t9" Sep 11 08:47:09.937180 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:09.937045 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/64b9c16d-d0c5-423f-98e3-67d3fcff6128-registry-tls\") pod \"image-registry-6fdf7ff479-xlfkd\" (UID: \"64b9c16d-d0c5-423f-98e3-67d3fcff6128\") " pod="openshift-image-registry/image-registry-6fdf7ff479-xlfkd" Sep 11 08:47:09.937180 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:47:09.937147 2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 08:47:09.937323 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:47:09.937241 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ea2d42c-f484-483d-8714-603b8f05cc54-metrics-tls podName:8ea2d42c-f484-483d-8714-603b8f05cc54 nodeName:}" failed. No retries permitted until 2026-09-11 08:47:25.937200904 +0000 UTC m=+66.211776593 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8ea2d42c-f484-483d-8714-603b8f05cc54-metrics-tls") pod "dns-default-d67t9" (UID: "8ea2d42c-f484-483d-8714-603b8f05cc54") : secret "dns-default-metrics-tls" not found Sep 11 08:47:09.937323 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:47:09.937238 2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 11 08:47:09.937323 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:47:09.937264 2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6fdf7ff479-xlfkd: secret "image-registry-tls" not found Sep 11 08:47:09.937433 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:47:09.937338 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/64b9c16d-d0c5-423f-98e3-67d3fcff6128-registry-tls podName:64b9c16d-d0c5-423f-98e3-67d3fcff6128 nodeName:}" failed. No retries permitted until 2026-09-11 08:47:25.937319233 +0000 UTC m=+66.211894925 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/64b9c16d-d0c5-423f-98e3-67d3fcff6128-registry-tls") pod "image-registry-6fdf7ff479-xlfkd" (UID: "64b9c16d-d0c5-423f-98e3-67d3fcff6128") : secret "image-registry-tls" not found Sep 11 08:47:10.303540 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:10.303454 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-srrd7"] Sep 11 08:47:10.306955 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:10.306935 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-srrd7" Sep 11 08:47:10.309469 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:10.309441 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 11 08:47:10.309563 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:10.309451 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-xbkxl\"" Sep 11 08:47:10.309784 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:10.309764 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 11 08:47:10.333410 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:10.333385 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-srrd7"] Sep 11 08:47:10.340470 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:10.340450 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b67622ee-ebc8-4db2-8211-df0e6e15cefd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-srrd7\" (UID: \"b67622ee-ebc8-4db2-8211-df0e6e15cefd\") " pod="openshift-insights/insights-runtime-extractor-srrd7" Sep 11 08:47:10.340548 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:10.340502 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b67622ee-ebc8-4db2-8211-df0e6e15cefd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-srrd7\" (UID: \"b67622ee-ebc8-4db2-8211-df0e6e15cefd\") " pod="openshift-insights/insights-runtime-extractor-srrd7" Sep 11 08:47:10.340605 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:10.340552 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b67622ee-ebc8-4db2-8211-df0e6e15cefd-data-volume\") pod \"insights-runtime-extractor-srrd7\" (UID: \"b67622ee-ebc8-4db2-8211-df0e6e15cefd\") " pod="openshift-insights/insights-runtime-extractor-srrd7" Sep 11 08:47:10.340605 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:10.340568 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b67622ee-ebc8-4db2-8211-df0e6e15cefd-crio-socket\") pod \"insights-runtime-extractor-srrd7\" (UID: \"b67622ee-ebc8-4db2-8211-df0e6e15cefd\") " pod="openshift-insights/insights-runtime-extractor-srrd7" Sep 11 08:47:10.340605 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:10.340598 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6j85\" (UniqueName: \"kubernetes.io/projected/b67622ee-ebc8-4db2-8211-df0e6e15cefd-kube-api-access-w6j85\") pod \"insights-runtime-extractor-srrd7\" (UID: \"b67622ee-ebc8-4db2-8211-df0e6e15cefd\") " pod="openshift-insights/insights-runtime-extractor-srrd7" Sep 11 08:47:10.441668 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:10.441636 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b67622ee-ebc8-4db2-8211-df0e6e15cefd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-srrd7\" (UID: \"b67622ee-ebc8-4db2-8211-df0e6e15cefd\") " pod="openshift-insights/insights-runtime-extractor-srrd7" Sep 11 08:47:10.441803 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:10.441706 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b67622ee-ebc8-4db2-8211-df0e6e15cefd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-srrd7\" (UID: \"b67622ee-ebc8-4db2-8211-df0e6e15cefd\") " pod="openshift-insights/insights-runtime-extractor-srrd7" Sep 11 08:47:10.441803 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:47:10.441754 2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 11 08:47:10.441803 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:10.441755 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b67622ee-ebc8-4db2-8211-df0e6e15cefd-data-volume\") pod \"insights-runtime-extractor-srrd7\" (UID: \"b67622ee-ebc8-4db2-8211-df0e6e15cefd\") " pod="openshift-insights/insights-runtime-extractor-srrd7" Sep 11 08:47:10.441953 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:47:10.441817 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b67622ee-ebc8-4db2-8211-df0e6e15cefd-kube-rbac-proxy-cm podName:b67622ee-ebc8-4db2-8211-df0e6e15cefd nodeName:}" failed. No retries permitted until 2026-09-11 08:47:10.941797008 +0000 UTC m=+51.216372690 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b67622ee-ebc8-4db2-8211-df0e6e15cefd-kube-rbac-proxy-cm") pod "insights-runtime-extractor-srrd7" (UID: "b67622ee-ebc8-4db2-8211-df0e6e15cefd") : configmap "kube-rbac-proxy" not found Sep 11 08:47:10.441953 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:10.441846 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b67622ee-ebc8-4db2-8211-df0e6e15cefd-crio-socket\") pod \"insights-runtime-extractor-srrd7\" (UID: \"b67622ee-ebc8-4db2-8211-df0e6e15cefd\") " pod="openshift-insights/insights-runtime-extractor-srrd7" Sep 11 08:47:10.441953 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:47:10.441855 2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 11 08:47:10.441953 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:10.441894 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6j85\" (UniqueName: \"kubernetes.io/projected/b67622ee-ebc8-4db2-8211-df0e6e15cefd-kube-api-access-w6j85\") pod \"insights-runtime-extractor-srrd7\" (UID: \"b67622ee-ebc8-4db2-8211-df0e6e15cefd\") " pod="openshift-insights/insights-runtime-extractor-srrd7" Sep 11 08:47:10.441953 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:47:10.441945 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b67622ee-ebc8-4db2-8211-df0e6e15cefd-insights-runtime-extractor-tls podName:b67622ee-ebc8-4db2-8211-df0e6e15cefd nodeName:}" failed. No retries permitted until 2026-09-11 08:47:10.941933846 +0000 UTC m=+51.216509524 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b67622ee-ebc8-4db2-8211-df0e6e15cefd-insights-runtime-extractor-tls") pod "insights-runtime-extractor-srrd7" (UID: "b67622ee-ebc8-4db2-8211-df0e6e15cefd") : secret "insights-runtime-extractor-tls" not found Sep 11 08:47:10.442171 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:10.442018 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b67622ee-ebc8-4db2-8211-df0e6e15cefd-crio-socket\") pod \"insights-runtime-extractor-srrd7\" (UID: \"b67622ee-ebc8-4db2-8211-df0e6e15cefd\") " pod="openshift-insights/insights-runtime-extractor-srrd7" Sep 11 08:47:10.442171 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:10.442057 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b67622ee-ebc8-4db2-8211-df0e6e15cefd-data-volume\") pod \"insights-runtime-extractor-srrd7\" (UID: \"b67622ee-ebc8-4db2-8211-df0e6e15cefd\") " pod="openshift-insights/insights-runtime-extractor-srrd7" Sep 11 08:47:10.449763 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:10.449740 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6j85\" (UniqueName: \"kubernetes.io/projected/b67622ee-ebc8-4db2-8211-df0e6e15cefd-kube-api-access-w6j85\") pod \"insights-runtime-extractor-srrd7\" (UID: \"b67622ee-ebc8-4db2-8211-df0e6e15cefd\") " pod="openshift-insights/insights-runtime-extractor-srrd7" Sep 11 08:47:10.505335 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:10.505312 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-xjs9g_546705a6-1aaf-49fe-9f7f-3fc9e566437b/console-operator/1.log" Sep 11 08:47:10.505716 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:10.505701 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-xjs9g_546705a6-1aaf-49fe-9f7f-3fc9e566437b/console-operator/0.log" Sep 11 08:47:10.505762 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:10.505736 2572 generic.go:356] "Generic (PLEG): container finished" podID="546705a6-1aaf-49fe-9f7f-3fc9e566437b" containerID="fe209c91bcf7683166fbd4b4a75c65e5b07a451b291a217ac7d895d6b0768122" exitCode=255 Sep 11 08:47:10.505819 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:10.505807 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-xjs9g" event={"ID":"546705a6-1aaf-49fe-9f7f-3fc9e566437b","Type":"ContainerDied","Data":"fe209c91bcf7683166fbd4b4a75c65e5b07a451b291a217ac7d895d6b0768122"} Sep 11 08:47:10.505855 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:10.505850 2572 scope.go:117] "RemoveContainer" containerID="4e5f26bbdcb29980a9ea3a50823c11a3c32cf415db5112b58baa6e31a5528320" Sep 11 08:47:10.506050 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:10.506034 2572 scope.go:117] "RemoveContainer" containerID="fe209c91bcf7683166fbd4b4a75c65e5b07a451b291a217ac7d895d6b0768122" Sep 11 08:47:10.506257 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:47:10.506217 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-xjs9g_openshift-console-operator(546705a6-1aaf-49fe-9f7f-3fc9e566437b)\"" pod="openshift-console-operator/console-operator-696884fb9c-xjs9g" podUID="546705a6-1aaf-49fe-9f7f-3fc9e566437b" Sep 11 08:47:10.944699 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:10.944665 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b67622ee-ebc8-4db2-8211-df0e6e15cefd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-srrd7\" (UID: \"b67622ee-ebc8-4db2-8211-df0e6e15cefd\") " pod="openshift-insights/insights-runtime-extractor-srrd7" Sep 11 08:47:10.944928 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:10.944816 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b67622ee-ebc8-4db2-8211-df0e6e15cefd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-srrd7\" (UID: \"b67622ee-ebc8-4db2-8211-df0e6e15cefd\") " pod="openshift-insights/insights-runtime-extractor-srrd7" Sep 11 08:47:10.944928 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:47:10.944833 2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 11 08:47:10.944928 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:47:10.944894 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b67622ee-ebc8-4db2-8211-df0e6e15cefd-insights-runtime-extractor-tls podName:b67622ee-ebc8-4db2-8211-df0e6e15cefd nodeName:}" failed. No retries permitted until 2026-09-11 08:47:11.944876454 +0000 UTC m=+52.219452147 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b67622ee-ebc8-4db2-8211-df0e6e15cefd-insights-runtime-extractor-tls") pod "insights-runtime-extractor-srrd7" (UID: "b67622ee-ebc8-4db2-8211-df0e6e15cefd") : secret "insights-runtime-extractor-tls" not found Sep 11 08:47:10.945388 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:10.945371 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b67622ee-ebc8-4db2-8211-df0e6e15cefd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-srrd7\" (UID: \"b67622ee-ebc8-4db2-8211-df0e6e15cefd\") " pod="openshift-insights/insights-runtime-extractor-srrd7" Sep 11 08:47:11.372443 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:11.372365 2572 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-xjs9g" Sep 11 08:47:11.372443 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:11.372415 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-xjs9g" Sep 11 08:47:11.510380 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:11.510349 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-xjs9g_546705a6-1aaf-49fe-9f7f-3fc9e566437b/console-operator/1.log" Sep 11 08:47:11.510726 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:11.510655 2572 scope.go:117] "RemoveContainer" containerID="fe209c91bcf7683166fbd4b4a75c65e5b07a451b291a217ac7d895d6b0768122" Sep 11 08:47:11.510818 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:47:11.510800 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-xjs9g_openshift-console-operator(546705a6-1aaf-49fe-9f7f-3fc9e566437b)\"" pod="openshift-console-operator/console-operator-696884fb9c-xjs9g" podUID="546705a6-1aaf-49fe-9f7f-3fc9e566437b" Sep 11 08:47:11.952247 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:11.952199 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b67622ee-ebc8-4db2-8211-df0e6e15cefd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-srrd7\" (UID: \"b67622ee-ebc8-4db2-8211-df0e6e15cefd\") " pod="openshift-insights/insights-runtime-extractor-srrd7" Sep 11 08:47:11.952431 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:47:11.952368 2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 11 08:47:11.952497 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:47:11.952448 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b67622ee-ebc8-4db2-8211-df0e6e15cefd-insights-runtime-extractor-tls podName:b67622ee-ebc8-4db2-8211-df0e6e15cefd nodeName:}" failed. No retries permitted until 2026-09-11 08:47:13.952425918 +0000 UTC m=+54.227001605 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b67622ee-ebc8-4db2-8211-df0e6e15cefd-insights-runtime-extractor-tls") pod "insights-runtime-extractor-srrd7" (UID: "b67622ee-ebc8-4db2-8211-df0e6e15cefd") : secret "insights-runtime-extractor-tls" not found Sep 11 08:47:12.513312 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:12.513286 2572 scope.go:117] "RemoveContainer" containerID="fe209c91bcf7683166fbd4b4a75c65e5b07a451b291a217ac7d895d6b0768122" Sep 11 08:47:12.513751 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:47:12.513431 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-xjs9g_openshift-console-operator(546705a6-1aaf-49fe-9f7f-3fc9e566437b)\"" pod="openshift-console-operator/console-operator-696884fb9c-xjs9g" podUID="546705a6-1aaf-49fe-9f7f-3fc9e566437b" Sep 11 08:47:12.871540 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:12.871460 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-4vdzw_82687de7-ba90-4516-99cc-43f545df75bd/dns-node-resolver/0.log" Sep 11 08:47:13.869323 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:13.869294 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-67p7z_eb6eba26-53e8-4946-9491-cf154a8286f5/node-ca/0.log" Sep 11 08:47:13.967847 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:13.967820 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b67622ee-ebc8-4db2-8211-df0e6e15cefd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-srrd7\" (UID: \"b67622ee-ebc8-4db2-8211-df0e6e15cefd\") " pod="openshift-insights/insights-runtime-extractor-srrd7" Sep 11 08:47:13.967982 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:47:13.967968 2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 11 08:47:13.968038 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:47:13.968029 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b67622ee-ebc8-4db2-8211-df0e6e15cefd-insights-runtime-extractor-tls podName:b67622ee-ebc8-4db2-8211-df0e6e15cefd nodeName:}" failed. No retries permitted until 2026-09-11 08:47:17.968012615 +0000 UTC m=+58.242588290 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b67622ee-ebc8-4db2-8211-df0e6e15cefd-insights-runtime-extractor-tls") pod "insights-runtime-extractor-srrd7" (UID: "b67622ee-ebc8-4db2-8211-df0e6e15cefd") : secret "insights-runtime-extractor-tls" not found Sep 11 08:47:14.681787 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:14.681760 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-2tlm8_a9917159-f37b-4842-ae12-10c24123dff1/serve-healthcheck-canary/0.log" Sep 11 08:47:15.672134 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:15.672099 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-wdwcs_9222c5e5-755c-460e-8568-db40848b8123/kube-storage-version-migrator-operator/0.log" Sep 11 08:47:16.686893 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:16.686862 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7dae9305-821b-4929-900e-2d7bf09d5654-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-rsjdv\" (UID: \"7dae9305-821b-4929-900e-2d7bf09d5654\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-rsjdv" Sep 11 08:47:16.689529 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:16.689504 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7dae9305-821b-4929-900e-2d7bf09d5654-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-rsjdv\" (UID: \"7dae9305-821b-4929-900e-2d7bf09d5654\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-rsjdv" Sep 11 08:47:16.725374 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:16.725349 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-rsjdv" Sep 11 08:47:16.787820 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:16.787792 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/aa036e63-1ef3-481a-9786-90b2f397ab9a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-blrfj\" (UID: \"aa036e63-1ef3-481a-9786-90b2f397ab9a\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-blrfj" Sep 11 08:47:16.788042 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:47:16.788024 2572 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 11 08:47:16.788121 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:47:16.788113 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aa036e63-1ef3-481a-9786-90b2f397ab9a-cluster-monitoring-operator-tls podName:aa036e63-1ef3-481a-9786-90b2f397ab9a nodeName:}" failed. No retries permitted until 2026-09-11 08:47:32.788091939 +0000 UTC m=+73.062667628 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/aa036e63-1ef3-481a-9786-90b2f397ab9a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-blrfj" (UID: "aa036e63-1ef3-481a-9786-90b2f397ab9a") : secret "cluster-monitoring-operator-tls" not found Sep 11 08:47:16.888692 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:16.888663 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a882da04-e40f-472b-b823-dd7650334c17-service-ca-bundle\") pod \"router-default-57979bfb5b-9fr79\" (UID: \"a882da04-e40f-472b-b823-dd7650334c17\") " pod="openshift-ingress/router-default-57979bfb5b-9fr79" Sep 11 08:47:16.888846 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:16.888781 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a882da04-e40f-472b-b823-dd7650334c17-metrics-certs\") pod \"router-default-57979bfb5b-9fr79\" (UID: \"a882da04-e40f-472b-b823-dd7650334c17\") " pod="openshift-ingress/router-default-57979bfb5b-9fr79" Sep 11 08:47:16.888846 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:47:16.888836 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a882da04-e40f-472b-b823-dd7650334c17-service-ca-bundle podName:a882da04-e40f-472b-b823-dd7650334c17 nodeName:}" failed. No retries permitted until 2026-09-11 08:47:32.888817569 +0000 UTC m=+73.163393259 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/a882da04-e40f-472b-b823-dd7650334c17-service-ca-bundle") pod "router-default-57979bfb5b-9fr79" (UID: "a882da04-e40f-472b-b823-dd7650334c17") : configmap references non-existent config key: service-ca.crt Sep 11 08:47:16.888930 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:47:16.888905 2572 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 11 08:47:16.888973 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:47:16.888962 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a882da04-e40f-472b-b823-dd7650334c17-metrics-certs podName:a882da04-e40f-472b-b823-dd7650334c17 nodeName:}" failed. No retries permitted until 2026-09-11 08:47:32.888947307 +0000 UTC m=+73.163522986 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a882da04-e40f-472b-b823-dd7650334c17-metrics-certs") pod "router-default-57979bfb5b-9fr79" (UID: "a882da04-e40f-472b-b823-dd7650334c17") : secret "router-metrics-certs-default" not found Sep 11 08:47:16.995168 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:16.995145 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-rsjdv"] Sep 11 08:47:17.529108 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:17.529076 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-rsjdv" event={"ID":"7dae9305-821b-4929-900e-2d7bf09d5654","Type":"ContainerStarted","Data":"f0d7a842a8e8920be6d1a4aab15630bbc0b10642d10694e613658a5292e1314e"} Sep 11 08:47:17.997495 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:17.997462 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b67622ee-ebc8-4db2-8211-df0e6e15cefd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-srrd7\" (UID: \"b67622ee-ebc8-4db2-8211-df0e6e15cefd\") " pod="openshift-insights/insights-runtime-extractor-srrd7" Sep 11 08:47:17.997871 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:47:17.997626 2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 11 08:47:17.997871 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:47:17.997694 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b67622ee-ebc8-4db2-8211-df0e6e15cefd-insights-runtime-extractor-tls podName:b67622ee-ebc8-4db2-8211-df0e6e15cefd nodeName:}" failed. No retries permitted until 2026-09-11 08:47:25.997677637 +0000 UTC m=+66.272253311 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b67622ee-ebc8-4db2-8211-df0e6e15cefd-insights-runtime-extractor-tls") pod "insights-runtime-extractor-srrd7" (UID: "b67622ee-ebc8-4db2-8211-df0e6e15cefd") : secret "insights-runtime-extractor-tls" not found Sep 11 08:47:19.538603 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:19.538529 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-rsjdv" event={"ID":"7dae9305-821b-4929-900e-2d7bf09d5654","Type":"ContainerStarted","Data":"4cefc3ac18fc902e46f39d984f43d28c465871f675347e6944a7aa8fb57ca4df"} Sep 11 08:47:19.538955 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:19.538604 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-rsjdv" event={"ID":"7dae9305-821b-4929-900e-2d7bf09d5654","Type":"ContainerStarted","Data":"40b60e1a630612307a92a3c47c1bcc0960d33459d9bb16e3fd062379c0f9b60e"} Sep 11 08:47:19.553717 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:19.553664 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-rsjdv" podStartSLOduration=17.341385996 podStartE2EDuration="19.553650675s" totalImagesPullingTime="2.212264679s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 08:47:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 08:47:19.553629744 +0000 UTC m=+59.828205451" watchObservedRunningTime="2026-09-11 08:47:19.553650675 +0000 UTC m=+59.828226371" Sep 11 08:47:21.402737 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:21.402709 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lhkkf" Sep 11 08:47:25.247689 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:25.247658 2572 scope.go:117] "RemoveContainer" containerID="fe209c91bcf7683166fbd4b4a75c65e5b07a451b291a217ac7d895d6b0768122" Sep 11 08:47:25.558375 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:25.558300 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-xjs9g_546705a6-1aaf-49fe-9f7f-3fc9e566437b/console-operator/2.log" Sep 11 08:47:25.558724 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:25.558709 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-xjs9g_546705a6-1aaf-49fe-9f7f-3fc9e566437b/console-operator/1.log" Sep 11 08:47:25.558770 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:25.558742 2572 generic.go:356] "Generic (PLEG): container finished" podID="546705a6-1aaf-49fe-9f7f-3fc9e566437b" containerID="dfaf6ecd2d2db5f62d6749e8a82dfc0faaebba93e281008c9bf4f7d3d013a8ef" exitCode=255 Sep 11 08:47:25.558849 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:25.558786 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-xjs9g" event={"ID":"546705a6-1aaf-49fe-9f7f-3fc9e566437b","Type":"ContainerDied","Data":"dfaf6ecd2d2db5f62d6749e8a82dfc0faaebba93e281008c9bf4f7d3d013a8ef"} Sep 11 08:47:25.558849 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:25.558840 2572 scope.go:117] "RemoveContainer" containerID="fe209c91bcf7683166fbd4b4a75c65e5b07a451b291a217ac7d895d6b0768122" Sep 11 08:47:25.559261 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:25.559210 2572 scope.go:117] "RemoveContainer" containerID="dfaf6ecd2d2db5f62d6749e8a82dfc0faaebba93e281008c9bf4f7d3d013a8ef" Sep 11 08:47:25.559424 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:47:25.559407 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-696884fb9c-xjs9g_openshift-console-operator(546705a6-1aaf-49fe-9f7f-3fc9e566437b)\"" pod="openshift-console-operator/console-operator-696884fb9c-xjs9g" podUID="546705a6-1aaf-49fe-9f7f-3fc9e566437b" Sep 11 08:47:25.967932 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:25.967902 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e78a69db-9d98-4923-b082-ccb30b77cc9e-metrics-certs\") pod \"network-metrics-daemon-ntkhj\" (UID: \"e78a69db-9d98-4923-b082-ccb30b77cc9e\") " pod="openshift-multus/network-metrics-daemon-ntkhj" Sep 11 08:47:25.968156 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:25.967937 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8ea2d42c-f484-483d-8714-603b8f05cc54-metrics-tls\") pod \"dns-default-d67t9\" (UID: \"8ea2d42c-f484-483d-8714-603b8f05cc54\") " pod="openshift-dns/dns-default-d67t9" Sep 11 08:47:25.968156 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:25.967984 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/64b9c16d-d0c5-423f-98e3-67d3fcff6128-registry-tls\") pod \"image-registry-6fdf7ff479-xlfkd\" (UID: \"64b9c16d-d0c5-423f-98e3-67d3fcff6128\") " pod="openshift-image-registry/image-registry-6fdf7ff479-xlfkd" Sep 11 08:47:25.970296 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:25.970272 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 11 08:47:25.970445 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:25.970428 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8ea2d42c-f484-483d-8714-603b8f05cc54-metrics-tls\") pod \"dns-default-d67t9\" (UID: \"8ea2d42c-f484-483d-8714-603b8f05cc54\") " pod="openshift-dns/dns-default-d67t9" Sep 11 08:47:25.970699 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:25.970683 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/64b9c16d-d0c5-423f-98e3-67d3fcff6128-registry-tls\") pod \"image-registry-6fdf7ff479-xlfkd\" (UID: \"64b9c16d-d0c5-423f-98e3-67d3fcff6128\") " pod="openshift-image-registry/image-registry-6fdf7ff479-xlfkd" Sep 11 08:47:25.980973 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:25.980953 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e78a69db-9d98-4923-b082-ccb30b77cc9e-metrics-certs\") pod \"network-metrics-daemon-ntkhj\" (UID: \"e78a69db-9d98-4923-b082-ccb30b77cc9e\") " pod="openshift-multus/network-metrics-daemon-ntkhj" Sep 11 08:47:26.068450 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:26.068400 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b67622ee-ebc8-4db2-8211-df0e6e15cefd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-srrd7\" (UID: \"b67622ee-ebc8-4db2-8211-df0e6e15cefd\") " pod="openshift-insights/insights-runtime-extractor-srrd7" Sep 11 08:47:26.068450 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:26.068452 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwc96\" (UniqueName: \"kubernetes.io/projected/12ac768c-2c6a-43e7-9bd3-44679781ebd7-kube-api-access-zwc96\") pod \"network-check-target-82xp9\" (UID: \"12ac768c-2c6a-43e7-9bd3-44679781ebd7\") " pod="openshift-network-diagnostics/network-check-target-82xp9" Sep 11 08:47:26.068653 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:26.068483 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a370ef3a-98cb-4251-86b9-cb3155c587a5-original-pull-secret\") pod \"global-pull-secret-syncer-v7fx2\" (UID: \"a370ef3a-98cb-4251-86b9-cb3155c587a5\") " pod="kube-system/global-pull-secret-syncer-v7fx2" Sep 11 08:47:26.070646 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:26.070622 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 11 08:47:26.070860 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:26.070840 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b67622ee-ebc8-4db2-8211-df0e6e15cefd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-srrd7\" (UID: \"b67622ee-ebc8-4db2-8211-df0e6e15cefd\") " pod="openshift-insights/insights-runtime-extractor-srrd7" Sep 11 08:47:26.071056 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:26.071041 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwc96\" (UniqueName: \"kubernetes.io/projected/12ac768c-2c6a-43e7-9bd3-44679781ebd7-kube-api-access-zwc96\") pod \"network-check-target-82xp9\" (UID: \"12ac768c-2c6a-43e7-9bd3-44679781ebd7\") " pod="openshift-network-diagnostics/network-check-target-82xp9" Sep 11 08:47:26.081263 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:26.081219 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a370ef3a-98cb-4251-86b9-cb3155c587a5-original-pull-secret\") pod \"global-pull-secret-syncer-v7fx2\" (UID: \"a370ef3a-98cb-4251-86b9-cb3155c587a5\") " pod="kube-system/global-pull-secret-syncer-v7fx2" Sep 11 08:47:26.161047 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:26.161023 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xm9k4\"" Sep 11 08:47:26.164938 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:26.164915 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-wbz7z\"" Sep 11 08:47:26.168306 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:26.168290 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v7fx2" Sep 11 08:47:26.169362 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:26.169347 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ntkhj" Sep 11 08:47:26.173768 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:26.173746 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-82xp9" Sep 11 08:47:26.182217 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:26.182194 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-l57fw\"" Sep 11 08:47:26.190630 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:26.190605 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6fdf7ff479-xlfkd" Sep 11 08:47:26.192343 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:26.192327 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-7fqkq\"" Sep 11 08:47:26.200945 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:26.200913 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-d67t9" Sep 11 08:47:26.218361 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:26.218277 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-xbkxl\"" Sep 11 08:47:26.226567 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:26.226529 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-srrd7" Sep 11 08:47:26.453559 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:26.453530 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-82xp9"] Sep 11 08:47:26.456301 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:47:26.456276 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12ac768c_2c6a_43e7_9bd3_44679781ebd7.slice/crio-bd86b4969bce6fa6487542e16748b5b0ad0af5ed7dc15c152d6f9a25fab5f1da WatchSource:0}: Error finding container bd86b4969bce6fa6487542e16748b5b0ad0af5ed7dc15c152d6f9a25fab5f1da: Status 404 returned error can't find the container with id bd86b4969bce6fa6487542e16748b5b0ad0af5ed7dc15c152d6f9a25fab5f1da Sep 11 08:47:26.473295 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:26.473160 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-v7fx2"] Sep 11 08:47:26.476248 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:47:26.476104 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda370ef3a_98cb_4251_86b9_cb3155c587a5.slice/crio-8b2695a86a8ac8ff287bc7b6475cb5cb55dfb5723daa30b3a7f3d772653dd802 WatchSource:0}: Error finding container 8b2695a86a8ac8ff287bc7b6475cb5cb55dfb5723daa30b3a7f3d772653dd802: Status 404 returned error can't find the container with id 8b2695a86a8ac8ff287bc7b6475cb5cb55dfb5723daa30b3a7f3d772653dd802 Sep 11 08:47:26.477621 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:26.477598 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-ntkhj"] Sep 11 08:47:26.480581 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:47:26.480554 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode78a69db_9d98_4923_b082_ccb30b77cc9e.slice/crio-f8158c64b0d62dcf2013afe044f4063ffeab6932dab51d5013d8f5c06c58f8b3 WatchSource:0}: Error finding container f8158c64b0d62dcf2013afe044f4063ffeab6932dab51d5013d8f5c06c58f8b3: Status 404 returned error can't find the container with id f8158c64b0d62dcf2013afe044f4063ffeab6932dab51d5013d8f5c06c58f8b3 Sep 11 08:47:26.489839 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:26.489465 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6fdf7ff479-xlfkd"] Sep 11 08:47:26.491585 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:47:26.491563 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64b9c16d_d0c5_423f_98e3_67d3fcff6128.slice/crio-9563f7d329a69b8f7a025b6678a6cf4dd4967905e29f29fdee0fecb66f915f24 WatchSource:0}: Error finding container 9563f7d329a69b8f7a025b6678a6cf4dd4967905e29f29fdee0fecb66f915f24: Status 404 returned error can't find the container with id 9563f7d329a69b8f7a025b6678a6cf4dd4967905e29f29fdee0fecb66f915f24 Sep 11 08:47:26.540693 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:26.540667 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-d67t9"] Sep 11 08:47:26.542780 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:47:26.542755 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ea2d42c_f484_483d_8714_603b8f05cc54.slice/crio-62ce29c0db6a0c71b39ed59edc2d6e38c27b99f8eea9cdaedf3c2c572c36a24d WatchSource:0}: Error finding container 62ce29c0db6a0c71b39ed59edc2d6e38c27b99f8eea9cdaedf3c2c572c36a24d: Status 404 returned error can't find the container with id 62ce29c0db6a0c71b39ed59edc2d6e38c27b99f8eea9cdaedf3c2c572c36a24d Sep 11 08:47:26.563396 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:26.563375 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-srrd7"] Sep 11 08:47:26.564336 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:26.564257 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-82xp9" event={"ID":"12ac768c-2c6a-43e7-9bd3-44679781ebd7","Type":"ContainerStarted","Data":"21ce53ff4cc9520217c32fd07cb35bc87757e68ca0decbeb9e978a94544a61fc"} Sep 11 08:47:26.564336 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:26.564305 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-82xp9" event={"ID":"12ac768c-2c6a-43e7-9bd3-44679781ebd7","Type":"ContainerStarted","Data":"bd86b4969bce6fa6487542e16748b5b0ad0af5ed7dc15c152d6f9a25fab5f1da"} Sep 11 08:47:26.564455 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:26.564431 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-82xp9" Sep 11 08:47:26.565830 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:26.565806 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ntkhj" event={"ID":"e78a69db-9d98-4923-b082-ccb30b77cc9e","Type":"ContainerStarted","Data":"f8158c64b0d62dcf2013afe044f4063ffeab6932dab51d5013d8f5c06c58f8b3"} Sep 11 08:47:26.566454 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:47:26.566426 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb67622ee_ebc8_4db2_8211_df0e6e15cefd.slice/crio-bde7a8019c249ac4a194caf3d48f333b90f2a07028595ace66c598f928652ea9 WatchSource:0}: Error finding container bde7a8019c249ac4a194caf3d48f333b90f2a07028595ace66c598f928652ea9: Status 404 returned error can't find the container with id bde7a8019c249ac4a194caf3d48f333b90f2a07028595ace66c598f928652ea9 Sep 11 08:47:26.568205 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:26.568174 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6fdf7ff479-xlfkd" event={"ID":"64b9c16d-d0c5-423f-98e3-67d3fcff6128","Type":"ContainerStarted","Data":"ee578656e3630e62ad3357dae7ed819cad48dad856f75cc08bee497e79a0a1cf"} Sep 11 08:47:26.568337 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:26.568260 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6fdf7ff479-xlfkd" event={"ID":"64b9c16d-d0c5-423f-98e3-67d3fcff6128","Type":"ContainerStarted","Data":"9563f7d329a69b8f7a025b6678a6cf4dd4967905e29f29fdee0fecb66f915f24"} Sep 11 08:47:26.568537 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:26.568518 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6fdf7ff479-xlfkd" Sep 11 08:47:26.570935 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:26.570910 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-v7fx2" event={"ID":"a370ef3a-98cb-4251-86b9-cb3155c587a5","Type":"ContainerStarted","Data":"8b2695a86a8ac8ff287bc7b6475cb5cb55dfb5723daa30b3a7f3d772653dd802"} Sep 11 08:47:26.572604 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:26.572583 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-xjs9g_546705a6-1aaf-49fe-9f7f-3fc9e566437b/console-operator/2.log" Sep 11 08:47:26.573798 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:26.573749 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-d67t9" event={"ID":"8ea2d42c-f484-483d-8714-603b8f05cc54","Type":"ContainerStarted","Data":"62ce29c0db6a0c71b39ed59edc2d6e38c27b99f8eea9cdaedf3c2c572c36a24d"} Sep 11 08:47:26.577780 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:26.577737 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-82xp9" podStartSLOduration=66.577724001 podStartE2EDuration="1m6.577724001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 08:46:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 08:47:26.577073875 +0000 UTC m=+66.851649571" watchObservedRunningTime="2026-09-11 08:47:26.577724001 +0000 UTC m=+66.852299702" Sep 11 08:47:26.596451 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:26.596399 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6fdf7ff479-xlfkd" podStartSLOduration=66.5963849 podStartE2EDuration="1m6.5963849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 08:46:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 08:47:26.595654104 +0000 UTC m=+66.870229802" watchObservedRunningTime="2026-09-11 08:47:26.5963849 +0000 UTC m=+66.870960590" Sep 11 08:47:27.580720 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:27.580597 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-srrd7" event={"ID":"b67622ee-ebc8-4db2-8211-df0e6e15cefd","Type":"ContainerStarted","Data":"6fc188152b10581a46a17200dce426f3e3c14014ebb894f7ea043dbfab535701"} Sep 11 08:47:27.580720 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:27.580674 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-srrd7" event={"ID":"b67622ee-ebc8-4db2-8211-df0e6e15cefd","Type":"ContainerStarted","Data":"bde7a8019c249ac4a194caf3d48f333b90f2a07028595ace66c598f928652ea9"} Sep 11 08:47:28.586915 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:28.586876 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-d67t9" event={"ID":"8ea2d42c-f484-483d-8714-603b8f05cc54","Type":"ContainerStarted","Data":"75607928e5c626a21432e6d0fda48fee6a724ba274ff86933e345f1001daaefd"} Sep 11 08:47:28.587349 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:28.586981 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-d67t9" event={"ID":"8ea2d42c-f484-483d-8714-603b8f05cc54","Type":"ContainerStarted","Data":"41a6c92a4297c81ba3a9d231f2167ac64af048ed9f047dd66fe97b0db597d31b"} Sep 11 08:47:28.587349 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:28.587130 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-d67t9" Sep 11 08:47:28.589363 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:28.589327 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ntkhj" event={"ID":"e78a69db-9d98-4923-b082-ccb30b77cc9e","Type":"ContainerStarted","Data":"ef978511d0a5fbf278b56cb49aa49b5434e50453aba057b576b187d0ade6af88"} Sep 11 08:47:28.589487 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:28.589381 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ntkhj" event={"ID":"e78a69db-9d98-4923-b082-ccb30b77cc9e","Type":"ContainerStarted","Data":"41228d8fc436e8ad7064a2895aaf23b28d2c646f4ba3085a519de432f8587edf"} Sep 11 08:47:28.591640 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:28.591615 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-srrd7" event={"ID":"b67622ee-ebc8-4db2-8211-df0e6e15cefd","Type":"ContainerStarted","Data":"75c9de0202d3d94c899485feb5c397475d984600d05d99030b26fac2f6b80c73"} Sep 11 08:47:28.603661 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:28.603615 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-d67t9" podStartSLOduration=32.835126413 podStartE2EDuration="34.603601743s" totalImagesPullingTime="1.76847533s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 08:46:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 08:47:28.601642849 +0000 UTC m=+68.876218571" watchObservedRunningTime="2026-09-11 08:47:28.603601743 +0000 UTC m=+68.878177438" Sep 11 08:47:28.616117 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:28.616076 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-ntkhj" podStartSLOduration=67.38941823 podStartE2EDuration="1m8.616061245s" totalImagesPullingTime="1.226643015s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 08:46:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 08:47:28.615883225 +0000 UTC m=+68.890458922" watchObservedRunningTime="2026-09-11 08:47:28.616061245 +0000 UTC m=+68.890636952" Sep 11 08:47:31.372468 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:31.372361 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-xjs9g" Sep 11 08:47:31.372468 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:31.372444 2572 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-xjs9g" Sep 11 08:47:31.372975 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:31.372843 2572 scope.go:117] "RemoveContainer" containerID="dfaf6ecd2d2db5f62d6749e8a82dfc0faaebba93e281008c9bf4f7d3d013a8ef" Sep 11 08:47:31.373064 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:47:31.373042 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-696884fb9c-xjs9g_openshift-console-operator(546705a6-1aaf-49fe-9f7f-3fc9e566437b)\"" pod="openshift-console-operator/console-operator-696884fb9c-xjs9g" podUID="546705a6-1aaf-49fe-9f7f-3fc9e566437b" Sep 11 08:47:31.606497 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:31.606461 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-srrd7" event={"ID":"b67622ee-ebc8-4db2-8211-df0e6e15cefd","Type":"ContainerStarted","Data":"9d0ae672c97a7f55e4a24b4b227974b4e0c46b58e78398ac7b54bcd5a0bfae94"} Sep 11 08:47:31.607954 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:31.607928 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-v7fx2" event={"ID":"a370ef3a-98cb-4251-86b9-cb3155c587a5","Type":"ContainerStarted","Data":"6d3ff7fb3307bc37b11c0fade92e4e2412bce5ea6d61f4333c0882b800e72972"} Sep 11 08:47:31.626709 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:31.626632 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-srrd7" podStartSLOduration=17.815585009 podStartE2EDuration="21.626618529s" totalImagesPullingTime="3.81103352s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 08:47:10 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 08:47:31.625462576 +0000 UTC m=+71.900038272" watchObservedRunningTime="2026-09-11 08:47:31.626618529 +0000 UTC m=+71.901194225" Sep 11 08:47:31.642657 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:31.642610 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-v7fx2" podStartSLOduration=67.036628202 podStartE2EDuration="1m11.642599485s" totalImagesPullingTime="4.605971283s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 08:46:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 08:47:31.642079906 +0000 UTC m=+71.916655601" watchObservedRunningTime="2026-09-11 08:47:31.642599485 +0000 UTC m=+71.917175181" Sep 11 08:47:32.835447 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:32.835409 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/aa036e63-1ef3-481a-9786-90b2f397ab9a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-blrfj\" (UID: \"aa036e63-1ef3-481a-9786-90b2f397ab9a\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-blrfj" Sep 11 08:47:32.837919 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:32.837897 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/aa036e63-1ef3-481a-9786-90b2f397ab9a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-blrfj\" (UID: \"aa036e63-1ef3-481a-9786-90b2f397ab9a\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-blrfj" Sep 11 08:47:32.936033 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:32.936000 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a882da04-e40f-472b-b823-dd7650334c17-service-ca-bundle\") pod \"router-default-57979bfb5b-9fr79\" (UID: \"a882da04-e40f-472b-b823-dd7650334c17\") " pod="openshift-ingress/router-default-57979bfb5b-9fr79" Sep 11 08:47:32.936199 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:32.936074 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a882da04-e40f-472b-b823-dd7650334c17-metrics-certs\") pod \"router-default-57979bfb5b-9fr79\" (UID: \"a882da04-e40f-472b-b823-dd7650334c17\") " pod="openshift-ingress/router-default-57979bfb5b-9fr79" Sep 11 08:47:32.936667 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:32.936642 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a882da04-e40f-472b-b823-dd7650334c17-service-ca-bundle\") pod \"router-default-57979bfb5b-9fr79\" (UID: \"a882da04-e40f-472b-b823-dd7650334c17\") " pod="openshift-ingress/router-default-57979bfb5b-9fr79" Sep 11 08:47:32.938523 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:32.938503 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a882da04-e40f-472b-b823-dd7650334c17-metrics-certs\") pod \"router-default-57979bfb5b-9fr79\" (UID: \"a882da04-e40f-472b-b823-dd7650334c17\") " pod="openshift-ingress/router-default-57979bfb5b-9fr79" Sep 11 08:47:33.077885 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:33.077854 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-xwrd9\"" Sep 11 08:47:33.086318 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:33.086263 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-blrfj" Sep 11 08:47:33.099522 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:33.099502 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-dwngz\"" Sep 11 08:47:33.108410 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:33.108387 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-57979bfb5b-9fr79" Sep 11 08:47:33.389471 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:33.389448 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-blrfj"] Sep 11 08:47:33.391947 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:47:33.391922 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa036e63_1ef3_481a_9786_90b2f397ab9a.slice/crio-71144c5f3ef582bb81f5cac900f305b08972b28ed142b3ab5b9309eb1dfc0be1 WatchSource:0}: Error finding container 71144c5f3ef582bb81f5cac900f305b08972b28ed142b3ab5b9309eb1dfc0be1: Status 404 returned error can't find the container with id 71144c5f3ef582bb81f5cac900f305b08972b28ed142b3ab5b9309eb1dfc0be1 Sep 11 08:47:33.417853 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:33.417751 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-57979bfb5b-9fr79"] Sep 11 08:47:33.420114 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:47:33.420096 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda882da04_e40f_472b_b823_dd7650334c17.slice/crio-9f9c74134c405ea8e8e916ef34042621692acd4e769e19863be9e9f5633857df WatchSource:0}: Error finding container 9f9c74134c405ea8e8e916ef34042621692acd4e769e19863be9e9f5633857df: Status 404 returned error can't find the container with id 9f9c74134c405ea8e8e916ef34042621692acd4e769e19863be9e9f5633857df Sep 11 08:47:33.616293 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:33.616260 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-blrfj" event={"ID":"aa036e63-1ef3-481a-9786-90b2f397ab9a","Type":"ContainerStarted","Data":"71144c5f3ef582bb81f5cac900f305b08972b28ed142b3ab5b9309eb1dfc0be1"} Sep 11 08:47:33.617751 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:33.617727 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-57979bfb5b-9fr79" event={"ID":"a882da04-e40f-472b-b823-dd7650334c17","Type":"ContainerStarted","Data":"33aed0b4122118c03e1a801f95cd7e761e3d1e10b3c2acaf4c1c3cd707f8f94f"} Sep 11 08:47:33.617825 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:33.617772 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-57979bfb5b-9fr79" event={"ID":"a882da04-e40f-472b-b823-dd7650334c17","Type":"ContainerStarted","Data":"9f9c74134c405ea8e8e916ef34042621692acd4e769e19863be9e9f5633857df"} Sep 11 08:47:33.636984 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:33.636931 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-57979bfb5b-9fr79" podStartSLOduration=33.636915987 podStartE2EDuration="33.636915987s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 08:47:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 08:47:33.635504665 +0000 UTC m=+73.910080361" watchObservedRunningTime="2026-09-11 08:47:33.636915987 +0000 UTC m=+73.911491680" Sep 11 08:47:34.109195 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:34.109154 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-57979bfb5b-9fr79" Sep 11 08:47:34.111909 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:34.111883 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-57979bfb5b-9fr79" Sep 11 08:47:34.621721 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:34.621677 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-57979bfb5b-9fr79" Sep 11 08:47:34.623074 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:34.623045 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-57979bfb5b-9fr79" Sep 11 08:47:35.630185 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:35.630155 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-blrfj" event={"ID":"aa036e63-1ef3-481a-9786-90b2f397ab9a","Type":"ContainerStarted","Data":"a6d9db565254a22e0bcedea8a00b123a15142c572df3f0f22937d1e0e89db14b"} Sep 11 08:47:36.656762 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:36.656704 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-blrfj" podStartSLOduration=34.534301382 podStartE2EDuration="36.656688737s" totalImagesPullingTime="2.122387355s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 08:47:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 08:47:36.656417678 +0000 UTC m=+76.930993386" watchObservedRunningTime="2026-09-11 08:47:36.656688737 +0000 UTC m=+76.931264432" Sep 11 08:47:38.598908 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:38.598881 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-d67t9" Sep 11 08:47:44.507990 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:44.507882 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-pfjml"] Sep 11 08:47:44.512005 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:44.511989 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-pfjml" Sep 11 08:47:44.514377 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:44.514351 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Sep 11 08:47:44.514492 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:44.514354 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Sep 11 08:47:44.514492 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:44.514354 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 11 08:47:44.514492 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:44.514360 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-lvjw7\"" Sep 11 08:47:44.522323 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:44.522301 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-pfjml"] Sep 11 08:47:44.527701 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:44.527671 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-lpvfd"] Sep 11 08:47:44.531382 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:44.531348 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-lpvfd" Sep 11 08:47:44.533852 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:44.533831 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 11 08:47:44.533978 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:44.533862 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-2w95z\"" Sep 11 08:47:44.533978 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:44.533898 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 11 08:47:44.534249 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:44.534216 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 11 08:47:44.627417 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:44.627385 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9fb78fb0-b471-4ef3-a08c-3ac9b7db0cc8-metrics-client-ca\") pod \"node-exporter-lpvfd\" (UID: \"9fb78fb0-b471-4ef3-a08c-3ac9b7db0cc8\") " pod="openshift-monitoring/node-exporter-lpvfd" Sep 11 08:47:44.627558 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:44.627430 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9fb78fb0-b471-4ef3-a08c-3ac9b7db0cc8-node-exporter-wtmp\") pod \"node-exporter-lpvfd\" (UID: \"9fb78fb0-b471-4ef3-a08c-3ac9b7db0cc8\") " pod="openshift-monitoring/node-exporter-lpvfd" Sep 11 08:47:44.627558 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:44.627488 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/201f5aa0-d681-4869-90c6-d9b1b804f99c-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-pfjml\" (UID: \"201f5aa0-d681-4869-90c6-d9b1b804f99c\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-pfjml" Sep 11 08:47:44.627558 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:44.627505 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9fb78fb0-b471-4ef3-a08c-3ac9b7db0cc8-sys\") pod \"node-exporter-lpvfd\" (UID: \"9fb78fb0-b471-4ef3-a08c-3ac9b7db0cc8\") " pod="openshift-monitoring/node-exporter-lpvfd" Sep 11 08:47:44.627558 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:44.627554 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppp4m\" (UniqueName: \"kubernetes.io/projected/201f5aa0-d681-4869-90c6-d9b1b804f99c-kube-api-access-ppp4m\") pod \"openshift-state-metrics-67dcf8c979-pfjml\" (UID: \"201f5aa0-d681-4869-90c6-d9b1b804f99c\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-pfjml" Sep 11 08:47:44.627726 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:44.627572 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9fb78fb0-b471-4ef3-a08c-3ac9b7db0cc8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lpvfd\" (UID: \"9fb78fb0-b471-4ef3-a08c-3ac9b7db0cc8\") " pod="openshift-monitoring/node-exporter-lpvfd" Sep 11 08:47:44.627726 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:44.627588 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jj4f\" (UniqueName: \"kubernetes.io/projected/9fb78fb0-b471-4ef3-a08c-3ac9b7db0cc8-kube-api-access-8jj4f\") pod \"node-exporter-lpvfd\" (UID: \"9fb78fb0-b471-4ef3-a08c-3ac9b7db0cc8\") " pod="openshift-monitoring/node-exporter-lpvfd" Sep 11 08:47:44.627726 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:44.627644 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9fb78fb0-b471-4ef3-a08c-3ac9b7db0cc8-node-exporter-textfile\") pod \"node-exporter-lpvfd\" (UID: \"9fb78fb0-b471-4ef3-a08c-3ac9b7db0cc8\") " pod="openshift-monitoring/node-exporter-lpvfd" Sep 11 08:47:44.627726 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:44.627673 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9fb78fb0-b471-4ef3-a08c-3ac9b7db0cc8-node-exporter-accelerators-collector-config\") pod \"node-exporter-lpvfd\" (UID: \"9fb78fb0-b471-4ef3-a08c-3ac9b7db0cc8\") " pod="openshift-monitoring/node-exporter-lpvfd" Sep 11 08:47:44.627726 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:44.627699 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/201f5aa0-d681-4869-90c6-d9b1b804f99c-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-pfjml\" (UID: \"201f5aa0-d681-4869-90c6-d9b1b804f99c\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-pfjml" Sep 11 08:47:44.627726 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:44.627715 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9fb78fb0-b471-4ef3-a08c-3ac9b7db0cc8-node-exporter-tls\") pod \"node-exporter-lpvfd\" (UID: \"9fb78fb0-b471-4ef3-a08c-3ac9b7db0cc8\") " pod="openshift-monitoring/node-exporter-lpvfd" Sep 11 08:47:44.627980 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:44.627821 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/201f5aa0-d681-4869-90c6-d9b1b804f99c-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-pfjml\" (UID: \"201f5aa0-d681-4869-90c6-d9b1b804f99c\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-pfjml" Sep 11 08:47:44.627980 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:44.627862 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9fb78fb0-b471-4ef3-a08c-3ac9b7db0cc8-root\") pod \"node-exporter-lpvfd\" (UID: \"9fb78fb0-b471-4ef3-a08c-3ac9b7db0cc8\") " pod="openshift-monitoring/node-exporter-lpvfd" Sep 11 08:47:44.728315 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:44.728289 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppp4m\" (UniqueName: \"kubernetes.io/projected/201f5aa0-d681-4869-90c6-d9b1b804f99c-kube-api-access-ppp4m\") pod \"openshift-state-metrics-67dcf8c979-pfjml\" (UID: \"201f5aa0-d681-4869-90c6-d9b1b804f99c\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-pfjml" Sep 11 08:47:44.728469 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:44.728320 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9fb78fb0-b471-4ef3-a08c-3ac9b7db0cc8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lpvfd\" (UID: \"9fb78fb0-b471-4ef3-a08c-3ac9b7db0cc8\") " pod="openshift-monitoring/node-exporter-lpvfd" Sep 11 08:47:44.728469 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:44.728336 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jj4f\" (UniqueName: \"kubernetes.io/projected/9fb78fb0-b471-4ef3-a08c-3ac9b7db0cc8-kube-api-access-8jj4f\") pod \"node-exporter-lpvfd\" (UID: \"9fb78fb0-b471-4ef3-a08c-3ac9b7db0cc8\") " pod="openshift-monitoring/node-exporter-lpvfd" Sep 11 08:47:44.728469 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:44.728358 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9fb78fb0-b471-4ef3-a08c-3ac9b7db0cc8-node-exporter-textfile\") pod \"node-exporter-lpvfd\" (UID: \"9fb78fb0-b471-4ef3-a08c-3ac9b7db0cc8\") " pod="openshift-monitoring/node-exporter-lpvfd" Sep 11 08:47:44.728469 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:44.728376 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9fb78fb0-b471-4ef3-a08c-3ac9b7db0cc8-node-exporter-accelerators-collector-config\") pod \"node-exporter-lpvfd\" (UID: \"9fb78fb0-b471-4ef3-a08c-3ac9b7db0cc8\") " pod="openshift-monitoring/node-exporter-lpvfd" Sep 11 08:47:44.728469 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:44.728404 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/201f5aa0-d681-4869-90c6-d9b1b804f99c-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-pfjml\" (UID: \"201f5aa0-d681-4869-90c6-d9b1b804f99c\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-pfjml" Sep 11 08:47:44.728469 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:44.728427 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9fb78fb0-b471-4ef3-a08c-3ac9b7db0cc8-node-exporter-tls\") pod \"node-exporter-lpvfd\" (UID: \"9fb78fb0-b471-4ef3-a08c-3ac9b7db0cc8\") " pod="openshift-monitoring/node-exporter-lpvfd" Sep 11 08:47:44.728469 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:44.728460 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/201f5aa0-d681-4869-90c6-d9b1b804f99c-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-pfjml\" (UID: \"201f5aa0-d681-4869-90c6-d9b1b804f99c\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-pfjml" Sep 11 08:47:44.728817 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:44.728491 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9fb78fb0-b471-4ef3-a08c-3ac9b7db0cc8-root\") pod \"node-exporter-lpvfd\" (UID: \"9fb78fb0-b471-4ef3-a08c-3ac9b7db0cc8\") " pod="openshift-monitoring/node-exporter-lpvfd" Sep 11 08:47:44.728817 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:44.728516 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9fb78fb0-b471-4ef3-a08c-3ac9b7db0cc8-metrics-client-ca\") pod \"node-exporter-lpvfd\" (UID: \"9fb78fb0-b471-4ef3-a08c-3ac9b7db0cc8\") " pod="openshift-monitoring/node-exporter-lpvfd" Sep 11 08:47:44.728817 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:44.728543 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9fb78fb0-b471-4ef3-a08c-3ac9b7db0cc8-node-exporter-wtmp\") pod \"node-exporter-lpvfd\" (UID: \"9fb78fb0-b471-4ef3-a08c-3ac9b7db0cc8\") " pod="openshift-monitoring/node-exporter-lpvfd" Sep 11 08:47:44.728817 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:44.728585 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/201f5aa0-d681-4869-90c6-d9b1b804f99c-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-pfjml\" (UID: \"201f5aa0-d681-4869-90c6-d9b1b804f99c\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-pfjml" Sep 11 08:47:44.728817 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:44.728615 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9fb78fb0-b471-4ef3-a08c-3ac9b7db0cc8-sys\") pod \"node-exporter-lpvfd\" (UID: \"9fb78fb0-b471-4ef3-a08c-3ac9b7db0cc8\") " pod="openshift-monitoring/node-exporter-lpvfd" Sep 11 08:47:44.728817 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:47:44.728634 2572 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Sep 11 08:47:44.728817 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:44.728687 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9fb78fb0-b471-4ef3-a08c-3ac9b7db0cc8-sys\") pod \"node-exporter-lpvfd\" (UID: \"9fb78fb0-b471-4ef3-a08c-3ac9b7db0cc8\") " pod="openshift-monitoring/node-exporter-lpvfd" Sep 11 08:47:44.728817 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:47:44.728701 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9fb78fb0-b471-4ef3-a08c-3ac9b7db0cc8-node-exporter-tls podName:9fb78fb0-b471-4ef3-a08c-3ac9b7db0cc8 nodeName:}" failed. No retries permitted until 2026-09-11 08:47:45.228680672 +0000 UTC m=+85.503256360 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/9fb78fb0-b471-4ef3-a08c-3ac9b7db0cc8-node-exporter-tls") pod "node-exporter-lpvfd" (UID: "9fb78fb0-b471-4ef3-a08c-3ac9b7db0cc8") : secret "node-exporter-tls" not found Sep 11 08:47:44.728817 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:44.728746 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9fb78fb0-b471-4ef3-a08c-3ac9b7db0cc8-root\") pod \"node-exporter-lpvfd\" (UID: \"9fb78fb0-b471-4ef3-a08c-3ac9b7db0cc8\") " pod="openshift-monitoring/node-exporter-lpvfd" Sep 11 08:47:44.728817 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:44.728762 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9fb78fb0-b471-4ef3-a08c-3ac9b7db0cc8-node-exporter-textfile\") pod \"node-exporter-lpvfd\" (UID: \"9fb78fb0-b471-4ef3-a08c-3ac9b7db0cc8\") " pod="openshift-monitoring/node-exporter-lpvfd" Sep 11 08:47:44.728817 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:44.728805 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9fb78fb0-b471-4ef3-a08c-3ac9b7db0cc8-node-exporter-wtmp\") pod \"node-exporter-lpvfd\" (UID: \"9fb78fb0-b471-4ef3-a08c-3ac9b7db0cc8\") " pod="openshift-monitoring/node-exporter-lpvfd" Sep 11 08:47:44.729284 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:44.729219 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9fb78fb0-b471-4ef3-a08c-3ac9b7db0cc8-metrics-client-ca\") pod \"node-exporter-lpvfd\" (UID: \"9fb78fb0-b471-4ef3-a08c-3ac9b7db0cc8\") " pod="openshift-monitoring/node-exporter-lpvfd" Sep 11 08:47:44.729544 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:44.729521 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/201f5aa0-d681-4869-90c6-d9b1b804f99c-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-pfjml\" (UID: \"201f5aa0-d681-4869-90c6-d9b1b804f99c\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-pfjml" Sep 11 08:47:44.729749 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:44.729723 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9fb78fb0-b471-4ef3-a08c-3ac9b7db0cc8-node-exporter-accelerators-collector-config\") pod \"node-exporter-lpvfd\" (UID: \"9fb78fb0-b471-4ef3-a08c-3ac9b7db0cc8\") " pod="openshift-monitoring/node-exporter-lpvfd" Sep 11 08:47:44.731121 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:44.731089 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9fb78fb0-b471-4ef3-a08c-3ac9b7db0cc8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lpvfd\" (UID: \"9fb78fb0-b471-4ef3-a08c-3ac9b7db0cc8\") " pod="openshift-monitoring/node-exporter-lpvfd" Sep 11 08:47:44.731266 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:44.731248 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/201f5aa0-d681-4869-90c6-d9b1b804f99c-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-pfjml\" (UID: \"201f5aa0-d681-4869-90c6-d9b1b804f99c\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-pfjml" Sep 11 08:47:44.731332 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:44.731288 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/201f5aa0-d681-4869-90c6-d9b1b804f99c-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-pfjml\" (UID: \"201f5aa0-d681-4869-90c6-d9b1b804f99c\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-pfjml" Sep 11 08:47:44.742392 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:44.742367 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jj4f\" (UniqueName: \"kubernetes.io/projected/9fb78fb0-b471-4ef3-a08c-3ac9b7db0cc8-kube-api-access-8jj4f\") pod \"node-exporter-lpvfd\" (UID: \"9fb78fb0-b471-4ef3-a08c-3ac9b7db0cc8\") " pod="openshift-monitoring/node-exporter-lpvfd" Sep 11 08:47:44.742562 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:44.742547 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppp4m\" (UniqueName: \"kubernetes.io/projected/201f5aa0-d681-4869-90c6-d9b1b804f99c-kube-api-access-ppp4m\") pod \"openshift-state-metrics-67dcf8c979-pfjml\" (UID: \"201f5aa0-d681-4869-90c6-d9b1b804f99c\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-pfjml" Sep 11 08:47:44.820977 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:44.820927 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-pfjml" Sep 11 08:47:45.094834 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:45.094767 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-pfjml"] Sep 11 08:47:45.097703 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:47:45.097673 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod201f5aa0_d681_4869_90c6_d9b1b804f99c.slice/crio-f627d3c956a030600db24d2957b6d9fd92531de2cc8e70c8c0f45c697d8f09d1 WatchSource:0}: Error finding container f627d3c956a030600db24d2957b6d9fd92531de2cc8e70c8c0f45c697d8f09d1: Status 404 returned error can't find the container with id f627d3c956a030600db24d2957b6d9fd92531de2cc8e70c8c0f45c697d8f09d1 Sep 11 08:47:45.234211 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:45.234175 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9fb78fb0-b471-4ef3-a08c-3ac9b7db0cc8-node-exporter-tls\") pod \"node-exporter-lpvfd\" (UID: \"9fb78fb0-b471-4ef3-a08c-3ac9b7db0cc8\") " pod="openshift-monitoring/node-exporter-lpvfd" Sep 11 08:47:45.234390 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:47:45.234305 2572 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Sep 11 08:47:45.234390 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:47:45.234361 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9fb78fb0-b471-4ef3-a08c-3ac9b7db0cc8-node-exporter-tls podName:9fb78fb0-b471-4ef3-a08c-3ac9b7db0cc8 nodeName:}" failed. No retries permitted until 2026-09-11 08:47:46.234345231 +0000 UTC m=+86.508920904 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/9fb78fb0-b471-4ef3-a08c-3ac9b7db0cc8-node-exporter-tls") pod "node-exporter-lpvfd" (UID: "9fb78fb0-b471-4ef3-a08c-3ac9b7db0cc8") : secret "node-exporter-tls" not found Sep 11 08:47:45.667082 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:45.667048 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-pfjml" event={"ID":"201f5aa0-d681-4869-90c6-d9b1b804f99c","Type":"ContainerStarted","Data":"5699b2ba0a967107f1f886a9ec3edef00b7bb01d344de6d220c06faf6c0950a1"} Sep 11 08:47:45.667465 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:45.667107 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-pfjml" event={"ID":"201f5aa0-d681-4869-90c6-d9b1b804f99c","Type":"ContainerStarted","Data":"d5a55b32c46feaea37767f16a66b45e010425ad2b34cd63c5ba4ef491564cae3"} Sep 11 08:47:45.667465 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:45.667126 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-pfjml" event={"ID":"201f5aa0-d681-4869-90c6-d9b1b804f99c","Type":"ContainerStarted","Data":"f627d3c956a030600db24d2957b6d9fd92531de2cc8e70c8c0f45c697d8f09d1"} Sep 11 08:47:46.243255 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:46.243211 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9fb78fb0-b471-4ef3-a08c-3ac9b7db0cc8-node-exporter-tls\") pod \"node-exporter-lpvfd\" (UID: \"9fb78fb0-b471-4ef3-a08c-3ac9b7db0cc8\") " pod="openshift-monitoring/node-exporter-lpvfd" Sep 11 08:47:46.245403 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:46.245384 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9fb78fb0-b471-4ef3-a08c-3ac9b7db0cc8-node-exporter-tls\") pod \"node-exporter-lpvfd\" (UID: \"9fb78fb0-b471-4ef3-a08c-3ac9b7db0cc8\") " pod="openshift-monitoring/node-exporter-lpvfd" Sep 11 08:47:46.341238 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:46.341198 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-lpvfd" Sep 11 08:47:46.352353 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:47:46.352330 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fb78fb0_b471_4ef3_a08c_3ac9b7db0cc8.slice/crio-2d25dad958b789be9a83add76d6285b95b2fb0861eb6da239be08087f97ab780 WatchSource:0}: Error finding container 2d25dad958b789be9a83add76d6285b95b2fb0861eb6da239be08087f97ab780: Status 404 returned error can't find the container with id 2d25dad958b789be9a83add76d6285b95b2fb0861eb6da239be08087f97ab780 Sep 11 08:47:46.672941 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:46.672855 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-pfjml" event={"ID":"201f5aa0-d681-4869-90c6-d9b1b804f99c","Type":"ContainerStarted","Data":"a73e75921ac42532609253a03d927a1f93dd89b7738749f2cb35bd38eca6f17a"} Sep 11 08:47:46.674291 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:46.674266 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lpvfd" event={"ID":"9fb78fb0-b471-4ef3-a08c-3ac9b7db0cc8","Type":"ContainerStarted","Data":"2d25dad958b789be9a83add76d6285b95b2fb0861eb6da239be08087f97ab780"} Sep 11 08:47:46.692454 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:46.692411 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-pfjml" podStartSLOduration=1.704188915 podStartE2EDuration="2.692395848s" totalImagesPullingTime="988.206933ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 08:47:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 08:47:46.691218943 +0000 UTC m=+86.965794639" watchObservedRunningTime="2026-09-11 08:47:46.692395848 +0000 UTC m=+86.966971543" Sep 11 08:47:47.247170 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:47.247144 2572 scope.go:117] "RemoveContainer" containerID="dfaf6ecd2d2db5f62d6749e8a82dfc0faaebba93e281008c9bf4f7d3d013a8ef" Sep 11 08:47:47.586656 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:47.586576 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6fdf7ff479-xlfkd" Sep 11 08:47:47.682259 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:47.682087 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-xjs9g_546705a6-1aaf-49fe-9f7f-3fc9e566437b/console-operator/2.log" Sep 11 08:47:47.682259 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:47.682216 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-xjs9g" event={"ID":"546705a6-1aaf-49fe-9f7f-3fc9e566437b","Type":"ContainerStarted","Data":"305b9a819ca7da2c9c5bae1d2cc420b415431a64604edf46165dc6dc93c6182f"} Sep 11 08:47:47.699701 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:47.699601 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-696884fb9c-xjs9g" podStartSLOduration=41.308386463 podStartE2EDuration="47.699580848s" totalImagesPullingTime="6.391194385s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 08:47:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 08:47:47.698600217 +0000 UTC m=+87.973175910" watchObservedRunningTime="2026-09-11 08:47:47.699580848 +0000 UTC m=+87.974156548" Sep 11 08:47:48.063982 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:48.063950 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-76f99584f4-8vnhx"] Sep 11 08:47:48.066973 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:48.066954 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-8vnhx" Sep 11 08:47:48.069287 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:48.069266 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 11 08:47:48.069366 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:48.069273 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-cplp7\"" Sep 11 08:47:48.069366 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:48.069275 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 11 08:47:48.076998 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:48.076979 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-8vnhx"] Sep 11 08:47:48.262359 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:48.262333 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dw828\" (UniqueName: \"kubernetes.io/projected/47bbded3-73b6-4258-b620-c9c8f6e918cf-kube-api-access-dw828\") pod \"downloads-76f99584f4-8vnhx\" (UID: \"47bbded3-73b6-4258-b620-c9c8f6e918cf\") " pod="openshift-console/downloads-76f99584f4-8vnhx" Sep 11 08:47:48.362691 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:48.362627 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dw828\" (UniqueName: \"kubernetes.io/projected/47bbded3-73b6-4258-b620-c9c8f6e918cf-kube-api-access-dw828\") pod \"downloads-76f99584f4-8vnhx\" (UID: \"47bbded3-73b6-4258-b620-c9c8f6e918cf\") " pod="openshift-console/downloads-76f99584f4-8vnhx" Sep 11 08:47:48.370199 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:48.370170 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dw828\" (UniqueName: \"kubernetes.io/projected/47bbded3-73b6-4258-b620-c9c8f6e918cf-kube-api-access-dw828\") pod \"downloads-76f99584f4-8vnhx\" (UID: \"47bbded3-73b6-4258-b620-c9c8f6e918cf\") " pod="openshift-console/downloads-76f99584f4-8vnhx" Sep 11 08:47:48.375941 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:48.375920 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-8vnhx" Sep 11 08:47:48.651992 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:48.651867 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-8vnhx"] Sep 11 08:47:48.655645 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:47:48.655611 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47bbded3_73b6_4258_b620_c9c8f6e918cf.slice/crio-c52fc184f11e5ce514a1d0ba06849ad7d46fc8455508cc3be24fb9e0831fed83 WatchSource:0}: Error finding container c52fc184f11e5ce514a1d0ba06849ad7d46fc8455508cc3be24fb9e0831fed83: Status 404 returned error can't find the container with id c52fc184f11e5ce514a1d0ba06849ad7d46fc8455508cc3be24fb9e0831fed83 Sep 11 08:47:48.687282 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:48.687254 2572 generic.go:356] "Generic (PLEG): container finished" podID="9fb78fb0-b471-4ef3-a08c-3ac9b7db0cc8" containerID="dab56adffaf5550aa2304ce6a77263dd311bb6827fa4582f55867741b31a5a9a" exitCode=0 Sep 11 08:47:48.687619 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:48.687340 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lpvfd" event={"ID":"9fb78fb0-b471-4ef3-a08c-3ac9b7db0cc8","Type":"ContainerDied","Data":"dab56adffaf5550aa2304ce6a77263dd311bb6827fa4582f55867741b31a5a9a"} Sep 11 08:47:48.688520 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:48.688497 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-8vnhx" event={"ID":"47bbded3-73b6-4258-b620-c9c8f6e918cf","Type":"ContainerStarted","Data":"c52fc184f11e5ce514a1d0ba06849ad7d46fc8455508cc3be24fb9e0831fed83"} Sep 11 08:47:48.927914 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:48.927840 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-fdbb5cd6-nqdhs"] Sep 11 08:47:48.931076 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:48.931058 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-fdbb5cd6-nqdhs" Sep 11 08:47:48.934165 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:48.934135 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Sep 11 08:47:48.934406 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:48.934390 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Sep 11 08:47:48.934476 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:48.934411 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Sep 11 08:47:48.934530 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:48.934490 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-aspdetaeaicib\"" Sep 11 08:47:48.934648 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:48.934634 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-kcqmk\"" Sep 11 08:47:48.936196 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:48.936170 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Sep 11 08:47:48.945287 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:48.945267 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-fdbb5cd6-nqdhs"] Sep 11 08:47:49.068480 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:49.068446 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3e3367f-aee2-495a-aec0-16c4363d5403-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-fdbb5cd6-nqdhs\" (UID: \"f3e3367f-aee2-495a-aec0-16c4363d5403\") " pod="openshift-monitoring/metrics-server-fdbb5cd6-nqdhs" Sep 11 08:47:49.068661 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:49.068567 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/f3e3367f-aee2-495a-aec0-16c4363d5403-metrics-server-audit-profiles\") pod \"metrics-server-fdbb5cd6-nqdhs\" (UID: \"f3e3367f-aee2-495a-aec0-16c4363d5403\") " pod="openshift-monitoring/metrics-server-fdbb5cd6-nqdhs" Sep 11 08:47:49.068661 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:49.068611 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/f3e3367f-aee2-495a-aec0-16c4363d5403-audit-log\") pod \"metrics-server-fdbb5cd6-nqdhs\" (UID: \"f3e3367f-aee2-495a-aec0-16c4363d5403\") " pod="openshift-monitoring/metrics-server-fdbb5cd6-nqdhs" Sep 11 08:47:49.068661 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:49.068647 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/f3e3367f-aee2-495a-aec0-16c4363d5403-secret-metrics-server-tls\") pod \"metrics-server-fdbb5cd6-nqdhs\" (UID: \"f3e3367f-aee2-495a-aec0-16c4363d5403\") " pod="openshift-monitoring/metrics-server-fdbb5cd6-nqdhs" Sep 11 08:47:49.068788 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:49.068702 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/f3e3367f-aee2-495a-aec0-16c4363d5403-secret-metrics-server-client-certs\") pod \"metrics-server-fdbb5cd6-nqdhs\" (UID: \"f3e3367f-aee2-495a-aec0-16c4363d5403\") " pod="openshift-monitoring/metrics-server-fdbb5cd6-nqdhs" Sep 11 08:47:49.068788 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:49.068732 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/f3e3367f-aee2-495a-aec0-16c4363d5403-client-ca-bundle\") pod \"metrics-server-fdbb5cd6-nqdhs\" (UID: \"f3e3367f-aee2-495a-aec0-16c4363d5403\") " pod="openshift-monitoring/metrics-server-fdbb5cd6-nqdhs" Sep 11 08:47:49.068788 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:49.068758 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8nwj\" (UniqueName: \"kubernetes.io/projected/f3e3367f-aee2-495a-aec0-16c4363d5403-kube-api-access-s8nwj\") pod \"metrics-server-fdbb5cd6-nqdhs\" (UID: \"f3e3367f-aee2-495a-aec0-16c4363d5403\") " pod="openshift-monitoring/metrics-server-fdbb5cd6-nqdhs" Sep 11 08:47:49.169745 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:49.169697 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3e3367f-aee2-495a-aec0-16c4363d5403-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-fdbb5cd6-nqdhs\" (UID: \"f3e3367f-aee2-495a-aec0-16c4363d5403\") " pod="openshift-monitoring/metrics-server-fdbb5cd6-nqdhs" Sep 11 08:47:49.169937 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:49.169751 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/f3e3367f-aee2-495a-aec0-16c4363d5403-metrics-server-audit-profiles\") pod \"metrics-server-fdbb5cd6-nqdhs\" (UID: \"f3e3367f-aee2-495a-aec0-16c4363d5403\") " pod="openshift-monitoring/metrics-server-fdbb5cd6-nqdhs" Sep 11 08:47:49.169937 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:49.169789 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/f3e3367f-aee2-495a-aec0-16c4363d5403-audit-log\") pod \"metrics-server-fdbb5cd6-nqdhs\" (UID: \"f3e3367f-aee2-495a-aec0-16c4363d5403\") " pod="openshift-monitoring/metrics-server-fdbb5cd6-nqdhs" Sep 11 08:47:49.169937 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:49.169823 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/f3e3367f-aee2-495a-aec0-16c4363d5403-secret-metrics-server-tls\") pod \"metrics-server-fdbb5cd6-nqdhs\" (UID: \"f3e3367f-aee2-495a-aec0-16c4363d5403\") " pod="openshift-monitoring/metrics-server-fdbb5cd6-nqdhs" Sep 11 08:47:49.169937 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:49.169870 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/f3e3367f-aee2-495a-aec0-16c4363d5403-secret-metrics-server-client-certs\") pod \"metrics-server-fdbb5cd6-nqdhs\" (UID: \"f3e3367f-aee2-495a-aec0-16c4363d5403\") " pod="openshift-monitoring/metrics-server-fdbb5cd6-nqdhs" Sep 11 08:47:49.169937 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:49.169891 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/f3e3367f-aee2-495a-aec0-16c4363d5403-client-ca-bundle\") pod \"metrics-server-fdbb5cd6-nqdhs\" (UID: \"f3e3367f-aee2-495a-aec0-16c4363d5403\") " pod="openshift-monitoring/metrics-server-fdbb5cd6-nqdhs" Sep 11 08:47:49.169937 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:49.169916 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8nwj\" (UniqueName: \"kubernetes.io/projected/f3e3367f-aee2-495a-aec0-16c4363d5403-kube-api-access-s8nwj\") pod \"metrics-server-fdbb5cd6-nqdhs\" (UID: \"f3e3367f-aee2-495a-aec0-16c4363d5403\") " pod="openshift-monitoring/metrics-server-fdbb5cd6-nqdhs" Sep 11 08:47:49.170458 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:49.170408 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/f3e3367f-aee2-495a-aec0-16c4363d5403-audit-log\") pod \"metrics-server-fdbb5cd6-nqdhs\" (UID: \"f3e3367f-aee2-495a-aec0-16c4363d5403\") " pod="openshift-monitoring/metrics-server-fdbb5cd6-nqdhs" Sep 11 08:47:49.171006 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:49.170960 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/f3e3367f-aee2-495a-aec0-16c4363d5403-metrics-server-audit-profiles\") pod \"metrics-server-fdbb5cd6-nqdhs\" (UID: \"f3e3367f-aee2-495a-aec0-16c4363d5403\") " pod="openshift-monitoring/metrics-server-fdbb5cd6-nqdhs" Sep 11 08:47:49.171095 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:49.170995 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3e3367f-aee2-495a-aec0-16c4363d5403-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-fdbb5cd6-nqdhs\" (UID: \"f3e3367f-aee2-495a-aec0-16c4363d5403\") " pod="openshift-monitoring/metrics-server-fdbb5cd6-nqdhs" Sep 11 08:47:49.172633 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:49.172608 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/f3e3367f-aee2-495a-aec0-16c4363d5403-secret-metrics-server-tls\") pod \"metrics-server-fdbb5cd6-nqdhs\" (UID: \"f3e3367f-aee2-495a-aec0-16c4363d5403\") " pod="openshift-monitoring/metrics-server-fdbb5cd6-nqdhs" Sep 11 08:47:49.172721 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:49.172705 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/f3e3367f-aee2-495a-aec0-16c4363d5403-secret-metrics-server-client-certs\") pod \"metrics-server-fdbb5cd6-nqdhs\" (UID: \"f3e3367f-aee2-495a-aec0-16c4363d5403\") " pod="openshift-monitoring/metrics-server-fdbb5cd6-nqdhs" Sep 11 08:47:49.172759 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:49.172710 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/f3e3367f-aee2-495a-aec0-16c4363d5403-client-ca-bundle\") pod \"metrics-server-fdbb5cd6-nqdhs\" (UID: \"f3e3367f-aee2-495a-aec0-16c4363d5403\") " pod="openshift-monitoring/metrics-server-fdbb5cd6-nqdhs" Sep 11 08:47:49.177053 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:49.177035 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8nwj\" (UniqueName: \"kubernetes.io/projected/f3e3367f-aee2-495a-aec0-16c4363d5403-kube-api-access-s8nwj\") pod \"metrics-server-fdbb5cd6-nqdhs\" (UID: \"f3e3367f-aee2-495a-aec0-16c4363d5403\") " pod="openshift-monitoring/metrics-server-fdbb5cd6-nqdhs" Sep 11 08:47:49.240031 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:49.239983 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-fdbb5cd6-nqdhs" Sep 11 08:47:49.533367 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:49.533285 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-fdbb5cd6-nqdhs"] Sep 11 08:47:49.536291 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:47:49.536253 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3e3367f_aee2_495a_aec0_16c4363d5403.slice/crio-9874c5b2211687b3d90a487158e7492871af1e9543092d7d6fc91fff23597cb4 WatchSource:0}: Error finding container 9874c5b2211687b3d90a487158e7492871af1e9543092d7d6fc91fff23597cb4: Status 404 returned error can't find the container with id 9874c5b2211687b3d90a487158e7492871af1e9543092d7d6fc91fff23597cb4 Sep 11 08:47:49.696387 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:49.696352 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lpvfd" event={"ID":"9fb78fb0-b471-4ef3-a08c-3ac9b7db0cc8","Type":"ContainerStarted","Data":"932e0ab7d6872fbb3e7e4eaef4f35ef5ec12095ce7452429df4d8c8545578538"} Sep 11 08:47:49.696893 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:49.696416 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lpvfd" event={"ID":"9fb78fb0-b471-4ef3-a08c-3ac9b7db0cc8","Type":"ContainerStarted","Data":"af5c5241fa048f49bced1e57eaccc8bb35d98cd1af8f7e1a453a1c96a84d1ff4"} Sep 11 08:47:49.697856 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:49.697829 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-fdbb5cd6-nqdhs" event={"ID":"f3e3367f-aee2-495a-aec0-16c4363d5403","Type":"ContainerStarted","Data":"9874c5b2211687b3d90a487158e7492871af1e9543092d7d6fc91fff23597cb4"} Sep 11 08:47:49.716735 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:49.716683 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-lpvfd" podStartSLOduration=3.908114318 podStartE2EDuration="5.716664792s" totalImagesPullingTime="1.74051468s" totalInitContainerRuntime="68.035794ms" isStatefulPod=true podCreationTimestamp="2026-09-11 08:47:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 08:47:49.716484357 +0000 UTC m=+89.991060062" watchObservedRunningTime="2026-09-11 08:47:49.716664792 +0000 UTC m=+89.991240488" Sep 11 08:47:51.709457 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:51.709420 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-fdbb5cd6-nqdhs" event={"ID":"f3e3367f-aee2-495a-aec0-16c4363d5403","Type":"ContainerStarted","Data":"1d26525109ba8ed6e0d473753c4bb8cbad0be9929b331f04ac94e4e73265a308"} Sep 11 08:47:51.727089 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:51.727030 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-fdbb5cd6-nqdhs" podStartSLOduration=2.201798287 podStartE2EDuration="3.727007576s" totalImagesPullingTime="1.525209289s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 08:47:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 08:47:51.725550631 +0000 UTC m=+92.000126330" watchObservedRunningTime="2026-09-11 08:47:51.727007576 +0000 UTC m=+92.001583273" Sep 11 08:47:56.688614 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:56.688576 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6b776758d8-mvh7l"] Sep 11 08:47:56.694436 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:56.694409 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b776758d8-mvh7l" Sep 11 08:47:56.697471 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:56.697445 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 11 08:47:56.697585 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:56.697445 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 11 08:47:56.697585 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:56.697490 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 11 08:47:56.697719 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:56.697651 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 11 08:47:56.697888 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:56.697865 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 11 08:47:56.697888 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:56.697875 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-4zmj8\"" Sep 11 08:47:56.705559 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:56.705539 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6b776758d8-mvh7l"] Sep 11 08:47:56.739426 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:56.739393 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1cc84abb-cf26-4eb7-b01e-cdb1c8e84fa0-service-ca\") pod \"console-6b776758d8-mvh7l\" (UID: \"1cc84abb-cf26-4eb7-b01e-cdb1c8e84fa0\") " pod="openshift-console/console-6b776758d8-mvh7l" Sep 11 08:47:56.739591 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:56.739469 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1cc84abb-cf26-4eb7-b01e-cdb1c8e84fa0-console-config\") pod \"console-6b776758d8-mvh7l\" (UID: \"1cc84abb-cf26-4eb7-b01e-cdb1c8e84fa0\") " pod="openshift-console/console-6b776758d8-mvh7l" Sep 11 08:47:56.739591 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:56.739496 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1cc84abb-cf26-4eb7-b01e-cdb1c8e84fa0-oauth-serving-cert\") pod \"console-6b776758d8-mvh7l\" (UID: \"1cc84abb-cf26-4eb7-b01e-cdb1c8e84fa0\") " pod="openshift-console/console-6b776758d8-mvh7l" Sep 11 08:47:56.739711 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:56.739597 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxsgx\" (UniqueName: \"kubernetes.io/projected/1cc84abb-cf26-4eb7-b01e-cdb1c8e84fa0-kube-api-access-zxsgx\") pod \"console-6b776758d8-mvh7l\" (UID: \"1cc84abb-cf26-4eb7-b01e-cdb1c8e84fa0\") " pod="openshift-console/console-6b776758d8-mvh7l" Sep 11 08:47:56.739711 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:56.739651 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1cc84abb-cf26-4eb7-b01e-cdb1c8e84fa0-console-serving-cert\") pod \"console-6b776758d8-mvh7l\" (UID: \"1cc84abb-cf26-4eb7-b01e-cdb1c8e84fa0\") " pod="openshift-console/console-6b776758d8-mvh7l" Sep 11 08:47:56.739807 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:56.739717 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1cc84abb-cf26-4eb7-b01e-cdb1c8e84fa0-console-oauth-config\") pod \"console-6b776758d8-mvh7l\" (UID: \"1cc84abb-cf26-4eb7-b01e-cdb1c8e84fa0\") " pod="openshift-console/console-6b776758d8-mvh7l" Sep 11 08:47:56.840678 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:56.840638 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxsgx\" (UniqueName: \"kubernetes.io/projected/1cc84abb-cf26-4eb7-b01e-cdb1c8e84fa0-kube-api-access-zxsgx\") pod \"console-6b776758d8-mvh7l\" (UID: \"1cc84abb-cf26-4eb7-b01e-cdb1c8e84fa0\") " pod="openshift-console/console-6b776758d8-mvh7l" Sep 11 08:47:56.840877 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:56.840694 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1cc84abb-cf26-4eb7-b01e-cdb1c8e84fa0-console-serving-cert\") pod \"console-6b776758d8-mvh7l\" (UID: \"1cc84abb-cf26-4eb7-b01e-cdb1c8e84fa0\") " pod="openshift-console/console-6b776758d8-mvh7l" Sep 11 08:47:56.840877 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:56.840726 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1cc84abb-cf26-4eb7-b01e-cdb1c8e84fa0-console-oauth-config\") pod \"console-6b776758d8-mvh7l\" (UID: \"1cc84abb-cf26-4eb7-b01e-cdb1c8e84fa0\") " pod="openshift-console/console-6b776758d8-mvh7l" Sep 11 08:47:56.840877 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:56.840752 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1cc84abb-cf26-4eb7-b01e-cdb1c8e84fa0-service-ca\") pod \"console-6b776758d8-mvh7l\" (UID: \"1cc84abb-cf26-4eb7-b01e-cdb1c8e84fa0\") " pod="openshift-console/console-6b776758d8-mvh7l" Sep 11 08:47:56.840877 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:56.840796 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1cc84abb-cf26-4eb7-b01e-cdb1c8e84fa0-console-config\") pod \"console-6b776758d8-mvh7l\" (UID: \"1cc84abb-cf26-4eb7-b01e-cdb1c8e84fa0\") " pod="openshift-console/console-6b776758d8-mvh7l" Sep 11 08:47:56.840877 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:56.840821 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1cc84abb-cf26-4eb7-b01e-cdb1c8e84fa0-oauth-serving-cert\") pod \"console-6b776758d8-mvh7l\" (UID: \"1cc84abb-cf26-4eb7-b01e-cdb1c8e84fa0\") " pod="openshift-console/console-6b776758d8-mvh7l" Sep 11 08:47:56.841636 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:56.841613 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1cc84abb-cf26-4eb7-b01e-cdb1c8e84fa0-oauth-serving-cert\") pod \"console-6b776758d8-mvh7l\" (UID: \"1cc84abb-cf26-4eb7-b01e-cdb1c8e84fa0\") " pod="openshift-console/console-6b776758d8-mvh7l" Sep 11 08:47:56.841797 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:56.841775 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1cc84abb-cf26-4eb7-b01e-cdb1c8e84fa0-console-config\") pod \"console-6b776758d8-mvh7l\" (UID: \"1cc84abb-cf26-4eb7-b01e-cdb1c8e84fa0\") " pod="openshift-console/console-6b776758d8-mvh7l" Sep 11 08:47:56.841848 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:56.841786 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1cc84abb-cf26-4eb7-b01e-cdb1c8e84fa0-service-ca\") pod \"console-6b776758d8-mvh7l\" (UID: \"1cc84abb-cf26-4eb7-b01e-cdb1c8e84fa0\") " pod="openshift-console/console-6b776758d8-mvh7l" Sep 11 08:47:56.843976 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:56.843951 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1cc84abb-cf26-4eb7-b01e-cdb1c8e84fa0-console-oauth-config\") pod \"console-6b776758d8-mvh7l\" (UID: \"1cc84abb-cf26-4eb7-b01e-cdb1c8e84fa0\") " pod="openshift-console/console-6b776758d8-mvh7l" Sep 11 08:47:56.844067 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:56.843982 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1cc84abb-cf26-4eb7-b01e-cdb1c8e84fa0-console-serving-cert\") pod \"console-6b776758d8-mvh7l\" (UID: \"1cc84abb-cf26-4eb7-b01e-cdb1c8e84fa0\") " pod="openshift-console/console-6b776758d8-mvh7l" Sep 11 08:47:56.848767 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:56.848743 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxsgx\" (UniqueName: \"kubernetes.io/projected/1cc84abb-cf26-4eb7-b01e-cdb1c8e84fa0-kube-api-access-zxsgx\") pod \"console-6b776758d8-mvh7l\" (UID: \"1cc84abb-cf26-4eb7-b01e-cdb1c8e84fa0\") " pod="openshift-console/console-6b776758d8-mvh7l" Sep 11 08:47:57.005877 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:57.005796 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b776758d8-mvh7l" Sep 11 08:47:57.286321 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:57.286297 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6b776758d8-mvh7l"] Sep 11 08:47:57.288863 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:47:57.288843 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cc84abb_cf26_4eb7_b01e_cdb1c8e84fa0.slice/crio-57e5ec89a70ea1f879eff52adb9c053d837daaa4b26e65278a68b36b06e07878 WatchSource:0}: Error finding container 57e5ec89a70ea1f879eff52adb9c053d837daaa4b26e65278a68b36b06e07878: Status 404 returned error can't find the container with id 57e5ec89a70ea1f879eff52adb9c053d837daaa4b26e65278a68b36b06e07878 Sep 11 08:47:57.583572 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:57.583484 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-82xp9" Sep 11 08:47:57.600403 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:57.600370 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6fdf7ff479-xlfkd"] Sep 11 08:47:57.683173 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:57.683137 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-xjs9g" Sep 11 08:47:57.687675 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:57.687650 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-696884fb9c-xjs9g" Sep 11 08:47:57.737281 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:47:57.737215 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b776758d8-mvh7l" event={"ID":"1cc84abb-cf26-4eb7-b01e-cdb1c8e84fa0","Type":"ContainerStarted","Data":"57e5ec89a70ea1f879eff52adb9c053d837daaa4b26e65278a68b36b06e07878"} Sep 11 08:48:02.970283 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:48:02.970246 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-64bf664cf5-w4xqt"] Sep 11 08:48:02.995285 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:48:02.995251 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-64bf664cf5-w4xqt"] Sep 11 08:48:02.995470 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:48:02.995373 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64bf664cf5-w4xqt" Sep 11 08:48:03.002984 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:48:03.002959 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Sep 11 08:48:03.097536 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:48:03.097495 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/223b0e88-dc9c-464f-ae62-eab365dc7abd-console-serving-cert\") pod \"console-64bf664cf5-w4xqt\" (UID: \"223b0e88-dc9c-464f-ae62-eab365dc7abd\") " pod="openshift-console/console-64bf664cf5-w4xqt" Sep 11 08:48:03.097730 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:48:03.097555 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/223b0e88-dc9c-464f-ae62-eab365dc7abd-console-config\") pod \"console-64bf664cf5-w4xqt\" (UID: \"223b0e88-dc9c-464f-ae62-eab365dc7abd\") " pod="openshift-console/console-64bf664cf5-w4xqt" Sep 11 08:48:03.097730 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:48:03.097584 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/223b0e88-dc9c-464f-ae62-eab365dc7abd-oauth-serving-cert\") pod \"console-64bf664cf5-w4xqt\" (UID: \"223b0e88-dc9c-464f-ae62-eab365dc7abd\") " pod="openshift-console/console-64bf664cf5-w4xqt" Sep 11 08:48:03.097730 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:48:03.097669 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/223b0e88-dc9c-464f-ae62-eab365dc7abd-trusted-ca-bundle\") pod \"console-64bf664cf5-w4xqt\" (UID: \"223b0e88-dc9c-464f-ae62-eab365dc7abd\") " pod="openshift-console/console-64bf664cf5-w4xqt" Sep 11 08:48:03.097730 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:48:03.097711 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/223b0e88-dc9c-464f-ae62-eab365dc7abd-console-oauth-config\") pod \"console-64bf664cf5-w4xqt\" (UID: \"223b0e88-dc9c-464f-ae62-eab365dc7abd\") " pod="openshift-console/console-64bf664cf5-w4xqt" Sep 11 08:48:03.097949 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:48:03.097824 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/223b0e88-dc9c-464f-ae62-eab365dc7abd-service-ca\") pod \"console-64bf664cf5-w4xqt\" (UID: \"223b0e88-dc9c-464f-ae62-eab365dc7abd\") " pod="openshift-console/console-64bf664cf5-w4xqt" Sep 11 08:48:03.097949 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:48:03.097849 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mctsh\" (UniqueName: \"kubernetes.io/projected/223b0e88-dc9c-464f-ae62-eab365dc7abd-kube-api-access-mctsh\") pod \"console-64bf664cf5-w4xqt\" (UID: \"223b0e88-dc9c-464f-ae62-eab365dc7abd\") " pod="openshift-console/console-64bf664cf5-w4xqt" Sep 11 08:48:03.198945 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:48:03.198904 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/223b0e88-dc9c-464f-ae62-eab365dc7abd-service-ca\") pod \"console-64bf664cf5-w4xqt\" (UID: \"223b0e88-dc9c-464f-ae62-eab365dc7abd\") " pod="openshift-console/console-64bf664cf5-w4xqt" Sep 11 08:48:03.199130 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:48:03.198954 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mctsh\" (UniqueName: \"kubernetes.io/projected/223b0e88-dc9c-464f-ae62-eab365dc7abd-kube-api-access-mctsh\") pod \"console-64bf664cf5-w4xqt\" (UID: \"223b0e88-dc9c-464f-ae62-eab365dc7abd\") " pod="openshift-console/console-64bf664cf5-w4xqt" Sep 11 08:48:03.199130 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:48:03.198986 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/223b0e88-dc9c-464f-ae62-eab365dc7abd-console-serving-cert\") pod \"console-64bf664cf5-w4xqt\" (UID: \"223b0e88-dc9c-464f-ae62-eab365dc7abd\") " pod="openshift-console/console-64bf664cf5-w4xqt" Sep 11 08:48:03.199130 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:48:03.199020 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/223b0e88-dc9c-464f-ae62-eab365dc7abd-console-config\") pod \"console-64bf664cf5-w4xqt\" (UID: \"223b0e88-dc9c-464f-ae62-eab365dc7abd\") " pod="openshift-console/console-64bf664cf5-w4xqt" Sep 11 08:48:03.199130 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:48:03.199041 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/223b0e88-dc9c-464f-ae62-eab365dc7abd-oauth-serving-cert\") pod \"console-64bf664cf5-w4xqt\" (UID: \"223b0e88-dc9c-464f-ae62-eab365dc7abd\") " pod="openshift-console/console-64bf664cf5-w4xqt" Sep 11 08:48:03.199130 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:48:03.199064 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/223b0e88-dc9c-464f-ae62-eab365dc7abd-trusted-ca-bundle\") pod \"console-64bf664cf5-w4xqt\" (UID: \"223b0e88-dc9c-464f-ae62-eab365dc7abd\") " pod="openshift-console/console-64bf664cf5-w4xqt" Sep 11 08:48:03.199130 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:48:03.199092 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/223b0e88-dc9c-464f-ae62-eab365dc7abd-console-oauth-config\") pod \"console-64bf664cf5-w4xqt\" (UID: \"223b0e88-dc9c-464f-ae62-eab365dc7abd\") " pod="openshift-console/console-64bf664cf5-w4xqt" Sep 11 08:48:03.199869 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:48:03.199840 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/223b0e88-dc9c-464f-ae62-eab365dc7abd-service-ca\") pod \"console-64bf664cf5-w4xqt\" (UID: \"223b0e88-dc9c-464f-ae62-eab365dc7abd\") " pod="openshift-console/console-64bf664cf5-w4xqt" Sep 11 08:48:03.199996 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:48:03.199867 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/223b0e88-dc9c-464f-ae62-eab365dc7abd-console-config\") pod \"console-64bf664cf5-w4xqt\" (UID: \"223b0e88-dc9c-464f-ae62-eab365dc7abd\") " pod="openshift-console/console-64bf664cf5-w4xqt" Sep 11 08:48:03.200203 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:48:03.200181 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/223b0e88-dc9c-464f-ae62-eab365dc7abd-oauth-serving-cert\") pod \"console-64bf664cf5-w4xqt\" (UID: \"223b0e88-dc9c-464f-ae62-eab365dc7abd\") " pod="openshift-console/console-64bf664cf5-w4xqt" Sep 11 08:48:03.200640 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:48:03.200591 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/223b0e88-dc9c-464f-ae62-eab365dc7abd-trusted-ca-bundle\") pod \"console-64bf664cf5-w4xqt\" (UID: \"223b0e88-dc9c-464f-ae62-eab365dc7abd\") " pod="openshift-console/console-64bf664cf5-w4xqt" Sep 11 08:48:03.202144 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:48:03.202121 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/223b0e88-dc9c-464f-ae62-eab365dc7abd-console-oauth-config\") pod \"console-64bf664cf5-w4xqt\" (UID: \"223b0e88-dc9c-464f-ae62-eab365dc7abd\") " pod="openshift-console/console-64bf664cf5-w4xqt" Sep 11 08:48:03.202254 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:48:03.202191 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/223b0e88-dc9c-464f-ae62-eab365dc7abd-console-serving-cert\") pod \"console-64bf664cf5-w4xqt\" (UID: \"223b0e88-dc9c-464f-ae62-eab365dc7abd\") " pod="openshift-console/console-64bf664cf5-w4xqt" Sep 11 08:48:03.207614 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:48:03.207593 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mctsh\" (UniqueName: \"kubernetes.io/projected/223b0e88-dc9c-464f-ae62-eab365dc7abd-kube-api-access-mctsh\") pod \"console-64bf664cf5-w4xqt\" (UID: \"223b0e88-dc9c-464f-ae62-eab365dc7abd\") " pod="openshift-console/console-64bf664cf5-w4xqt" Sep 11 08:48:03.310585 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:48:03.310492 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64bf664cf5-w4xqt" Sep 11 08:48:07.104307 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:48:07.104150 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-64bf664cf5-w4xqt"] Sep 11 08:48:07.106822 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:48:07.106793 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod223b0e88_dc9c_464f_ae62_eab365dc7abd.slice/crio-212e0c43f14dfd52fa73c8dd66808768a9d7d07a0aa9bb7d6c0c2491ceb660f7 WatchSource:0}: Error finding container 212e0c43f14dfd52fa73c8dd66808768a9d7d07a0aa9bb7d6c0c2491ceb660f7: Status 404 returned error can't find the container with id 212e0c43f14dfd52fa73c8dd66808768a9d7d07a0aa9bb7d6c0c2491ceb660f7 Sep 11 08:48:07.793839 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:48:07.793785 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b776758d8-mvh7l" event={"ID":"1cc84abb-cf26-4eb7-b01e-cdb1c8e84fa0","Type":"ContainerStarted","Data":"02af566951639302e6fd7d6e4d0a02f7de26c25104c6418dd7241dd80a20b902"} Sep 11 08:48:07.796357 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:48:07.796326 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-8vnhx" event={"ID":"47bbded3-73b6-4258-b620-c9c8f6e918cf","Type":"ContainerStarted","Data":"36349c1dff0dad19ea8855341af0bdff809da116587d6b7cf54d091bce6bc9f9"} Sep 11 08:48:07.796500 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:48:07.796377 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-76f99584f4-8vnhx" Sep 11 08:48:07.798704 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:48:07.798554 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64bf664cf5-w4xqt" event={"ID":"223b0e88-dc9c-464f-ae62-eab365dc7abd","Type":"ContainerStarted","Data":"a81122664fd28d3688224602fe9b32d67d7860172b8667dae4814efd6ea18b1d"} Sep 11 08:48:07.798704 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:48:07.798591 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64bf664cf5-w4xqt" event={"ID":"223b0e88-dc9c-464f-ae62-eab365dc7abd","Type":"ContainerStarted","Data":"212e0c43f14dfd52fa73c8dd66808768a9d7d07a0aa9bb7d6c0c2491ceb660f7"} Sep 11 08:48:07.810568 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:48:07.810520 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6b776758d8-mvh7l" podStartSLOduration=2.259134531 podStartE2EDuration="11.810502148s" totalImagesPullingTime="9.551367617s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 08:47:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 08:48:07.810200674 +0000 UTC m=+108.084776370" watchObservedRunningTime="2026-09-11 08:48:07.810502148 +0000 UTC m=+108.085077847" Sep 11 08:48:07.810910 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:48:07.810822 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-76f99584f4-8vnhx" Sep 11 08:48:07.831017 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:48:07.830967 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-64bf664cf5-w4xqt" podStartSLOduration=5.830954037 podStartE2EDuration="5.830954037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 08:48:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 08:48:07.82983417 +0000 UTC m=+108.104409945" watchObservedRunningTime="2026-09-11 08:48:07.830954037 +0000 UTC m=+108.105529733" Sep 11 08:48:07.845997 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:48:07.845956 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-76f99584f4-8vnhx" podStartSLOduration=1.63005543 podStartE2EDuration="19.845942269s" totalImagesPullingTime="18.215886839s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 08:47:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 08:48:07.844361343 +0000 UTC m=+108.118937041" watchObservedRunningTime="2026-09-11 08:48:07.845942269 +0000 UTC m=+108.120517965" Sep 11 08:48:09.241042 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:48:09.240995 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-fdbb5cd6-nqdhs" Sep 11 08:48:09.241608 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:48:09.241081 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-fdbb5cd6-nqdhs" Sep 11 08:48:13.311033 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:48:13.310992 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-64bf664cf5-w4xqt" Sep 11 08:48:13.311546 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:48:13.311060 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-64bf664cf5-w4xqt" Sep 11 08:48:13.316733 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:48:13.316705 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-64bf664cf5-w4xqt" Sep 11 08:48:13.829921 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:48:13.829888 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-64bf664cf5-w4xqt" Sep 11 08:48:13.887827 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:48:13.887796 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6b776758d8-mvh7l"] Sep 11 08:48:17.006516 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:48:17.006479 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6b776758d8-mvh7l" Sep 11 08:48:22.624131 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:48:22.624087 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6fdf7ff479-xlfkd" podUID="64b9c16d-d0c5-423f-98e3-67d3fcff6128" containerName="registry" containerID="cri-o://ee578656e3630e62ad3357dae7ed819cad48dad856f75cc08bee497e79a0a1cf" gracePeriod=30 Sep 11 08:48:22.863319 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:48:22.863286 2572 generic.go:356] "Generic (PLEG): container finished" podID="64b9c16d-d0c5-423f-98e3-67d3fcff6128" containerID="ee578656e3630e62ad3357dae7ed819cad48dad856f75cc08bee497e79a0a1cf" exitCode=0 Sep 11 08:48:22.863492 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:48:22.863323 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6fdf7ff479-xlfkd" event={"ID":"64b9c16d-d0c5-423f-98e3-67d3fcff6128","Type":"ContainerDied","Data":"ee578656e3630e62ad3357dae7ed819cad48dad856f75cc08bee497e79a0a1cf"} Sep 11 08:48:22.899059 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:48:22.898998 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6fdf7ff479-xlfkd" Sep 11 08:48:23.081247 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:48:23.081195 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/64b9c16d-d0c5-423f-98e3-67d3fcff6128-bound-sa-token\") pod \"64b9c16d-d0c5-423f-98e3-67d3fcff6128\" (UID: \"64b9c16d-d0c5-423f-98e3-67d3fcff6128\") " Sep 11 08:48:23.081415 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:48:23.081267 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/64b9c16d-d0c5-423f-98e3-67d3fcff6128-ca-trust-extracted\") pod \"64b9c16d-d0c5-423f-98e3-67d3fcff6128\" (UID: \"64b9c16d-d0c5-423f-98e3-67d3fcff6128\") " Sep 11 08:48:23.081415 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:48:23.081295 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/64b9c16d-d0c5-423f-98e3-67d3fcff6128-installation-pull-secrets\") pod \"64b9c16d-d0c5-423f-98e3-67d3fcff6128\" (UID: \"64b9c16d-d0c5-423f-98e3-67d3fcff6128\") " Sep 11 08:48:23.081415 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:48:23.081321 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/64b9c16d-d0c5-423f-98e3-67d3fcff6128-registry-tls\") pod \"64b9c16d-d0c5-423f-98e3-67d3fcff6128\" (UID: \"64b9c16d-d0c5-423f-98e3-67d3fcff6128\") " Sep 11 08:48:23.081415 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:48:23.081339 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/64b9c16d-d0c5-423f-98e3-67d3fcff6128-registry-certificates\") pod \"64b9c16d-d0c5-423f-98e3-67d3fcff6128\" (UID: \"64b9c16d-d0c5-423f-98e3-67d3fcff6128\") " Sep 11 08:48:23.081415 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:48:23.081360 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/64b9c16d-d0c5-423f-98e3-67d3fcff6128-trusted-ca\") pod \"64b9c16d-d0c5-423f-98e3-67d3fcff6128\" (UID: \"64b9c16d-d0c5-423f-98e3-67d3fcff6128\") " Sep 11 08:48:23.081415 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:48:23.081384 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/64b9c16d-d0c5-423f-98e3-67d3fcff6128-image-registry-private-configuration\") pod \"64b9c16d-d0c5-423f-98e3-67d3fcff6128\" (UID: \"64b9c16d-d0c5-423f-98e3-67d3fcff6128\") " Sep 11 08:48:23.081415 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:48:23.081414 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8w4l9\" (UniqueName: \"kubernetes.io/projected/64b9c16d-d0c5-423f-98e3-67d3fcff6128-kube-api-access-8w4l9\") pod \"64b9c16d-d0c5-423f-98e3-67d3fcff6128\" (UID: \"64b9c16d-d0c5-423f-98e3-67d3fcff6128\") " Sep 11 08:48:23.082313 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:48:23.082253 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64b9c16d-d0c5-423f-98e3-67d3fcff6128-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "64b9c16d-d0c5-423f-98e3-67d3fcff6128" (UID: "64b9c16d-d0c5-423f-98e3-67d3fcff6128"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 08:48:23.082444 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:48:23.082368 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64b9c16d-d0c5-423f-98e3-67d3fcff6128-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "64b9c16d-d0c5-423f-98e3-67d3fcff6128" (UID: "64b9c16d-d0c5-423f-98e3-67d3fcff6128"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 08:48:23.084153 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:48:23.084107 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64b9c16d-d0c5-423f-98e3-67d3fcff6128-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "64b9c16d-d0c5-423f-98e3-67d3fcff6128" (UID: "64b9c16d-d0c5-423f-98e3-67d3fcff6128"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 08:48:23.084326 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:48:23.084300 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64b9c16d-d0c5-423f-98e3-67d3fcff6128-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "64b9c16d-d0c5-423f-98e3-67d3fcff6128" (UID: "64b9c16d-d0c5-423f-98e3-67d3fcff6128"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 08:48:23.084462 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:48:23.084433 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64b9c16d-d0c5-423f-98e3-67d3fcff6128-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "64b9c16d-d0c5-423f-98e3-67d3fcff6128" (UID: "64b9c16d-d0c5-423f-98e3-67d3fcff6128"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 08:48:23.084462 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:48:23.084448 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64b9c16d-d0c5-423f-98e3-67d3fcff6128-kube-api-access-8w4l9" (OuterVolumeSpecName: "kube-api-access-8w4l9") pod "64b9c16d-d0c5-423f-98e3-67d3fcff6128" (UID: "64b9c16d-d0c5-423f-98e3-67d3fcff6128"). InnerVolumeSpecName "kube-api-access-8w4l9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 08:48:23.084690 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:48:23.084667 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64b9c16d-d0c5-423f-98e3-67d3fcff6128-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "64b9c16d-d0c5-423f-98e3-67d3fcff6128" (UID: "64b9c16d-d0c5-423f-98e3-67d3fcff6128"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 08:48:23.089929 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:48:23.089907 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64b9c16d-d0c5-423f-98e3-67d3fcff6128-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "64b9c16d-d0c5-423f-98e3-67d3fcff6128" (UID: "64b9c16d-d0c5-423f-98e3-67d3fcff6128"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 08:48:23.182359 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:48:23.182335 2572 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/64b9c16d-d0c5-423f-98e3-67d3fcff6128-registry-tls\") on node \"ip-10-0-128-166.ec2.internal\" DevicePath \"\"" Sep 11 08:48:23.182359 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:48:23.182357 2572 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/64b9c16d-d0c5-423f-98e3-67d3fcff6128-registry-certificates\") on node \"ip-10-0-128-166.ec2.internal\" DevicePath \"\"" Sep 11 08:48:23.182483 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:48:23.182366 2572 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/64b9c16d-d0c5-423f-98e3-67d3fcff6128-trusted-ca\") on node \"ip-10-0-128-166.ec2.internal\" DevicePath \"\"" Sep 11 08:48:23.182483 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:48:23.182376 2572 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/64b9c16d-d0c5-423f-98e3-67d3fcff6128-image-registry-private-configuration\") on node \"ip-10-0-128-166.ec2.internal\" DevicePath \"\"" Sep 11 08:48:23.182483 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:48:23.182384 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8w4l9\" (UniqueName: \"kubernetes.io/projected/64b9c16d-d0c5-423f-98e3-67d3fcff6128-kube-api-access-8w4l9\") on node \"ip-10-0-128-166.ec2.internal\" DevicePath \"\"" Sep 11 08:48:23.182483 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:48:23.182392 2572 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/64b9c16d-d0c5-423f-98e3-67d3fcff6128-bound-sa-token\") on node \"ip-10-0-128-166.ec2.internal\" DevicePath \"\"" Sep 11 08:48:23.182483 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:48:23.182400 2572 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/64b9c16d-d0c5-423f-98e3-67d3fcff6128-ca-trust-extracted\") on node \"ip-10-0-128-166.ec2.internal\" DevicePath \"\"" Sep 11 08:48:23.182483 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:48:23.182407 2572 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/64b9c16d-d0c5-423f-98e3-67d3fcff6128-installation-pull-secrets\") on node \"ip-10-0-128-166.ec2.internal\" DevicePath \"\"" Sep 11 08:48:23.868963 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:48:23.868934 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6fdf7ff479-xlfkd" Sep 11 08:48:23.869360 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:48:23.868935 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6fdf7ff479-xlfkd" event={"ID":"64b9c16d-d0c5-423f-98e3-67d3fcff6128","Type":"ContainerDied","Data":"9563f7d329a69b8f7a025b6678a6cf4dd4967905e29f29fdee0fecb66f915f24"} Sep 11 08:48:23.869360 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:48:23.869068 2572 scope.go:117] "RemoveContainer" containerID="ee578656e3630e62ad3357dae7ed819cad48dad856f75cc08bee497e79a0a1cf" Sep 11 08:48:23.890921 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:48:23.890896 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6fdf7ff479-xlfkd"] Sep 11 08:48:23.896166 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:48:23.896141 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6fdf7ff479-xlfkd"] Sep 11 08:48:24.252093 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:48:24.252062 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="64b9c16d-d0c5-423f-98e3-67d3fcff6128" path="/var/lib/kubelet/pods/64b9c16d-d0c5-423f-98e3-67d3fcff6128/volumes" Sep 11 08:48:29.245488 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:48:29.245416 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-fdbb5cd6-nqdhs" Sep 11 08:48:29.249323 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:48:29.249305 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-fdbb5cd6-nqdhs" Sep 11 08:48:29.893562 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:48:29.893530 2572 generic.go:356] "Generic (PLEG): container finished" podID="9222c5e5-755c-460e-8568-db40848b8123" containerID="3e3b15d35c6ad814d1a61a29f22a86aa7167be0b01231763adad2ce0d8f93aac" exitCode=0 Sep 11 08:48:29.893733 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:48:29.893602 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-wdwcs" event={"ID":"9222c5e5-755c-460e-8568-db40848b8123","Type":"ContainerDied","Data":"3e3b15d35c6ad814d1a61a29f22a86aa7167be0b01231763adad2ce0d8f93aac"} Sep 11 08:48:29.894028 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:48:29.894012 2572 scope.go:117] "RemoveContainer" containerID="3e3b15d35c6ad814d1a61a29f22a86aa7167be0b01231763adad2ce0d8f93aac" Sep 11 08:48:30.899536 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:48:30.899506 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-wdwcs" event={"ID":"9222c5e5-755c-460e-8568-db40848b8123","Type":"ContainerStarted","Data":"db4204539c99be0dc9adc2ff1da8762ab70a24b1c1d19a9cac732dcde3a339cc"} Sep 11 08:48:34.914999 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:48:34.914913 2572 generic.go:356] "Generic (PLEG): container finished" podID="8a3bbadc-80e8-4436-9c8e-077a0dd03df3" containerID="14fde40295bb309abcd85304c4a727a71b17c41cdbda28bf67a17a8ed499f6e5" exitCode=0 Sep 11 08:48:34.914999 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:48:34.914985 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-xjhvj" event={"ID":"8a3bbadc-80e8-4436-9c8e-077a0dd03df3","Type":"ContainerDied","Data":"14fde40295bb309abcd85304c4a727a71b17c41cdbda28bf67a17a8ed499f6e5"} Sep 11 08:48:34.915405 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:48:34.915312 2572 scope.go:117] "RemoveContainer" containerID="14fde40295bb309abcd85304c4a727a71b17c41cdbda28bf67a17a8ed499f6e5" Sep 11 08:48:35.921006 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:48:35.920967 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-xjhvj" event={"ID":"8a3bbadc-80e8-4436-9c8e-077a0dd03df3","Type":"ContainerStarted","Data":"25ce01555596167fadae7a8c880822364bc3fa868ced54495b0de3cfa1153ff2"} Sep 11 08:48:38.913746 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:48:38.913696 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6b776758d8-mvh7l" podUID="1cc84abb-cf26-4eb7-b01e-cdb1c8e84fa0" containerName="console" containerID="cri-o://02af566951639302e6fd7d6e4d0a02f7de26c25104c6418dd7241dd80a20b902" gracePeriod=15 Sep 11 08:48:39.186910 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:48:39.186886 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b776758d8-mvh7l" Sep 11 08:48:39.197260 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:48:39.197218 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1cc84abb-cf26-4eb7-b01e-cdb1c8e84fa0-oauth-serving-cert\") pod \"1cc84abb-cf26-4eb7-b01e-cdb1c8e84fa0\" (UID: \"1cc84abb-cf26-4eb7-b01e-cdb1c8e84fa0\") " Sep 11 08:48:39.197361 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:48:39.197265 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1cc84abb-cf26-4eb7-b01e-cdb1c8e84fa0-console-oauth-config\") pod \"1cc84abb-cf26-4eb7-b01e-cdb1c8e84fa0\" (UID: \"1cc84abb-cf26-4eb7-b01e-cdb1c8e84fa0\") " Sep 11 08:48:39.197361 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:48:39.197287 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1cc84abb-cf26-4eb7-b01e-cdb1c8e84fa0-service-ca\") pod \"1cc84abb-cf26-4eb7-b01e-cdb1c8e84fa0\" (UID: \"1cc84abb-cf26-4eb7-b01e-cdb1c8e84fa0\") " Sep 11 08:48:39.197361 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:48:39.197337 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1cc84abb-cf26-4eb7-b01e-cdb1c8e84fa0-console-config\") pod \"1cc84abb-cf26-4eb7-b01e-cdb1c8e84fa0\" (UID: \"1cc84abb-cf26-4eb7-b01e-cdb1c8e84fa0\") " Sep 11 08:48:39.197484 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:48:39.197379 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1cc84abb-cf26-4eb7-b01e-cdb1c8e84fa0-console-serving-cert\") pod \"1cc84abb-cf26-4eb7-b01e-cdb1c8e84fa0\" (UID: \"1cc84abb-cf26-4eb7-b01e-cdb1c8e84fa0\") " Sep 11 08:48:39.197484 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:48:39.197437 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zxsgx\" (UniqueName: \"kubernetes.io/projected/1cc84abb-cf26-4eb7-b01e-cdb1c8e84fa0-kube-api-access-zxsgx\") pod \"1cc84abb-cf26-4eb7-b01e-cdb1c8e84fa0\" (UID: \"1cc84abb-cf26-4eb7-b01e-cdb1c8e84fa0\") " Sep 11 08:48:39.197688 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:48:39.197654 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cc84abb-cf26-4eb7-b01e-cdb1c8e84fa0-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "1cc84abb-cf26-4eb7-b01e-cdb1c8e84fa0" (UID: "1cc84abb-cf26-4eb7-b01e-cdb1c8e84fa0"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 08:48:39.197795 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:48:39.197719 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cc84abb-cf26-4eb7-b01e-cdb1c8e84fa0-console-config" (OuterVolumeSpecName: "console-config") pod "1cc84abb-cf26-4eb7-b01e-cdb1c8e84fa0" (UID: "1cc84abb-cf26-4eb7-b01e-cdb1c8e84fa0"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 08:48:39.197795 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:48:39.197727 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cc84abb-cf26-4eb7-b01e-cdb1c8e84fa0-service-ca" (OuterVolumeSpecName: "service-ca") pod "1cc84abb-cf26-4eb7-b01e-cdb1c8e84fa0" (UID: "1cc84abb-cf26-4eb7-b01e-cdb1c8e84fa0"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 08:48:39.199814 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:48:39.199787 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1cc84abb-cf26-4eb7-b01e-cdb1c8e84fa0-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "1cc84abb-cf26-4eb7-b01e-cdb1c8e84fa0" (UID: "1cc84abb-cf26-4eb7-b01e-cdb1c8e84fa0"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 08:48:39.199814 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:48:39.199800 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cc84abb-cf26-4eb7-b01e-cdb1c8e84fa0-kube-api-access-zxsgx" (OuterVolumeSpecName: "kube-api-access-zxsgx") pod "1cc84abb-cf26-4eb7-b01e-cdb1c8e84fa0" (UID: "1cc84abb-cf26-4eb7-b01e-cdb1c8e84fa0"). InnerVolumeSpecName "kube-api-access-zxsgx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 08:48:39.199952 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:48:39.199890 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1cc84abb-cf26-4eb7-b01e-cdb1c8e84fa0-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "1cc84abb-cf26-4eb7-b01e-cdb1c8e84fa0" (UID: "1cc84abb-cf26-4eb7-b01e-cdb1c8e84fa0"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 08:48:39.298521 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:48:39.298494 2572 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1cc84abb-cf26-4eb7-b01e-cdb1c8e84fa0-console-config\") on node \"ip-10-0-128-166.ec2.internal\" DevicePath \"\"" Sep 11 08:48:39.298521 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:48:39.298518 2572 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1cc84abb-cf26-4eb7-b01e-cdb1c8e84fa0-console-serving-cert\") on node \"ip-10-0-128-166.ec2.internal\" DevicePath \"\"" Sep 11 08:48:39.298521 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:48:39.298528 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxsgx\" (UniqueName: \"kubernetes.io/projected/1cc84abb-cf26-4eb7-b01e-cdb1c8e84fa0-kube-api-access-zxsgx\") on node \"ip-10-0-128-166.ec2.internal\" DevicePath \"\"" Sep 11 08:48:39.298711 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:48:39.298537 2572 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1cc84abb-cf26-4eb7-b01e-cdb1c8e84fa0-oauth-serving-cert\") on node \"ip-10-0-128-166.ec2.internal\" DevicePath \"\"" Sep 11 08:48:39.298711 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:48:39.298546 2572 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1cc84abb-cf26-4eb7-b01e-cdb1c8e84fa0-console-oauth-config\") on node \"ip-10-0-128-166.ec2.internal\" DevicePath \"\"" Sep 11 08:48:39.298711 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:48:39.298554 2572 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1cc84abb-cf26-4eb7-b01e-cdb1c8e84fa0-service-ca\") on node \"ip-10-0-128-166.ec2.internal\" DevicePath \"\"" Sep 11 08:48:39.939775 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:48:39.939733 2572 generic.go:356] "Generic (PLEG): container finished" podID="1d7a379e-1344-48f2-9540-7ac11ee14eae" containerID="86f878c7af8a075aeaa91c1f3e43973413a38bcaf80ee60387b4610ff32092f6" exitCode=0 Sep 11 08:48:39.940323 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:48:39.939812 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-z8sdf" event={"ID":"1d7a379e-1344-48f2-9540-7ac11ee14eae","Type":"ContainerDied","Data":"86f878c7af8a075aeaa91c1f3e43973413a38bcaf80ee60387b4610ff32092f6"} Sep 11 08:48:39.940323 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:48:39.940192 2572 scope.go:117] "RemoveContainer" containerID="86f878c7af8a075aeaa91c1f3e43973413a38bcaf80ee60387b4610ff32092f6" Sep 11 08:48:39.941169 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:48:39.941144 2572 generic.go:356] "Generic (PLEG): container finished" podID="1cc84abb-cf26-4eb7-b01e-cdb1c8e84fa0" containerID="02af566951639302e6fd7d6e4d0a02f7de26c25104c6418dd7241dd80a20b902" exitCode=0 Sep 11 08:48:39.941307 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:48:39.941189 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b776758d8-mvh7l" Sep 11 08:48:39.941307 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:48:39.941255 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b776758d8-mvh7l" event={"ID":"1cc84abb-cf26-4eb7-b01e-cdb1c8e84fa0","Type":"ContainerDied","Data":"02af566951639302e6fd7d6e4d0a02f7de26c25104c6418dd7241dd80a20b902"} Sep 11 08:48:39.941307 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:48:39.941303 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b776758d8-mvh7l" event={"ID":"1cc84abb-cf26-4eb7-b01e-cdb1c8e84fa0","Type":"ContainerDied","Data":"57e5ec89a70ea1f879eff52adb9c053d837daaa4b26e65278a68b36b06e07878"} Sep 11 08:48:39.941435 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:48:39.941327 2572 scope.go:117] "RemoveContainer" containerID="02af566951639302e6fd7d6e4d0a02f7de26c25104c6418dd7241dd80a20b902" Sep 11 08:48:39.950699 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:48:39.950685 2572 scope.go:117] "RemoveContainer" containerID="02af566951639302e6fd7d6e4d0a02f7de26c25104c6418dd7241dd80a20b902" Sep 11 08:48:39.950950 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:48:39.950932 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02af566951639302e6fd7d6e4d0a02f7de26c25104c6418dd7241dd80a20b902\": container with ID starting with 02af566951639302e6fd7d6e4d0a02f7de26c25104c6418dd7241dd80a20b902 not found: ID does not exist" containerID="02af566951639302e6fd7d6e4d0a02f7de26c25104c6418dd7241dd80a20b902" Sep 11 08:48:39.951004 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:48:39.950956 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02af566951639302e6fd7d6e4d0a02f7de26c25104c6418dd7241dd80a20b902"} err="failed to get container status \"02af566951639302e6fd7d6e4d0a02f7de26c25104c6418dd7241dd80a20b902\": rpc error: code = NotFound desc = could not find container \"02af566951639302e6fd7d6e4d0a02f7de26c25104c6418dd7241dd80a20b902\": container with ID starting with 02af566951639302e6fd7d6e4d0a02f7de26c25104c6418dd7241dd80a20b902 not found: ID does not exist" Sep 11 08:48:39.967620 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:48:39.967599 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6b776758d8-mvh7l"] Sep 11 08:48:39.976577 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:48:39.976458 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6b776758d8-mvh7l"] Sep 11 08:48:40.252636 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:48:40.252552 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1cc84abb-cf26-4eb7-b01e-cdb1c8e84fa0" path="/var/lib/kubelet/pods/1cc84abb-cf26-4eb7-b01e-cdb1c8e84fa0/volumes" Sep 11 08:48:40.948556 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:48:40.948516 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-z8sdf" event={"ID":"1d7a379e-1344-48f2-9540-7ac11ee14eae","Type":"ContainerStarted","Data":"eb0e279fc86591dc8dbfdd262ac4553687486e73776800b8ed0382ca77bf3f2a"} Sep 11 08:49:08.841816 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:49:08.841779 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-7c67c9b8f-zfw99"] Sep 11 08:49:08.842337 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:49:08.842083 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1cc84abb-cf26-4eb7-b01e-cdb1c8e84fa0" containerName="console" Sep 11 08:49:08.842337 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:49:08.842094 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="1cc84abb-cf26-4eb7-b01e-cdb1c8e84fa0" containerName="console" Sep 11 08:49:08.842337 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:49:08.842103 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64b9c16d-d0c5-423f-98e3-67d3fcff6128" containerName="registry" Sep 11 08:49:08.842337 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:49:08.842109 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="64b9c16d-d0c5-423f-98e3-67d3fcff6128" containerName="registry" Sep 11 08:49:08.842337 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:49:08.842161 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="1cc84abb-cf26-4eb7-b01e-cdb1c8e84fa0" containerName="console" Sep 11 08:49:08.842337 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:49:08.842168 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="64b9c16d-d0c5-423f-98e3-67d3fcff6128" containerName="registry" Sep 11 08:49:08.846558 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:49:08.846539 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7c67c9b8f-zfw99" Sep 11 08:49:08.848656 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:49:08.848633 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Sep 11 08:49:08.848757 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:49:08.848656 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-c6knl\"" Sep 11 08:49:08.848959 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:49:08.848942 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Sep 11 08:49:08.849038 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:49:08.848967 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Sep 11 08:49:08.849038 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:49:08.849008 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Sep 11 08:49:08.849144 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:49:08.849072 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Sep 11 08:49:08.853740 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:49:08.853571 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Sep 11 08:49:08.856852 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:49:08.856833 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7c67c9b8f-zfw99"] Sep 11 08:49:08.911545 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:49:08.911521 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b0a05c4-b687-4c17-94e6-11d6d473c332-serving-certs-ca-bundle\") pod \"telemeter-client-7c67c9b8f-zfw99\" (UID: \"1b0a05c4-b687-4c17-94e6-11d6d473c332\") " pod="openshift-monitoring/telemeter-client-7c67c9b8f-zfw99" Sep 11 08:49:08.911646 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:49:08.911554 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/1b0a05c4-b687-4c17-94e6-11d6d473c332-telemeter-client-tls\") pod \"telemeter-client-7c67c9b8f-zfw99\" (UID: \"1b0a05c4-b687-4c17-94e6-11d6d473c332\") " pod="openshift-monitoring/telemeter-client-7c67c9b8f-zfw99" Sep 11 08:49:08.911646 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:49:08.911571 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1b0a05c4-b687-4c17-94e6-11d6d473c332-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7c67c9b8f-zfw99\" (UID: \"1b0a05c4-b687-4c17-94e6-11d6d473c332\") " pod="openshift-monitoring/telemeter-client-7c67c9b8f-zfw99" Sep 11 08:49:08.911646 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:49:08.911590 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b0a05c4-b687-4c17-94e6-11d6d473c332-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7c67c9b8f-zfw99\" (UID: \"1b0a05c4-b687-4c17-94e6-11d6d473c332\") " pod="openshift-monitoring/telemeter-client-7c67c9b8f-zfw99" Sep 11 08:49:08.911646 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:49:08.911606 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddctr\" (UniqueName: \"kubernetes.io/projected/1b0a05c4-b687-4c17-94e6-11d6d473c332-kube-api-access-ddctr\") pod \"telemeter-client-7c67c9b8f-zfw99\" (UID: \"1b0a05c4-b687-4c17-94e6-11d6d473c332\") " pod="openshift-monitoring/telemeter-client-7c67c9b8f-zfw99" Sep 11 08:49:08.911779 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:49:08.911709 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/1b0a05c4-b687-4c17-94e6-11d6d473c332-federate-client-tls\") pod \"telemeter-client-7c67c9b8f-zfw99\" (UID: \"1b0a05c4-b687-4c17-94e6-11d6d473c332\") " pod="openshift-monitoring/telemeter-client-7c67c9b8f-zfw99" Sep 11 08:49:08.911779 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:49:08.911746 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1b0a05c4-b687-4c17-94e6-11d6d473c332-metrics-client-ca\") pod \"telemeter-client-7c67c9b8f-zfw99\" (UID: \"1b0a05c4-b687-4c17-94e6-11d6d473c332\") " pod="openshift-monitoring/telemeter-client-7c67c9b8f-zfw99" Sep 11 08:49:08.911839 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:49:08.911777 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/1b0a05c4-b687-4c17-94e6-11d6d473c332-secret-telemeter-client\") pod \"telemeter-client-7c67c9b8f-zfw99\" (UID: \"1b0a05c4-b687-4c17-94e6-11d6d473c332\") " pod="openshift-monitoring/telemeter-client-7c67c9b8f-zfw99" Sep 11 08:49:09.012761 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:49:09.012739 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b0a05c4-b687-4c17-94e6-11d6d473c332-serving-certs-ca-bundle\") pod \"telemeter-client-7c67c9b8f-zfw99\" (UID: \"1b0a05c4-b687-4c17-94e6-11d6d473c332\") " pod="openshift-monitoring/telemeter-client-7c67c9b8f-zfw99" Sep 11 08:49:09.012858 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:49:09.012769 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/1b0a05c4-b687-4c17-94e6-11d6d473c332-telemeter-client-tls\") pod \"telemeter-client-7c67c9b8f-zfw99\" (UID: \"1b0a05c4-b687-4c17-94e6-11d6d473c332\") " pod="openshift-monitoring/telemeter-client-7c67c9b8f-zfw99" Sep 11 08:49:09.012858 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:49:09.012787 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1b0a05c4-b687-4c17-94e6-11d6d473c332-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7c67c9b8f-zfw99\" (UID: \"1b0a05c4-b687-4c17-94e6-11d6d473c332\") " pod="openshift-monitoring/telemeter-client-7c67c9b8f-zfw99" Sep 11 08:49:09.012858 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:49:09.012808 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b0a05c4-b687-4c17-94e6-11d6d473c332-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7c67c9b8f-zfw99\" (UID: \"1b0a05c4-b687-4c17-94e6-11d6d473c332\") " pod="openshift-monitoring/telemeter-client-7c67c9b8f-zfw99" Sep 11 08:49:09.012858 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:49:09.012830 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddctr\" (UniqueName: \"kubernetes.io/projected/1b0a05c4-b687-4c17-94e6-11d6d473c332-kube-api-access-ddctr\") pod \"telemeter-client-7c67c9b8f-zfw99\" (UID: \"1b0a05c4-b687-4c17-94e6-11d6d473c332\") " pod="openshift-monitoring/telemeter-client-7c67c9b8f-zfw99" Sep 11 08:49:09.013075 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:49:09.012879 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/1b0a05c4-b687-4c17-94e6-11d6d473c332-federate-client-tls\") pod \"telemeter-client-7c67c9b8f-zfw99\" (UID: \"1b0a05c4-b687-4c17-94e6-11d6d473c332\") " pod="openshift-monitoring/telemeter-client-7c67c9b8f-zfw99" Sep 11 08:49:09.013075 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:49:09.012916 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1b0a05c4-b687-4c17-94e6-11d6d473c332-metrics-client-ca\") pod \"telemeter-client-7c67c9b8f-zfw99\" (UID: \"1b0a05c4-b687-4c17-94e6-11d6d473c332\") " pod="openshift-monitoring/telemeter-client-7c67c9b8f-zfw99" Sep 11 08:49:09.013075 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:49:09.012949 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/1b0a05c4-b687-4c17-94e6-11d6d473c332-secret-telemeter-client\") pod \"telemeter-client-7c67c9b8f-zfw99\" (UID: \"1b0a05c4-b687-4c17-94e6-11d6d473c332\") " pod="openshift-monitoring/telemeter-client-7c67c9b8f-zfw99" Sep 11 08:49:09.013558 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:49:09.013533 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b0a05c4-b687-4c17-94e6-11d6d473c332-serving-certs-ca-bundle\") pod \"telemeter-client-7c67c9b8f-zfw99\" (UID: \"1b0a05c4-b687-4c17-94e6-11d6d473c332\") " pod="openshift-monitoring/telemeter-client-7c67c9b8f-zfw99" Sep 11 08:49:09.013745 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:49:09.013714 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1b0a05c4-b687-4c17-94e6-11d6d473c332-metrics-client-ca\") pod \"telemeter-client-7c67c9b8f-zfw99\" (UID: \"1b0a05c4-b687-4c17-94e6-11d6d473c332\") " pod="openshift-monitoring/telemeter-client-7c67c9b8f-zfw99" Sep 11 08:49:09.013828 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:49:09.013769 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b0a05c4-b687-4c17-94e6-11d6d473c332-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7c67c9b8f-zfw99\" (UID: \"1b0a05c4-b687-4c17-94e6-11d6d473c332\") " pod="openshift-monitoring/telemeter-client-7c67c9b8f-zfw99" Sep 11 08:49:09.015805 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:49:09.015779 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/1b0a05c4-b687-4c17-94e6-11d6d473c332-secret-telemeter-client\") pod \"telemeter-client-7c67c9b8f-zfw99\" (UID: \"1b0a05c4-b687-4c17-94e6-11d6d473c332\") " pod="openshift-monitoring/telemeter-client-7c67c9b8f-zfw99" Sep 11 08:49:09.015982 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:49:09.015965 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/1b0a05c4-b687-4c17-94e6-11d6d473c332-federate-client-tls\") pod \"telemeter-client-7c67c9b8f-zfw99\" (UID: \"1b0a05c4-b687-4c17-94e6-11d6d473c332\") " pod="openshift-monitoring/telemeter-client-7c67c9b8f-zfw99" Sep 11 08:49:09.016048 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:49:09.016003 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1b0a05c4-b687-4c17-94e6-11d6d473c332-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7c67c9b8f-zfw99\" (UID: \"1b0a05c4-b687-4c17-94e6-11d6d473c332\") " pod="openshift-monitoring/telemeter-client-7c67c9b8f-zfw99" Sep 11 08:49:09.016048 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:49:09.016034 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/1b0a05c4-b687-4c17-94e6-11d6d473c332-telemeter-client-tls\") pod \"telemeter-client-7c67c9b8f-zfw99\" (UID: \"1b0a05c4-b687-4c17-94e6-11d6d473c332\") " pod="openshift-monitoring/telemeter-client-7c67c9b8f-zfw99" Sep 11 08:49:09.024915 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:49:09.024893 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddctr\" (UniqueName: \"kubernetes.io/projected/1b0a05c4-b687-4c17-94e6-11d6d473c332-kube-api-access-ddctr\") pod \"telemeter-client-7c67c9b8f-zfw99\" (UID: \"1b0a05c4-b687-4c17-94e6-11d6d473c332\") " pod="openshift-monitoring/telemeter-client-7c67c9b8f-zfw99" Sep 11 08:49:09.157445 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:49:09.157384 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7c67c9b8f-zfw99" Sep 11 08:49:09.458765 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:49:09.458742 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7c67c9b8f-zfw99"] Sep 11 08:49:10.058630 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:49:10.058597 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7c67c9b8f-zfw99" event={"ID":"1b0a05c4-b687-4c17-94e6-11d6d473c332","Type":"ContainerStarted","Data":"c983cc02e3007bf6e7d394ed5bbdcae56f20c9e1113f8e797c88bef12fc7d3bb"} Sep 11 08:49:12.070570 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:49:12.070533 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7c67c9b8f-zfw99" event={"ID":"1b0a05c4-b687-4c17-94e6-11d6d473c332","Type":"ContainerStarted","Data":"5612e9b27d06a5aae1e2c273166ac05acc57d16317dc1872c05ddfa383a93628"} Sep 11 08:49:13.076629 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:49:13.076593 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7c67c9b8f-zfw99" event={"ID":"1b0a05c4-b687-4c17-94e6-11d6d473c332","Type":"ContainerStarted","Data":"e251e1bddebba5009566671300ed2711cc8323e864f1f6423a8685e5f58bb48c"} Sep 11 08:49:13.077070 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:49:13.076674 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7c67c9b8f-zfw99" event={"ID":"1b0a05c4-b687-4c17-94e6-11d6d473c332","Type":"ContainerStarted","Data":"06162a297cbef29c79302986b5db8a7e3c9fd8546a0198ceff80833f98ae130d"} Sep 11 08:49:13.098052 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:49:13.098006 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-7c67c9b8f-zfw99" podStartSLOduration=2.051273307 podStartE2EDuration="5.0979935s" totalImagesPullingTime="3.046720193s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 08:49:08 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 08:49:13.097128073 +0000 UTC m=+173.371703771" watchObservedRunningTime="2026-09-11 08:49:13.0979935 +0000 UTC m=+173.372569196" Sep 11 08:49:13.181610 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:49:13.181581 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-65dd8997c9-5s7kb"] Sep 11 08:49:13.185219 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:49:13.185200 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65dd8997c9-5s7kb" Sep 11 08:49:13.196778 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:49:13.196718 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-65dd8997c9-5s7kb"] Sep 11 08:49:13.251512 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:49:13.251484 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6bfb1765-e529-43b8-952a-41e30698841a-console-config\") pod \"console-65dd8997c9-5s7kb\" (UID: \"6bfb1765-e529-43b8-952a-41e30698841a\") " pod="openshift-console/console-65dd8997c9-5s7kb" Sep 11 08:49:13.251657 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:49:13.251516 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fb2vr\" (UniqueName: \"kubernetes.io/projected/6bfb1765-e529-43b8-952a-41e30698841a-kube-api-access-fb2vr\") pod \"console-65dd8997c9-5s7kb\" (UID: \"6bfb1765-e529-43b8-952a-41e30698841a\") " pod="openshift-console/console-65dd8997c9-5s7kb" Sep 11 08:49:13.251657 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:49:13.251542 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6bfb1765-e529-43b8-952a-41e30698841a-console-serving-cert\") pod \"console-65dd8997c9-5s7kb\" (UID: \"6bfb1765-e529-43b8-952a-41e30698841a\") " pod="openshift-console/console-65dd8997c9-5s7kb" Sep 11 08:49:13.251657 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:49:13.251563 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6bfb1765-e529-43b8-952a-41e30698841a-console-oauth-config\") pod \"console-65dd8997c9-5s7kb\" (UID: \"6bfb1765-e529-43b8-952a-41e30698841a\") " pod="openshift-console/console-65dd8997c9-5s7kb" Sep 11 08:49:13.251657 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:49:13.251610 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6bfb1765-e529-43b8-952a-41e30698841a-trusted-ca-bundle\") pod \"console-65dd8997c9-5s7kb\" (UID: \"6bfb1765-e529-43b8-952a-41e30698841a\") " pod="openshift-console/console-65dd8997c9-5s7kb" Sep 11 08:49:13.251657 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:49:13.251650 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6bfb1765-e529-43b8-952a-41e30698841a-oauth-serving-cert\") pod \"console-65dd8997c9-5s7kb\" (UID: \"6bfb1765-e529-43b8-952a-41e30698841a\") " pod="openshift-console/console-65dd8997c9-5s7kb" Sep 11 08:49:13.251817 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:49:13.251721 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6bfb1765-e529-43b8-952a-41e30698841a-service-ca\") pod \"console-65dd8997c9-5s7kb\" (UID: \"6bfb1765-e529-43b8-952a-41e30698841a\") " pod="openshift-console/console-65dd8997c9-5s7kb" Sep 11 08:49:13.352537 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:49:13.352465 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6bfb1765-e529-43b8-952a-41e30698841a-service-ca\") pod \"console-65dd8997c9-5s7kb\" (UID: \"6bfb1765-e529-43b8-952a-41e30698841a\") " pod="openshift-console/console-65dd8997c9-5s7kb" Sep 11 08:49:13.352537 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:49:13.352497 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6bfb1765-e529-43b8-952a-41e30698841a-console-config\") pod \"console-65dd8997c9-5s7kb\" (UID: \"6bfb1765-e529-43b8-952a-41e30698841a\") " pod="openshift-console/console-65dd8997c9-5s7kb" Sep 11 08:49:13.352537 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:49:13.352514 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fb2vr\" (UniqueName: \"kubernetes.io/projected/6bfb1765-e529-43b8-952a-41e30698841a-kube-api-access-fb2vr\") pod \"console-65dd8997c9-5s7kb\" (UID: \"6bfb1765-e529-43b8-952a-41e30698841a\") " pod="openshift-console/console-65dd8997c9-5s7kb" Sep 11 08:49:13.352763 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:49:13.352539 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6bfb1765-e529-43b8-952a-41e30698841a-console-serving-cert\") pod \"console-65dd8997c9-5s7kb\" (UID: \"6bfb1765-e529-43b8-952a-41e30698841a\") " pod="openshift-console/console-65dd8997c9-5s7kb" Sep 11 08:49:13.352763 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:49:13.352568 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6bfb1765-e529-43b8-952a-41e30698841a-console-oauth-config\") pod \"console-65dd8997c9-5s7kb\" (UID: \"6bfb1765-e529-43b8-952a-41e30698841a\") " pod="openshift-console/console-65dd8997c9-5s7kb" Sep 11 08:49:13.352763 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:49:13.352676 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6bfb1765-e529-43b8-952a-41e30698841a-trusted-ca-bundle\") pod \"console-65dd8997c9-5s7kb\" (UID: \"6bfb1765-e529-43b8-952a-41e30698841a\") " pod="openshift-console/console-65dd8997c9-5s7kb" Sep 11 08:49:13.352763 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:49:13.352746 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6bfb1765-e529-43b8-952a-41e30698841a-oauth-serving-cert\") pod \"console-65dd8997c9-5s7kb\" (UID: \"6bfb1765-e529-43b8-952a-41e30698841a\") " pod="openshift-console/console-65dd8997c9-5s7kb" Sep 11 08:49:13.353351 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:49:13.353329 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6bfb1765-e529-43b8-952a-41e30698841a-service-ca\") pod \"console-65dd8997c9-5s7kb\" (UID: \"6bfb1765-e529-43b8-952a-41e30698841a\") " pod="openshift-console/console-65dd8997c9-5s7kb" Sep 11 08:49:13.353454 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:49:13.353360 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6bfb1765-e529-43b8-952a-41e30698841a-console-config\") pod \"console-65dd8997c9-5s7kb\" (UID: \"6bfb1765-e529-43b8-952a-41e30698841a\") " pod="openshift-console/console-65dd8997c9-5s7kb" Sep 11 08:49:13.353492 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:49:13.353465 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6bfb1765-e529-43b8-952a-41e30698841a-oauth-serving-cert\") pod \"console-65dd8997c9-5s7kb\" (UID: \"6bfb1765-e529-43b8-952a-41e30698841a\") " pod="openshift-console/console-65dd8997c9-5s7kb" Sep 11 08:49:13.353661 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:49:13.353644 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6bfb1765-e529-43b8-952a-41e30698841a-trusted-ca-bundle\") pod \"console-65dd8997c9-5s7kb\" (UID: \"6bfb1765-e529-43b8-952a-41e30698841a\") " pod="openshift-console/console-65dd8997c9-5s7kb" Sep 11 08:49:13.355103 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:49:13.355078 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6bfb1765-e529-43b8-952a-41e30698841a-console-serving-cert\") pod \"console-65dd8997c9-5s7kb\" (UID: \"6bfb1765-e529-43b8-952a-41e30698841a\") " pod="openshift-console/console-65dd8997c9-5s7kb" Sep 11 08:49:13.355176 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:49:13.355159 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6bfb1765-e529-43b8-952a-41e30698841a-console-oauth-config\") pod \"console-65dd8997c9-5s7kb\" (UID: \"6bfb1765-e529-43b8-952a-41e30698841a\") " pod="openshift-console/console-65dd8997c9-5s7kb" Sep 11 08:49:13.359965 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:49:13.359946 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fb2vr\" (UniqueName: \"kubernetes.io/projected/6bfb1765-e529-43b8-952a-41e30698841a-kube-api-access-fb2vr\") pod \"console-65dd8997c9-5s7kb\" (UID: \"6bfb1765-e529-43b8-952a-41e30698841a\") " pod="openshift-console/console-65dd8997c9-5s7kb" Sep 11 08:49:13.495587 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:49:13.495560 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65dd8997c9-5s7kb" Sep 11 08:49:13.767996 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:49:13.767709 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-65dd8997c9-5s7kb"] Sep 11 08:49:13.770278 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:49:13.770204 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bfb1765_e529_43b8_952a_41e30698841a.slice/crio-6e0a8bc0f6e8329ee28d706af20e76dd7874be082d4094d1e5cd3a3d42f1845a WatchSource:0}: Error finding container 6e0a8bc0f6e8329ee28d706af20e76dd7874be082d4094d1e5cd3a3d42f1845a: Status 404 returned error can't find the container with id 6e0a8bc0f6e8329ee28d706af20e76dd7874be082d4094d1e5cd3a3d42f1845a Sep 11 08:49:14.082367 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:49:14.082286 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65dd8997c9-5s7kb" event={"ID":"6bfb1765-e529-43b8-952a-41e30698841a","Type":"ContainerStarted","Data":"820a344ffa61938d0fd6a17ed2b02a959875412c1a9488e198ba3a9a46efd2e1"} Sep 11 08:49:14.082367 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:49:14.082341 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65dd8997c9-5s7kb" event={"ID":"6bfb1765-e529-43b8-952a-41e30698841a","Type":"ContainerStarted","Data":"6e0a8bc0f6e8329ee28d706af20e76dd7874be082d4094d1e5cd3a3d42f1845a"} Sep 11 08:49:14.101376 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:49:14.101330 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-65dd8997c9-5s7kb" podStartSLOduration=1.101315621 podStartE2EDuration="1.101315621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 08:49:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 08:49:14.100640801 +0000 UTC m=+174.375216497" watchObservedRunningTime="2026-09-11 08:49:14.101315621 +0000 UTC m=+174.375891318" Sep 11 08:49:23.496466 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:49:23.496432 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-65dd8997c9-5s7kb" Sep 11 08:49:23.496904 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:49:23.496533 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-65dd8997c9-5s7kb" Sep 11 08:49:23.501457 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:49:23.501435 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-65dd8997c9-5s7kb" Sep 11 08:49:24.129295 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:49:24.129267 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-65dd8997c9-5s7kb" Sep 11 08:49:24.172050 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:49:24.172018 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-64bf664cf5-w4xqt"] Sep 11 08:49:49.194386 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:49:49.194326 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-64bf664cf5-w4xqt" podUID="223b0e88-dc9c-464f-ae62-eab365dc7abd" containerName="console" containerID="cri-o://a81122664fd28d3688224602fe9b32d67d7860172b8667dae4814efd6ea18b1d" gracePeriod=15 Sep 11 08:49:49.424388 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:49:49.424362 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64bf664cf5-w4xqt" Sep 11 08:49:49.541835 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:49:49.541764 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/223b0e88-dc9c-464f-ae62-eab365dc7abd-console-oauth-config\") pod \"223b0e88-dc9c-464f-ae62-eab365dc7abd\" (UID: \"223b0e88-dc9c-464f-ae62-eab365dc7abd\") " Sep 11 08:49:49.541835 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:49:49.541800 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/223b0e88-dc9c-464f-ae62-eab365dc7abd-service-ca\") pod \"223b0e88-dc9c-464f-ae62-eab365dc7abd\" (UID: \"223b0e88-dc9c-464f-ae62-eab365dc7abd\") " Sep 11 08:49:49.541835 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:49:49.541836 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/223b0e88-dc9c-464f-ae62-eab365dc7abd-console-serving-cert\") pod \"223b0e88-dc9c-464f-ae62-eab365dc7abd\" (UID: \"223b0e88-dc9c-464f-ae62-eab365dc7abd\") " Sep 11 08:49:49.542077 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:49:49.541858 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mctsh\" (UniqueName: \"kubernetes.io/projected/223b0e88-dc9c-464f-ae62-eab365dc7abd-kube-api-access-mctsh\") pod \"223b0e88-dc9c-464f-ae62-eab365dc7abd\" (UID: \"223b0e88-dc9c-464f-ae62-eab365dc7abd\") " Sep 11 08:49:49.542077 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:49:49.541887 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/223b0e88-dc9c-464f-ae62-eab365dc7abd-console-config\") pod \"223b0e88-dc9c-464f-ae62-eab365dc7abd\" (UID: \"223b0e88-dc9c-464f-ae62-eab365dc7abd\") " Sep 11 08:49:49.542077 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:49:49.541916 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/223b0e88-dc9c-464f-ae62-eab365dc7abd-oauth-serving-cert\") pod \"223b0e88-dc9c-464f-ae62-eab365dc7abd\" (UID: \"223b0e88-dc9c-464f-ae62-eab365dc7abd\") " Sep 11 08:49:49.542077 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:49:49.541934 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/223b0e88-dc9c-464f-ae62-eab365dc7abd-trusted-ca-bundle\") pod \"223b0e88-dc9c-464f-ae62-eab365dc7abd\" (UID: \"223b0e88-dc9c-464f-ae62-eab365dc7abd\") " Sep 11 08:49:49.542336 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:49:49.542306 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/223b0e88-dc9c-464f-ae62-eab365dc7abd-service-ca" (OuterVolumeSpecName: "service-ca") pod "223b0e88-dc9c-464f-ae62-eab365dc7abd" (UID: "223b0e88-dc9c-464f-ae62-eab365dc7abd"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 08:49:49.542397 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:49:49.542369 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/223b0e88-dc9c-464f-ae62-eab365dc7abd-console-config" (OuterVolumeSpecName: "console-config") pod "223b0e88-dc9c-464f-ae62-eab365dc7abd" (UID: "223b0e88-dc9c-464f-ae62-eab365dc7abd"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 08:49:49.542443 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:49:49.542406 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/223b0e88-dc9c-464f-ae62-eab365dc7abd-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "223b0e88-dc9c-464f-ae62-eab365dc7abd" (UID: "223b0e88-dc9c-464f-ae62-eab365dc7abd"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 08:49:49.542492 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:49:49.542467 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/223b0e88-dc9c-464f-ae62-eab365dc7abd-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "223b0e88-dc9c-464f-ae62-eab365dc7abd" (UID: "223b0e88-dc9c-464f-ae62-eab365dc7abd"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 08:49:49.544276 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:49:49.544252 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/223b0e88-dc9c-464f-ae62-eab365dc7abd-kube-api-access-mctsh" (OuterVolumeSpecName: "kube-api-access-mctsh") pod "223b0e88-dc9c-464f-ae62-eab365dc7abd" (UID: "223b0e88-dc9c-464f-ae62-eab365dc7abd"). InnerVolumeSpecName "kube-api-access-mctsh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 08:49:49.544587 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:49:49.544569 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/223b0e88-dc9c-464f-ae62-eab365dc7abd-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "223b0e88-dc9c-464f-ae62-eab365dc7abd" (UID: "223b0e88-dc9c-464f-ae62-eab365dc7abd"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 08:49:49.544635 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:49:49.544592 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/223b0e88-dc9c-464f-ae62-eab365dc7abd-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "223b0e88-dc9c-464f-ae62-eab365dc7abd" (UID: "223b0e88-dc9c-464f-ae62-eab365dc7abd"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 08:49:49.643365 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:49:49.643341 2572 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/223b0e88-dc9c-464f-ae62-eab365dc7abd-console-config\") on node \"ip-10-0-128-166.ec2.internal\" DevicePath \"\"" Sep 11 08:49:49.643365 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:49:49.643361 2572 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/223b0e88-dc9c-464f-ae62-eab365dc7abd-oauth-serving-cert\") on node \"ip-10-0-128-166.ec2.internal\" DevicePath \"\"" Sep 11 08:49:49.643496 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:49:49.643382 2572 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/223b0e88-dc9c-464f-ae62-eab365dc7abd-trusted-ca-bundle\") on node \"ip-10-0-128-166.ec2.internal\" DevicePath \"\"" Sep 11 08:49:49.643496 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:49:49.643393 2572 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/223b0e88-dc9c-464f-ae62-eab365dc7abd-console-oauth-config\") on node \"ip-10-0-128-166.ec2.internal\" DevicePath \"\"" Sep 11 08:49:49.643496 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:49:49.643401 2572 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/223b0e88-dc9c-464f-ae62-eab365dc7abd-service-ca\") on node \"ip-10-0-128-166.ec2.internal\" DevicePath \"\"" Sep 11 08:49:49.643496 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:49:49.643410 2572 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/223b0e88-dc9c-464f-ae62-eab365dc7abd-console-serving-cert\") on node \"ip-10-0-128-166.ec2.internal\" DevicePath \"\"" Sep 11 08:49:49.643496 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:49:49.643417 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mctsh\" (UniqueName: \"kubernetes.io/projected/223b0e88-dc9c-464f-ae62-eab365dc7abd-kube-api-access-mctsh\") on node \"ip-10-0-128-166.ec2.internal\" DevicePath \"\"" Sep 11 08:49:50.231677 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:49:50.231638 2572 generic.go:356] "Generic (PLEG): container finished" podID="223b0e88-dc9c-464f-ae62-eab365dc7abd" containerID="a81122664fd28d3688224602fe9b32d67d7860172b8667dae4814efd6ea18b1d" exitCode=0 Sep 11 08:49:50.232069 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:49:50.231690 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64bf664cf5-w4xqt" Sep 11 08:49:50.232069 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:49:50.231694 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64bf664cf5-w4xqt" event={"ID":"223b0e88-dc9c-464f-ae62-eab365dc7abd","Type":"ContainerDied","Data":"a81122664fd28d3688224602fe9b32d67d7860172b8667dae4814efd6ea18b1d"} Sep 11 08:49:50.232069 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:49:50.231741 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64bf664cf5-w4xqt" event={"ID":"223b0e88-dc9c-464f-ae62-eab365dc7abd","Type":"ContainerDied","Data":"212e0c43f14dfd52fa73c8dd66808768a9d7d07a0aa9bb7d6c0c2491ceb660f7"} Sep 11 08:49:50.232069 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:49:50.231767 2572 scope.go:117] "RemoveContainer" containerID="a81122664fd28d3688224602fe9b32d67d7860172b8667dae4814efd6ea18b1d" Sep 11 08:49:50.240682 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:49:50.240666 2572 scope.go:117] "RemoveContainer" containerID="a81122664fd28d3688224602fe9b32d67d7860172b8667dae4814efd6ea18b1d" Sep 11 08:49:50.240929 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:49:50.240913 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a81122664fd28d3688224602fe9b32d67d7860172b8667dae4814efd6ea18b1d\": container with ID starting with a81122664fd28d3688224602fe9b32d67d7860172b8667dae4814efd6ea18b1d not found: ID does not exist" containerID="a81122664fd28d3688224602fe9b32d67d7860172b8667dae4814efd6ea18b1d" Sep 11 08:49:50.240969 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:49:50.240936 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a81122664fd28d3688224602fe9b32d67d7860172b8667dae4814efd6ea18b1d"} err="failed to get container status \"a81122664fd28d3688224602fe9b32d67d7860172b8667dae4814efd6ea18b1d\": rpc error: code = NotFound desc = could not find container \"a81122664fd28d3688224602fe9b32d67d7860172b8667dae4814efd6ea18b1d\": container with ID starting with a81122664fd28d3688224602fe9b32d67d7860172b8667dae4814efd6ea18b1d not found: ID does not exist" Sep 11 08:49:50.252819 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:49:50.252787 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-64bf664cf5-w4xqt"] Sep 11 08:49:50.258529 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:49:50.258504 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-64bf664cf5-w4xqt"] Sep 11 08:49:52.251423 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:49:52.251388 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="223b0e88-dc9c-464f-ae62-eab365dc7abd" path="/var/lib/kubelet/pods/223b0e88-dc9c-464f-ae62-eab365dc7abd/volumes" Sep 11 08:50:48.170690 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:50:48.170619 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7dbc8d665-fsk26"] Sep 11 08:50:48.171157 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:50:48.170938 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="223b0e88-dc9c-464f-ae62-eab365dc7abd" containerName="console" Sep 11 08:50:48.171157 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:50:48.170951 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="223b0e88-dc9c-464f-ae62-eab365dc7abd" containerName="console" Sep 11 08:50:48.171157 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:50:48.171014 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="223b0e88-dc9c-464f-ae62-eab365dc7abd" containerName="console" Sep 11 08:50:48.173921 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:50:48.173901 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7dbc8d665-fsk26" Sep 11 08:50:48.184186 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:50:48.184162 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7dbc8d665-fsk26"] Sep 11 08:50:48.215176 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:50:48.214261 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bfb07a08-7eb3-45ab-bd1f-bfafd18ce2f8-console-serving-cert\") pod \"console-7dbc8d665-fsk26\" (UID: \"bfb07a08-7eb3-45ab-bd1f-bfafd18ce2f8\") " pod="openshift-console/console-7dbc8d665-fsk26" Sep 11 08:50:48.215176 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:50:48.214308 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqrwn\" (UniqueName: \"kubernetes.io/projected/bfb07a08-7eb3-45ab-bd1f-bfafd18ce2f8-kube-api-access-dqrwn\") pod \"console-7dbc8d665-fsk26\" (UID: \"bfb07a08-7eb3-45ab-bd1f-bfafd18ce2f8\") " pod="openshift-console/console-7dbc8d665-fsk26" Sep 11 08:50:48.215176 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:50:48.214342 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bfb07a08-7eb3-45ab-bd1f-bfafd18ce2f8-trusted-ca-bundle\") pod \"console-7dbc8d665-fsk26\" (UID: \"bfb07a08-7eb3-45ab-bd1f-bfafd18ce2f8\") " pod="openshift-console/console-7dbc8d665-fsk26" Sep 11 08:50:48.215176 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:50:48.214430 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bfb07a08-7eb3-45ab-bd1f-bfafd18ce2f8-console-oauth-config\") pod \"console-7dbc8d665-fsk26\" (UID: \"bfb07a08-7eb3-45ab-bd1f-bfafd18ce2f8\") " pod="openshift-console/console-7dbc8d665-fsk26" Sep 11 08:50:48.215176 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:50:48.214481 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bfb07a08-7eb3-45ab-bd1f-bfafd18ce2f8-console-config\") pod \"console-7dbc8d665-fsk26\" (UID: \"bfb07a08-7eb3-45ab-bd1f-bfafd18ce2f8\") " pod="openshift-console/console-7dbc8d665-fsk26" Sep 11 08:50:48.215176 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:50:48.214504 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bfb07a08-7eb3-45ab-bd1f-bfafd18ce2f8-service-ca\") pod \"console-7dbc8d665-fsk26\" (UID: \"bfb07a08-7eb3-45ab-bd1f-bfafd18ce2f8\") " pod="openshift-console/console-7dbc8d665-fsk26" Sep 11 08:50:48.215176 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:50:48.214526 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bfb07a08-7eb3-45ab-bd1f-bfafd18ce2f8-oauth-serving-cert\") pod \"console-7dbc8d665-fsk26\" (UID: \"bfb07a08-7eb3-45ab-bd1f-bfafd18ce2f8\") " pod="openshift-console/console-7dbc8d665-fsk26" Sep 11 08:50:48.315339 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:50:48.315309 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bfb07a08-7eb3-45ab-bd1f-bfafd18ce2f8-console-config\") pod \"console-7dbc8d665-fsk26\" (UID: \"bfb07a08-7eb3-45ab-bd1f-bfafd18ce2f8\") " pod="openshift-console/console-7dbc8d665-fsk26" Sep 11 08:50:48.315339 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:50:48.315337 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bfb07a08-7eb3-45ab-bd1f-bfafd18ce2f8-service-ca\") pod \"console-7dbc8d665-fsk26\" (UID: \"bfb07a08-7eb3-45ab-bd1f-bfafd18ce2f8\") " pod="openshift-console/console-7dbc8d665-fsk26" Sep 11 08:50:48.315558 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:50:48.315354 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bfb07a08-7eb3-45ab-bd1f-bfafd18ce2f8-oauth-serving-cert\") pod \"console-7dbc8d665-fsk26\" (UID: \"bfb07a08-7eb3-45ab-bd1f-bfafd18ce2f8\") " pod="openshift-console/console-7dbc8d665-fsk26" Sep 11 08:50:48.315558 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:50:48.315372 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bfb07a08-7eb3-45ab-bd1f-bfafd18ce2f8-console-serving-cert\") pod \"console-7dbc8d665-fsk26\" (UID: \"bfb07a08-7eb3-45ab-bd1f-bfafd18ce2f8\") " pod="openshift-console/console-7dbc8d665-fsk26" Sep 11 08:50:48.315558 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:50:48.315393 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqrwn\" (UniqueName: \"kubernetes.io/projected/bfb07a08-7eb3-45ab-bd1f-bfafd18ce2f8-kube-api-access-dqrwn\") pod \"console-7dbc8d665-fsk26\" (UID: \"bfb07a08-7eb3-45ab-bd1f-bfafd18ce2f8\") " pod="openshift-console/console-7dbc8d665-fsk26" Sep 11 08:50:48.315558 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:50:48.315429 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bfb07a08-7eb3-45ab-bd1f-bfafd18ce2f8-trusted-ca-bundle\") pod \"console-7dbc8d665-fsk26\" (UID: \"bfb07a08-7eb3-45ab-bd1f-bfafd18ce2f8\") " pod="openshift-console/console-7dbc8d665-fsk26" Sep 11 08:50:48.315558 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:50:48.315483 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bfb07a08-7eb3-45ab-bd1f-bfafd18ce2f8-console-oauth-config\") pod \"console-7dbc8d665-fsk26\" (UID: \"bfb07a08-7eb3-45ab-bd1f-bfafd18ce2f8\") " pod="openshift-console/console-7dbc8d665-fsk26" Sep 11 08:50:48.316117 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:50:48.316090 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bfb07a08-7eb3-45ab-bd1f-bfafd18ce2f8-service-ca\") pod \"console-7dbc8d665-fsk26\" (UID: \"bfb07a08-7eb3-45ab-bd1f-bfafd18ce2f8\") " pod="openshift-console/console-7dbc8d665-fsk26" Sep 11 08:50:48.316261 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:50:48.316170 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bfb07a08-7eb3-45ab-bd1f-bfafd18ce2f8-oauth-serving-cert\") pod \"console-7dbc8d665-fsk26\" (UID: \"bfb07a08-7eb3-45ab-bd1f-bfafd18ce2f8\") " pod="openshift-console/console-7dbc8d665-fsk26" Sep 11 08:50:48.316261 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:50:48.316170 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bfb07a08-7eb3-45ab-bd1f-bfafd18ce2f8-console-config\") pod \"console-7dbc8d665-fsk26\" (UID: \"bfb07a08-7eb3-45ab-bd1f-bfafd18ce2f8\") " pod="openshift-console/console-7dbc8d665-fsk26" Sep 11 08:50:48.316400 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:50:48.316372 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bfb07a08-7eb3-45ab-bd1f-bfafd18ce2f8-trusted-ca-bundle\") pod \"console-7dbc8d665-fsk26\" (UID: \"bfb07a08-7eb3-45ab-bd1f-bfafd18ce2f8\") " pod="openshift-console/console-7dbc8d665-fsk26" Sep 11 08:50:48.318090 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:50:48.318072 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bfb07a08-7eb3-45ab-bd1f-bfafd18ce2f8-console-serving-cert\") pod \"console-7dbc8d665-fsk26\" (UID: \"bfb07a08-7eb3-45ab-bd1f-bfafd18ce2f8\") " pod="openshift-console/console-7dbc8d665-fsk26" Sep 11 08:50:48.318158 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:50:48.318097 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bfb07a08-7eb3-45ab-bd1f-bfafd18ce2f8-console-oauth-config\") pod \"console-7dbc8d665-fsk26\" (UID: \"bfb07a08-7eb3-45ab-bd1f-bfafd18ce2f8\") " pod="openshift-console/console-7dbc8d665-fsk26" Sep 11 08:50:48.324085 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:50:48.324066 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqrwn\" (UniqueName: \"kubernetes.io/projected/bfb07a08-7eb3-45ab-bd1f-bfafd18ce2f8-kube-api-access-dqrwn\") pod \"console-7dbc8d665-fsk26\" (UID: \"bfb07a08-7eb3-45ab-bd1f-bfafd18ce2f8\") " pod="openshift-console/console-7dbc8d665-fsk26" Sep 11 08:50:48.483914 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:50:48.483886 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7dbc8d665-fsk26" Sep 11 08:50:48.757633 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:50:48.757450 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7dbc8d665-fsk26"] Sep 11 08:50:48.760544 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:50:48.760507 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfb07a08_7eb3_45ab_bd1f_bfafd18ce2f8.slice/crio-146aa7633cab900c5e4c2b178e890f583a9c9880215461ee91bc68af2d2394fa WatchSource:0}: Error finding container 146aa7633cab900c5e4c2b178e890f583a9c9880215461ee91bc68af2d2394fa: Status 404 returned error can't find the container with id 146aa7633cab900c5e4c2b178e890f583a9c9880215461ee91bc68af2d2394fa Sep 11 08:50:49.463058 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:50:49.463010 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7dbc8d665-fsk26" event={"ID":"bfb07a08-7eb3-45ab-bd1f-bfafd18ce2f8","Type":"ContainerStarted","Data":"693a13b90b1bf8edcaec1e94e701b229f7ca0186f46141ce3bcec4220be1238b"} Sep 11 08:50:49.463547 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:50:49.463094 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7dbc8d665-fsk26" event={"ID":"bfb07a08-7eb3-45ab-bd1f-bfafd18ce2f8","Type":"ContainerStarted","Data":"146aa7633cab900c5e4c2b178e890f583a9c9880215461ee91bc68af2d2394fa"} Sep 11 08:50:49.480917 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:50:49.480863 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7dbc8d665-fsk26" podStartSLOduration=1.480846267 podStartE2EDuration="1.480846267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 08:50:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 08:50:49.478854559 +0000 UTC m=+269.753430266" watchObservedRunningTime="2026-09-11 08:50:49.480846267 +0000 UTC m=+269.755422010" Sep 11 08:50:58.484351 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:50:58.484312 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7dbc8d665-fsk26" Sep 11 08:50:58.484836 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:50:58.484604 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7dbc8d665-fsk26" Sep 11 08:50:58.489041 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:50:58.489021 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7dbc8d665-fsk26" Sep 11 08:50:58.502317 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:50:58.502293 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7dbc8d665-fsk26" Sep 11 08:50:58.546818 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:50:58.546784 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-65dd8997c9-5s7kb"] Sep 11 08:51:20.153302 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:51:20.153274 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-xjs9g_546705a6-1aaf-49fe-9f7f-3fc9e566437b/console-operator/2.log" Sep 11 08:51:20.155701 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:51:20.155676 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-xjs9g_546705a6-1aaf-49fe-9f7f-3fc9e566437b/console-operator/2.log" Sep 11 08:51:20.169313 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:51:20.169294 2572 kubelet.go:1648] "Image garbage collection succeeded" Sep 11 08:51:23.568054 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:51:23.567994 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-65dd8997c9-5s7kb" podUID="6bfb1765-e529-43b8-952a-41e30698841a" containerName="console" containerID="cri-o://820a344ffa61938d0fd6a17ed2b02a959875412c1a9488e198ba3a9a46efd2e1" gracePeriod=15 Sep 11 08:51:23.811174 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:51:23.811144 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65dd8997c9-5s7kb" Sep 11 08:51:23.912570 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:51:23.912497 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6bfb1765-e529-43b8-952a-41e30698841a-oauth-serving-cert\") pod \"6bfb1765-e529-43b8-952a-41e30698841a\" (UID: \"6bfb1765-e529-43b8-952a-41e30698841a\") " Sep 11 08:51:23.912570 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:51:23.912545 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6bfb1765-e529-43b8-952a-41e30698841a-console-config\") pod \"6bfb1765-e529-43b8-952a-41e30698841a\" (UID: \"6bfb1765-e529-43b8-952a-41e30698841a\") " Sep 11 08:51:23.912570 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:51:23.912566 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fb2vr\" (UniqueName: \"kubernetes.io/projected/6bfb1765-e529-43b8-952a-41e30698841a-kube-api-access-fb2vr\") pod \"6bfb1765-e529-43b8-952a-41e30698841a\" (UID: \"6bfb1765-e529-43b8-952a-41e30698841a\") " Sep 11 08:51:23.912801 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:51:23.912585 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6bfb1765-e529-43b8-952a-41e30698841a-console-oauth-config\") pod \"6bfb1765-e529-43b8-952a-41e30698841a\" (UID: \"6bfb1765-e529-43b8-952a-41e30698841a\") " Sep 11 08:51:23.912801 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:51:23.912602 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6bfb1765-e529-43b8-952a-41e30698841a-trusted-ca-bundle\") pod \"6bfb1765-e529-43b8-952a-41e30698841a\" (UID: \"6bfb1765-e529-43b8-952a-41e30698841a\") " Sep 11 08:51:23.912801 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:51:23.912640 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6bfb1765-e529-43b8-952a-41e30698841a-console-serving-cert\") pod \"6bfb1765-e529-43b8-952a-41e30698841a\" (UID: \"6bfb1765-e529-43b8-952a-41e30698841a\") " Sep 11 08:51:23.912801 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:51:23.912688 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6bfb1765-e529-43b8-952a-41e30698841a-service-ca\") pod \"6bfb1765-e529-43b8-952a-41e30698841a\" (UID: \"6bfb1765-e529-43b8-952a-41e30698841a\") " Sep 11 08:51:23.913057 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:51:23.912928 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bfb1765-e529-43b8-952a-41e30698841a-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "6bfb1765-e529-43b8-952a-41e30698841a" (UID: "6bfb1765-e529-43b8-952a-41e30698841a"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 08:51:23.913188 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:51:23.913139 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bfb1765-e529-43b8-952a-41e30698841a-service-ca" (OuterVolumeSpecName: "service-ca") pod "6bfb1765-e529-43b8-952a-41e30698841a" (UID: "6bfb1765-e529-43b8-952a-41e30698841a"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 08:51:23.913188 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:51:23.913166 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bfb1765-e529-43b8-952a-41e30698841a-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "6bfb1765-e529-43b8-952a-41e30698841a" (UID: "6bfb1765-e529-43b8-952a-41e30698841a"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 08:51:23.913503 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:51:23.913251 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bfb1765-e529-43b8-952a-41e30698841a-console-config" (OuterVolumeSpecName: "console-config") pod "6bfb1765-e529-43b8-952a-41e30698841a" (UID: "6bfb1765-e529-43b8-952a-41e30698841a"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 08:51:23.914912 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:51:23.914887 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bfb1765-e529-43b8-952a-41e30698841a-kube-api-access-fb2vr" (OuterVolumeSpecName: "kube-api-access-fb2vr") pod "6bfb1765-e529-43b8-952a-41e30698841a" (UID: "6bfb1765-e529-43b8-952a-41e30698841a"). InnerVolumeSpecName "kube-api-access-fb2vr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 08:51:23.914912 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:51:23.914895 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6bfb1765-e529-43b8-952a-41e30698841a-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "6bfb1765-e529-43b8-952a-41e30698841a" (UID: "6bfb1765-e529-43b8-952a-41e30698841a"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 08:51:23.915029 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:51:23.914961 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6bfb1765-e529-43b8-952a-41e30698841a-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "6bfb1765-e529-43b8-952a-41e30698841a" (UID: "6bfb1765-e529-43b8-952a-41e30698841a"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 08:51:24.013567 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:51:24.013518 2572 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6bfb1765-e529-43b8-952a-41e30698841a-oauth-serving-cert\") on node \"ip-10-0-128-166.ec2.internal\" DevicePath \"\"" Sep 11 08:51:24.013567 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:51:24.013561 2572 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6bfb1765-e529-43b8-952a-41e30698841a-console-config\") on node \"ip-10-0-128-166.ec2.internal\" DevicePath \"\"" Sep 11 08:51:24.013567 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:51:24.013575 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fb2vr\" (UniqueName: \"kubernetes.io/projected/6bfb1765-e529-43b8-952a-41e30698841a-kube-api-access-fb2vr\") on node \"ip-10-0-128-166.ec2.internal\" DevicePath \"\"" Sep 11 08:51:24.013798 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:51:24.013588 2572 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6bfb1765-e529-43b8-952a-41e30698841a-console-oauth-config\") on node \"ip-10-0-128-166.ec2.internal\" DevicePath \"\"" Sep 11 08:51:24.013798 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:51:24.013600 2572 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6bfb1765-e529-43b8-952a-41e30698841a-trusted-ca-bundle\") on node \"ip-10-0-128-166.ec2.internal\" DevicePath \"\"" Sep 11 08:51:24.013798 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:51:24.013611 2572 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6bfb1765-e529-43b8-952a-41e30698841a-console-serving-cert\") on node \"ip-10-0-128-166.ec2.internal\" DevicePath \"\"" Sep 11 08:51:24.013798 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:51:24.013623 2572 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6bfb1765-e529-43b8-952a-41e30698841a-service-ca\") on node \"ip-10-0-128-166.ec2.internal\" DevicePath \"\"" Sep 11 08:51:24.599308 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:51:24.599205 2572 generic.go:356] "Generic (PLEG): container finished" podID="6bfb1765-e529-43b8-952a-41e30698841a" containerID="820a344ffa61938d0fd6a17ed2b02a959875412c1a9488e198ba3a9a46efd2e1" exitCode=0 Sep 11 08:51:24.599308 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:51:24.599264 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65dd8997c9-5s7kb" Sep 11 08:51:24.599308 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:51:24.599291 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65dd8997c9-5s7kb" event={"ID":"6bfb1765-e529-43b8-952a-41e30698841a","Type":"ContainerDied","Data":"820a344ffa61938d0fd6a17ed2b02a959875412c1a9488e198ba3a9a46efd2e1"} Sep 11 08:51:24.599778 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:51:24.599344 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65dd8997c9-5s7kb" event={"ID":"6bfb1765-e529-43b8-952a-41e30698841a","Type":"ContainerDied","Data":"6e0a8bc0f6e8329ee28d706af20e76dd7874be082d4094d1e5cd3a3d42f1845a"} Sep 11 08:51:24.599778 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:51:24.599372 2572 scope.go:117] "RemoveContainer" containerID="820a344ffa61938d0fd6a17ed2b02a959875412c1a9488e198ba3a9a46efd2e1" Sep 11 08:51:24.607753 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:51:24.607729 2572 scope.go:117] "RemoveContainer" containerID="820a344ffa61938d0fd6a17ed2b02a959875412c1a9488e198ba3a9a46efd2e1" Sep 11 08:51:24.608040 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:51:24.608016 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"820a344ffa61938d0fd6a17ed2b02a959875412c1a9488e198ba3a9a46efd2e1\": container with ID starting with 820a344ffa61938d0fd6a17ed2b02a959875412c1a9488e198ba3a9a46efd2e1 not found: ID does not exist" containerID="820a344ffa61938d0fd6a17ed2b02a959875412c1a9488e198ba3a9a46efd2e1" Sep 11 08:51:24.608094 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:51:24.608050 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"820a344ffa61938d0fd6a17ed2b02a959875412c1a9488e198ba3a9a46efd2e1"} err="failed to get container status \"820a344ffa61938d0fd6a17ed2b02a959875412c1a9488e198ba3a9a46efd2e1\": rpc error: code = NotFound desc = could not find container \"820a344ffa61938d0fd6a17ed2b02a959875412c1a9488e198ba3a9a46efd2e1\": container with ID starting with 820a344ffa61938d0fd6a17ed2b02a959875412c1a9488e198ba3a9a46efd2e1 not found: ID does not exist" Sep 11 08:51:24.613760 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:51:24.613738 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-65dd8997c9-5s7kb"] Sep 11 08:51:24.617393 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:51:24.617371 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-65dd8997c9-5s7kb"] Sep 11 08:51:26.252919 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:51:26.252886 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6bfb1765-e529-43b8-952a-41e30698841a" path="/var/lib/kubelet/pods/6bfb1765-e529-43b8-952a-41e30698841a/volumes" Sep 11 08:52:40.953407 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:52:40.953377 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["feast-operator-system/feast-operator-controller-manager-58764ff88f-954hx"] Sep 11 08:52:40.953838 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:52:40.953671 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6bfb1765-e529-43b8-952a-41e30698841a" containerName="console" Sep 11 08:52:40.953838 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:52:40.953681 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="6bfb1765-e529-43b8-952a-41e30698841a" containerName="console" Sep 11 08:52:40.953838 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:52:40.953729 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="6bfb1765-e529-43b8-952a-41e30698841a" containerName="console" Sep 11 08:52:40.957723 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:52:40.957707 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-58764ff88f-954hx" Sep 11 08:52:40.959653 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:52:40.959621 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"feast-operator-system\"/\"kube-root-ca.crt\"" Sep 11 08:52:40.959770 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:52:40.959672 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"feast-operator-system\"/\"openshift-service-ca.crt\"" Sep 11 08:52:40.959770 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:52:40.959726 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"feast-operator-system\"/\"feast-operator-controller-manager-dockercfg-dkl85\"" Sep 11 08:52:40.965850 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:52:40.965830 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-58764ff88f-954hx"] Sep 11 08:52:40.997144 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:52:40.997120 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frkwn\" (UniqueName: \"kubernetes.io/projected/5dc2a815-4d2a-4c3b-aa18-735eb0a414cf-kube-api-access-frkwn\") pod \"feast-operator-controller-manager-58764ff88f-954hx\" (UID: \"5dc2a815-4d2a-4c3b-aa18-735eb0a414cf\") " pod="feast-operator-system/feast-operator-controller-manager-58764ff88f-954hx" Sep 11 08:52:41.097549 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:52:41.097515 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frkwn\" (UniqueName: \"kubernetes.io/projected/5dc2a815-4d2a-4c3b-aa18-735eb0a414cf-kube-api-access-frkwn\") pod \"feast-operator-controller-manager-58764ff88f-954hx\" (UID: \"5dc2a815-4d2a-4c3b-aa18-735eb0a414cf\") " pod="feast-operator-system/feast-operator-controller-manager-58764ff88f-954hx" Sep 11 08:52:41.105282 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:52:41.105260 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-frkwn\" (UniqueName: \"kubernetes.io/projected/5dc2a815-4d2a-4c3b-aa18-735eb0a414cf-kube-api-access-frkwn\") pod \"feast-operator-controller-manager-58764ff88f-954hx\" (UID: \"5dc2a815-4d2a-4c3b-aa18-735eb0a414cf\") " pod="feast-operator-system/feast-operator-controller-manager-58764ff88f-954hx" Sep 11 08:52:41.268742 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:52:41.268666 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-58764ff88f-954hx" Sep 11 08:52:41.535299 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:52:41.535274 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-58764ff88f-954hx"] Sep 11 08:52:41.537119 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:52:41.537093 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dc2a815_4d2a_4c3b_aa18_735eb0a414cf.slice/crio-d62810db6259609f4eee8e644020b4c101a686bc51988e18fb1423ec51c0c084 WatchSource:0}: Error finding container d62810db6259609f4eee8e644020b4c101a686bc51988e18fb1423ec51c0c084: Status 404 returned error can't find the container with id d62810db6259609f4eee8e644020b4c101a686bc51988e18fb1423ec51c0c084 Sep 11 08:52:41.539020 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:52:41.539004 2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 08:52:41.894081 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:52:41.893992 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-58764ff88f-954hx" event={"ID":"5dc2a815-4d2a-4c3b-aa18-735eb0a414cf","Type":"ContainerStarted","Data":"d62810db6259609f4eee8e644020b4c101a686bc51988e18fb1423ec51c0c084"} Sep 11 08:52:44.907982 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:52:44.907950 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-58764ff88f-954hx" event={"ID":"5dc2a815-4d2a-4c3b-aa18-735eb0a414cf","Type":"ContainerStarted","Data":"012012b2abdf56e1a76506234c309b9d9a8521b194263d9800215c7cb161a759"} Sep 11 08:52:44.908390 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:52:44.908073 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="feast-operator-system/feast-operator-controller-manager-58764ff88f-954hx" Sep 11 08:52:44.927476 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:52:44.927288 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="feast-operator-system/feast-operator-controller-manager-58764ff88f-954hx" podStartSLOduration=2.495932668 podStartE2EDuration="4.927269417s" totalImagesPullingTime="2.431336749s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 08:52:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 08:52:44.926631815 +0000 UTC m=+385.201207509" watchObservedRunningTime="2026-09-11 08:52:44.927269417 +0000 UTC m=+385.201845114" Sep 11 08:52:55.914159 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:52:55.914125 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="feast-operator-system/feast-operator-controller-manager-58764ff88f-954hx" Sep 11 08:52:59.136531 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:52:59.136501 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-65cff9fbff-n9cpf"] Sep 11 08:52:59.140192 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:52:59.140167 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65cff9fbff-n9cpf" Sep 11 08:52:59.149279 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:52:59.149254 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-65cff9fbff-n9cpf"] Sep 11 08:52:59.236403 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:52:59.236370 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2980cbe4-c160-4a9f-98ba-df1066e695be-console-config\") pod \"console-65cff9fbff-n9cpf\" (UID: \"2980cbe4-c160-4a9f-98ba-df1066e695be\") " pod="openshift-console/console-65cff9fbff-n9cpf" Sep 11 08:52:59.236553 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:52:59.236408 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2980cbe4-c160-4a9f-98ba-df1066e695be-console-serving-cert\") pod \"console-65cff9fbff-n9cpf\" (UID: \"2980cbe4-c160-4a9f-98ba-df1066e695be\") " pod="openshift-console/console-65cff9fbff-n9cpf" Sep 11 08:52:59.236553 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:52:59.236437 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fkdx\" (UniqueName: \"kubernetes.io/projected/2980cbe4-c160-4a9f-98ba-df1066e695be-kube-api-access-2fkdx\") pod \"console-65cff9fbff-n9cpf\" (UID: \"2980cbe4-c160-4a9f-98ba-df1066e695be\") " pod="openshift-console/console-65cff9fbff-n9cpf" Sep 11 08:52:59.236553 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:52:59.236485 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2980cbe4-c160-4a9f-98ba-df1066e695be-service-ca\") pod \"console-65cff9fbff-n9cpf\" (UID: \"2980cbe4-c160-4a9f-98ba-df1066e695be\") " pod="openshift-console/console-65cff9fbff-n9cpf" Sep 11 08:52:59.236655 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:52:59.236555 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2980cbe4-c160-4a9f-98ba-df1066e695be-console-oauth-config\") pod \"console-65cff9fbff-n9cpf\" (UID: \"2980cbe4-c160-4a9f-98ba-df1066e695be\") " pod="openshift-console/console-65cff9fbff-n9cpf" Sep 11 08:52:59.236655 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:52:59.236573 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2980cbe4-c160-4a9f-98ba-df1066e695be-trusted-ca-bundle\") pod \"console-65cff9fbff-n9cpf\" (UID: \"2980cbe4-c160-4a9f-98ba-df1066e695be\") " pod="openshift-console/console-65cff9fbff-n9cpf" Sep 11 08:52:59.236655 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:52:59.236596 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2980cbe4-c160-4a9f-98ba-df1066e695be-oauth-serving-cert\") pod \"console-65cff9fbff-n9cpf\" (UID: \"2980cbe4-c160-4a9f-98ba-df1066e695be\") " pod="openshift-console/console-65cff9fbff-n9cpf" Sep 11 08:52:59.337500 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:52:59.337427 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2980cbe4-c160-4a9f-98ba-df1066e695be-console-oauth-config\") pod \"console-65cff9fbff-n9cpf\" (UID: \"2980cbe4-c160-4a9f-98ba-df1066e695be\") " pod="openshift-console/console-65cff9fbff-n9cpf" Sep 11 08:52:59.337500 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:52:59.337456 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2980cbe4-c160-4a9f-98ba-df1066e695be-trusted-ca-bundle\") pod \"console-65cff9fbff-n9cpf\" (UID: \"2980cbe4-c160-4a9f-98ba-df1066e695be\") " pod="openshift-console/console-65cff9fbff-n9cpf" Sep 11 08:52:59.337500 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:52:59.337471 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2980cbe4-c160-4a9f-98ba-df1066e695be-oauth-serving-cert\") pod \"console-65cff9fbff-n9cpf\" (UID: \"2980cbe4-c160-4a9f-98ba-df1066e695be\") " pod="openshift-console/console-65cff9fbff-n9cpf" Sep 11 08:52:59.337767 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:52:59.337514 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2980cbe4-c160-4a9f-98ba-df1066e695be-console-config\") pod \"console-65cff9fbff-n9cpf\" (UID: \"2980cbe4-c160-4a9f-98ba-df1066e695be\") " pod="openshift-console/console-65cff9fbff-n9cpf" Sep 11 08:52:59.337767 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:52:59.337545 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2980cbe4-c160-4a9f-98ba-df1066e695be-console-serving-cert\") pod \"console-65cff9fbff-n9cpf\" (UID: \"2980cbe4-c160-4a9f-98ba-df1066e695be\") " pod="openshift-console/console-65cff9fbff-n9cpf" Sep 11 08:52:59.337767 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:52:59.337576 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fkdx\" (UniqueName: \"kubernetes.io/projected/2980cbe4-c160-4a9f-98ba-df1066e695be-kube-api-access-2fkdx\") pod \"console-65cff9fbff-n9cpf\" (UID: \"2980cbe4-c160-4a9f-98ba-df1066e695be\") " pod="openshift-console/console-65cff9fbff-n9cpf" Sep 11 08:52:59.337767 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:52:59.337611 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2980cbe4-c160-4a9f-98ba-df1066e695be-service-ca\") pod \"console-65cff9fbff-n9cpf\" (UID: \"2980cbe4-c160-4a9f-98ba-df1066e695be\") " pod="openshift-console/console-65cff9fbff-n9cpf" Sep 11 08:52:59.338315 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:52:59.338293 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2980cbe4-c160-4a9f-98ba-df1066e695be-oauth-serving-cert\") pod \"console-65cff9fbff-n9cpf\" (UID: \"2980cbe4-c160-4a9f-98ba-df1066e695be\") " pod="openshift-console/console-65cff9fbff-n9cpf" Sep 11 08:52:59.338315 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:52:59.338302 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2980cbe4-c160-4a9f-98ba-df1066e695be-service-ca\") pod \"console-65cff9fbff-n9cpf\" (UID: \"2980cbe4-c160-4a9f-98ba-df1066e695be\") " pod="openshift-console/console-65cff9fbff-n9cpf" Sep 11 08:52:59.338546 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:52:59.338401 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2980cbe4-c160-4a9f-98ba-df1066e695be-console-config\") pod \"console-65cff9fbff-n9cpf\" (UID: \"2980cbe4-c160-4a9f-98ba-df1066e695be\") " pod="openshift-console/console-65cff9fbff-n9cpf" Sep 11 08:52:59.338635 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:52:59.338610 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2980cbe4-c160-4a9f-98ba-df1066e695be-trusted-ca-bundle\") pod \"console-65cff9fbff-n9cpf\" (UID: \"2980cbe4-c160-4a9f-98ba-df1066e695be\") " pod="openshift-console/console-65cff9fbff-n9cpf" Sep 11 08:52:59.340032 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:52:59.340012 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2980cbe4-c160-4a9f-98ba-df1066e695be-console-oauth-config\") pod \"console-65cff9fbff-n9cpf\" (UID: \"2980cbe4-c160-4a9f-98ba-df1066e695be\") " pod="openshift-console/console-65cff9fbff-n9cpf" Sep 11 08:52:59.340260 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:52:59.340218 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2980cbe4-c160-4a9f-98ba-df1066e695be-console-serving-cert\") pod \"console-65cff9fbff-n9cpf\" (UID: \"2980cbe4-c160-4a9f-98ba-df1066e695be\") " pod="openshift-console/console-65cff9fbff-n9cpf" Sep 11 08:52:59.344897 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:52:59.344875 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fkdx\" (UniqueName: \"kubernetes.io/projected/2980cbe4-c160-4a9f-98ba-df1066e695be-kube-api-access-2fkdx\") pod \"console-65cff9fbff-n9cpf\" (UID: \"2980cbe4-c160-4a9f-98ba-df1066e695be\") " pod="openshift-console/console-65cff9fbff-n9cpf" Sep 11 08:52:59.450284 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:52:59.450250 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65cff9fbff-n9cpf" Sep 11 08:52:59.718495 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:52:59.718443 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-65cff9fbff-n9cpf"] Sep 11 08:52:59.720658 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:52:59.720624 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2980cbe4_c160_4a9f_98ba_df1066e695be.slice/crio-3d2f62639d67516681b6d659305d9c9d7825b2bfae6f680fc1fa0d6ad17a2f7a WatchSource:0}: Error finding container 3d2f62639d67516681b6d659305d9c9d7825b2bfae6f680fc1fa0d6ad17a2f7a: Status 404 returned error can't find the container with id 3d2f62639d67516681b6d659305d9c9d7825b2bfae6f680fc1fa0d6ad17a2f7a Sep 11 08:52:59.973835 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:52:59.973755 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65cff9fbff-n9cpf" event={"ID":"2980cbe4-c160-4a9f-98ba-df1066e695be","Type":"ContainerStarted","Data":"4af5414dbd1bb0cb4c2d8e5c8971df0670df6f7c3ffe85ee119277155a58f414"} Sep 11 08:52:59.974002 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:52:59.973843 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65cff9fbff-n9cpf" event={"ID":"2980cbe4-c160-4a9f-98ba-df1066e695be","Type":"ContainerStarted","Data":"3d2f62639d67516681b6d659305d9c9d7825b2bfae6f680fc1fa0d6ad17a2f7a"} Sep 11 08:52:59.991206 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:52:59.991160 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-65cff9fbff-n9cpf" podStartSLOduration=0.991145791 podStartE2EDuration="991.145791ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 08:52:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 08:52:59.989507696 +0000 UTC m=+400.264083403" watchObservedRunningTime="2026-09-11 08:52:59.991145791 +0000 UTC m=+400.265721486" Sep 11 08:53:09.451299 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:53:09.451265 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-65cff9fbff-n9cpf" Sep 11 08:53:09.451750 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:53:09.451344 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-65cff9fbff-n9cpf" Sep 11 08:53:09.456026 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:53:09.456004 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-65cff9fbff-n9cpf" Sep 11 08:53:10.014476 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:53:10.014442 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-65cff9fbff-n9cpf" Sep 11 08:53:10.064493 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:53:10.064463 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7dbc8d665-fsk26"] Sep 11 08:53:18.881685 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:53:18.881648 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-79d9f545c-m2vv2"] Sep 11 08:53:18.890464 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:53:18.890441 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-m2vv2" Sep 11 08:53:18.892592 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:53:18.892568 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-online-tls\"" Sep 11 08:53:18.892714 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:53:18.892694 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-intra-comm\"" Sep 11 08:53:18.892790 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:53:18.892620 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-registry-tls\"" Sep 11 08:53:18.892790 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:53:18.892644 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-lpppm\"" Sep 11 08:53:18.892906 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:53:18.892646 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\"" Sep 11 08:53:18.892906 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:53:18.892624 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-offline-tls\"" Sep 11 08:53:18.892906 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:53:18.892669 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\"" Sep 11 08:53:18.892906 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:53:18.892660 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-ui-tls\"" Sep 11 08:53:18.897476 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:53:18.897452 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-79d9f545c-m2vv2"] Sep 11 08:53:18.996780 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:53:18.996748 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/4007c1a5-f195-449d-a4fd-f008a1d9f10f-feast-data\") pod \"feast-simple-feast-setup-79d9f545c-m2vv2\" (UID: \"4007c1a5-f195-449d-a4fd-f008a1d9f10f\") " pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-m2vv2" Sep 11 08:53:18.996780 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:53:18.996780 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjlkk\" (UniqueName: \"kubernetes.io/projected/4007c1a5-f195-449d-a4fd-f008a1d9f10f-kube-api-access-wjlkk\") pod \"feast-simple-feast-setup-79d9f545c-m2vv2\" (UID: \"4007c1a5-f195-449d-a4fd-f008a1d9f10f\") " pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-m2vv2" Sep 11 08:53:18.996939 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:53:18.996816 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/4007c1a5-f195-449d-a4fd-f008a1d9f10f-online-tls\") pod \"feast-simple-feast-setup-79d9f545c-m2vv2\" (UID: \"4007c1a5-f195-449d-a4fd-f008a1d9f10f\") " pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-m2vv2" Sep 11 08:53:18.996939 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:53:18.996919 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/4007c1a5-f195-449d-a4fd-f008a1d9f10f-registry-tls\") pod \"feast-simple-feast-setup-79d9f545c-m2vv2\" (UID: \"4007c1a5-f195-449d-a4fd-f008a1d9f10f\") " pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-m2vv2" Sep 11 08:53:18.997009 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:53:18.996945 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/4007c1a5-f195-449d-a4fd-f008a1d9f10f-ui-tls\") pod \"feast-simple-feast-setup-79d9f545c-m2vv2\" (UID: \"4007c1a5-f195-449d-a4fd-f008a1d9f10f\") " pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-m2vv2" Sep 11 08:53:18.997009 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:53:18.996988 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/4007c1a5-f195-449d-a4fd-f008a1d9f10f-offline-tls\") pod \"feast-simple-feast-setup-79d9f545c-m2vv2\" (UID: \"4007c1a5-f195-449d-a4fd-f008a1d9f10f\") " pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-m2vv2" Sep 11 08:53:19.098270 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:53:19.098241 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/4007c1a5-f195-449d-a4fd-f008a1d9f10f-offline-tls\") pod \"feast-simple-feast-setup-79d9f545c-m2vv2\" (UID: \"4007c1a5-f195-449d-a4fd-f008a1d9f10f\") " pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-m2vv2" Sep 11 08:53:19.098412 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:53:19.098278 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/4007c1a5-f195-449d-a4fd-f008a1d9f10f-feast-data\") pod \"feast-simple-feast-setup-79d9f545c-m2vv2\" (UID: \"4007c1a5-f195-449d-a4fd-f008a1d9f10f\") " pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-m2vv2" Sep 11 08:53:19.098412 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:53:19.098297 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjlkk\" (UniqueName: \"kubernetes.io/projected/4007c1a5-f195-449d-a4fd-f008a1d9f10f-kube-api-access-wjlkk\") pod \"feast-simple-feast-setup-79d9f545c-m2vv2\" (UID: \"4007c1a5-f195-449d-a4fd-f008a1d9f10f\") " pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-m2vv2" Sep 11 08:53:19.098412 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:53:19.098322 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/4007c1a5-f195-449d-a4fd-f008a1d9f10f-online-tls\") pod \"feast-simple-feast-setup-79d9f545c-m2vv2\" (UID: \"4007c1a5-f195-449d-a4fd-f008a1d9f10f\") " pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-m2vv2" Sep 11 08:53:19.098412 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:53:19.098359 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/4007c1a5-f195-449d-a4fd-f008a1d9f10f-registry-tls\") pod \"feast-simple-feast-setup-79d9f545c-m2vv2\" (UID: \"4007c1a5-f195-449d-a4fd-f008a1d9f10f\") " pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-m2vv2" Sep 11 08:53:19.098412 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:53:19.098376 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/4007c1a5-f195-449d-a4fd-f008a1d9f10f-ui-tls\") pod \"feast-simple-feast-setup-79d9f545c-m2vv2\" (UID: \"4007c1a5-f195-449d-a4fd-f008a1d9f10f\") " pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-m2vv2" Sep 11 08:53:19.098412 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:53:19.098378 2572 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-offline-tls: secret "feast-simple-feast-setup-offline-tls" not found Sep 11 08:53:19.098677 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:53:19.098444 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4007c1a5-f195-449d-a4fd-f008a1d9f10f-offline-tls podName:4007c1a5-f195-449d-a4fd-f008a1d9f10f nodeName:}" failed. No retries permitted until 2026-09-11 08:53:19.598422817 +0000 UTC m=+419.872998491 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/4007c1a5-f195-449d-a4fd-f008a1d9f10f-offline-tls") pod "feast-simple-feast-setup-79d9f545c-m2vv2" (UID: "4007c1a5-f195-449d-a4fd-f008a1d9f10f") : secret "feast-simple-feast-setup-offline-tls" not found Sep 11 08:53:19.098677 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:53:19.098462 2572 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-ui-tls: secret "feast-simple-feast-setup-ui-tls" not found Sep 11 08:53:19.098677 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:53:19.098473 2572 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-online-tls: secret "feast-simple-feast-setup-online-tls" not found Sep 11 08:53:19.098677 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:53:19.098518 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4007c1a5-f195-449d-a4fd-f008a1d9f10f-ui-tls podName:4007c1a5-f195-449d-a4fd-f008a1d9f10f nodeName:}" failed. No retries permitted until 2026-09-11 08:53:19.598502562 +0000 UTC m=+419.873078238 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "ui-tls" (UniqueName: "kubernetes.io/secret/4007c1a5-f195-449d-a4fd-f008a1d9f10f-ui-tls") pod "feast-simple-feast-setup-79d9f545c-m2vv2" (UID: "4007c1a5-f195-449d-a4fd-f008a1d9f10f") : secret "feast-simple-feast-setup-ui-tls" not found Sep 11 08:53:19.098677 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:53:19.098531 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4007c1a5-f195-449d-a4fd-f008a1d9f10f-online-tls podName:4007c1a5-f195-449d-a4fd-f008a1d9f10f nodeName:}" failed. No retries permitted until 2026-09-11 08:53:19.598525417 +0000 UTC m=+419.873101090 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/4007c1a5-f195-449d-a4fd-f008a1d9f10f-online-tls") pod "feast-simple-feast-setup-79d9f545c-m2vv2" (UID: "4007c1a5-f195-449d-a4fd-f008a1d9f10f") : secret "feast-simple-feast-setup-online-tls" not found Sep 11 08:53:19.098677 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:53:19.098633 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/4007c1a5-f195-449d-a4fd-f008a1d9f10f-feast-data\") pod \"feast-simple-feast-setup-79d9f545c-m2vv2\" (UID: \"4007c1a5-f195-449d-a4fd-f008a1d9f10f\") " pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-m2vv2" Sep 11 08:53:19.101009 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:53:19.100989 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/4007c1a5-f195-449d-a4fd-f008a1d9f10f-registry-tls\") pod \"feast-simple-feast-setup-79d9f545c-m2vv2\" (UID: \"4007c1a5-f195-449d-a4fd-f008a1d9f10f\") " pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-m2vv2" Sep 11 08:53:19.107398 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:53:19.107379 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjlkk\" (UniqueName: \"kubernetes.io/projected/4007c1a5-f195-449d-a4fd-f008a1d9f10f-kube-api-access-wjlkk\") pod \"feast-simple-feast-setup-79d9f545c-m2vv2\" (UID: \"4007c1a5-f195-449d-a4fd-f008a1d9f10f\") " pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-m2vv2" Sep 11 08:53:19.603025 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:53:19.602992 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/4007c1a5-f195-449d-a4fd-f008a1d9f10f-online-tls\") pod \"feast-simple-feast-setup-79d9f545c-m2vv2\" (UID: \"4007c1a5-f195-449d-a4fd-f008a1d9f10f\") " pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-m2vv2" Sep 11 08:53:19.603196 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:53:19.603046 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/4007c1a5-f195-449d-a4fd-f008a1d9f10f-ui-tls\") pod \"feast-simple-feast-setup-79d9f545c-m2vv2\" (UID: \"4007c1a5-f195-449d-a4fd-f008a1d9f10f\") " pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-m2vv2" Sep 11 08:53:19.603196 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:53:19.603079 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/4007c1a5-f195-449d-a4fd-f008a1d9f10f-offline-tls\") pod \"feast-simple-feast-setup-79d9f545c-m2vv2\" (UID: \"4007c1a5-f195-449d-a4fd-f008a1d9f10f\") " pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-m2vv2" Sep 11 08:53:19.606106 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:53:19.606083 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/4007c1a5-f195-449d-a4fd-f008a1d9f10f-online-tls\") pod \"feast-simple-feast-setup-79d9f545c-m2vv2\" (UID: \"4007c1a5-f195-449d-a4fd-f008a1d9f10f\") " pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-m2vv2" Sep 11 08:53:19.606218 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:53:19.606167 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/4007c1a5-f195-449d-a4fd-f008a1d9f10f-ui-tls\") pod \"feast-simple-feast-setup-79d9f545c-m2vv2\" (UID: \"4007c1a5-f195-449d-a4fd-f008a1d9f10f\") " pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-m2vv2" Sep 11 08:53:19.606287 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:53:19.606242 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/4007c1a5-f195-449d-a4fd-f008a1d9f10f-offline-tls\") pod \"feast-simple-feast-setup-79d9f545c-m2vv2\" (UID: \"4007c1a5-f195-449d-a4fd-f008a1d9f10f\") " pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-m2vv2" Sep 11 08:53:19.810641 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:53:19.810604 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-m2vv2" Sep 11 08:53:20.089516 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:53:20.089492 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-79d9f545c-m2vv2"] Sep 11 08:53:20.090822 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:53:20.090796 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4007c1a5_f195_449d_a4fd_f008a1d9f10f.slice/crio-0399207d9e9357d107397154d5f676552e6d81ade0dc92a36452c5903eb5aedf WatchSource:0}: Error finding container 0399207d9e9357d107397154d5f676552e6d81ade0dc92a36452c5903eb5aedf: Status 404 returned error can't find the container with id 0399207d9e9357d107397154d5f676552e6d81ade0dc92a36452c5903eb5aedf Sep 11 08:53:21.059027 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:53:21.058980 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-m2vv2" event={"ID":"4007c1a5-f195-449d-a4fd-f008a1d9f10f","Type":"ContainerStarted","Data":"0399207d9e9357d107397154d5f676552e6d81ade0dc92a36452c5903eb5aedf"} Sep 11 08:53:35.086352 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:53:35.086269 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7dbc8d665-fsk26" podUID="bfb07a08-7eb3-45ab-bd1f-bfafd18ce2f8" containerName="console" containerID="cri-o://693a13b90b1bf8edcaec1e94e701b229f7ca0186f46141ce3bcec4220be1238b" gracePeriod=15 Sep 11 08:53:38.500194 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:53:38.500147 2572 patch_prober.go:28] interesting pod/console-7dbc8d665-fsk26 container/console namespace/openshift-console: Readiness probe status=failure output="Get \"https://10.132.0.26:8443/health\": dial tcp 10.132.0.26:8443: connect: connection refused" start-of-body= Sep 11 08:53:38.500637 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:53:38.500240 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/console-7dbc8d665-fsk26" podUID="bfb07a08-7eb3-45ab-bd1f-bfafd18ce2f8" containerName="console" probeResult="failure" output="Get \"https://10.132.0.26:8443/health\": dial tcp 10.132.0.26:8443: connect: connection refused" Sep 11 08:53:43.839411 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:53:43.839389 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7dbc8d665-fsk26" Sep 11 08:53:43.919743 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:53:43.919671 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bfb07a08-7eb3-45ab-bd1f-bfafd18ce2f8-console-oauth-config\") pod \"bfb07a08-7eb3-45ab-bd1f-bfafd18ce2f8\" (UID: \"bfb07a08-7eb3-45ab-bd1f-bfafd18ce2f8\") " Sep 11 08:53:43.919743 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:53:43.919702 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bfb07a08-7eb3-45ab-bd1f-bfafd18ce2f8-console-config\") pod \"bfb07a08-7eb3-45ab-bd1f-bfafd18ce2f8\" (UID: \"bfb07a08-7eb3-45ab-bd1f-bfafd18ce2f8\") " Sep 11 08:53:43.919743 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:53:43.919724 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dqrwn\" (UniqueName: \"kubernetes.io/projected/bfb07a08-7eb3-45ab-bd1f-bfafd18ce2f8-kube-api-access-dqrwn\") pod \"bfb07a08-7eb3-45ab-bd1f-bfafd18ce2f8\" (UID: \"bfb07a08-7eb3-45ab-bd1f-bfafd18ce2f8\") " Sep 11 08:53:43.919743 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:53:43.919744 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bfb07a08-7eb3-45ab-bd1f-bfafd18ce2f8-oauth-serving-cert\") pod \"bfb07a08-7eb3-45ab-bd1f-bfafd18ce2f8\" (UID: \"bfb07a08-7eb3-45ab-bd1f-bfafd18ce2f8\") " Sep 11 08:53:43.920041 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:53:43.919763 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bfb07a08-7eb3-45ab-bd1f-bfafd18ce2f8-service-ca\") pod \"bfb07a08-7eb3-45ab-bd1f-bfafd18ce2f8\" (UID: \"bfb07a08-7eb3-45ab-bd1f-bfafd18ce2f8\") " Sep 11 08:53:43.920041 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:53:43.919783 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bfb07a08-7eb3-45ab-bd1f-bfafd18ce2f8-console-serving-cert\") pod \"bfb07a08-7eb3-45ab-bd1f-bfafd18ce2f8\" (UID: \"bfb07a08-7eb3-45ab-bd1f-bfafd18ce2f8\") " Sep 11 08:53:43.920041 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:53:43.919805 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bfb07a08-7eb3-45ab-bd1f-bfafd18ce2f8-trusted-ca-bundle\") pod \"bfb07a08-7eb3-45ab-bd1f-bfafd18ce2f8\" (UID: \"bfb07a08-7eb3-45ab-bd1f-bfafd18ce2f8\") " Sep 11 08:53:43.920185 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:53:43.920116 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfb07a08-7eb3-45ab-bd1f-bfafd18ce2f8-console-config" (OuterVolumeSpecName: "console-config") pod "bfb07a08-7eb3-45ab-bd1f-bfafd18ce2f8" (UID: "bfb07a08-7eb3-45ab-bd1f-bfafd18ce2f8"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 08:53:43.920185 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:53:43.920170 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfb07a08-7eb3-45ab-bd1f-bfafd18ce2f8-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "bfb07a08-7eb3-45ab-bd1f-bfafd18ce2f8" (UID: "bfb07a08-7eb3-45ab-bd1f-bfafd18ce2f8"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 08:53:43.920341 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:53:43.920273 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfb07a08-7eb3-45ab-bd1f-bfafd18ce2f8-service-ca" (OuterVolumeSpecName: "service-ca") pod "bfb07a08-7eb3-45ab-bd1f-bfafd18ce2f8" (UID: "bfb07a08-7eb3-45ab-bd1f-bfafd18ce2f8"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 08:53:43.920341 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:53:43.920294 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfb07a08-7eb3-45ab-bd1f-bfafd18ce2f8-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "bfb07a08-7eb3-45ab-bd1f-bfafd18ce2f8" (UID: "bfb07a08-7eb3-45ab-bd1f-bfafd18ce2f8"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 08:53:43.921985 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:53:43.921962 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfb07a08-7eb3-45ab-bd1f-bfafd18ce2f8-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "bfb07a08-7eb3-45ab-bd1f-bfafd18ce2f8" (UID: "bfb07a08-7eb3-45ab-bd1f-bfafd18ce2f8"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 08:53:43.922161 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:53:43.922138 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfb07a08-7eb3-45ab-bd1f-bfafd18ce2f8-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "bfb07a08-7eb3-45ab-bd1f-bfafd18ce2f8" (UID: "bfb07a08-7eb3-45ab-bd1f-bfafd18ce2f8"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 08:53:43.922344 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:53:43.922325 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfb07a08-7eb3-45ab-bd1f-bfafd18ce2f8-kube-api-access-dqrwn" (OuterVolumeSpecName: "kube-api-access-dqrwn") pod "bfb07a08-7eb3-45ab-bd1f-bfafd18ce2f8" (UID: "bfb07a08-7eb3-45ab-bd1f-bfafd18ce2f8"). InnerVolumeSpecName "kube-api-access-dqrwn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 08:53:44.021137 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:53:44.021108 2572 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bfb07a08-7eb3-45ab-bd1f-bfafd18ce2f8-console-oauth-config\") on node \"ip-10-0-128-166.ec2.internal\" DevicePath \"\"" Sep 11 08:53:44.021137 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:53:44.021132 2572 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bfb07a08-7eb3-45ab-bd1f-bfafd18ce2f8-console-config\") on node \"ip-10-0-128-166.ec2.internal\" DevicePath \"\"" Sep 11 08:53:44.021137 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:53:44.021142 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqrwn\" (UniqueName: \"kubernetes.io/projected/bfb07a08-7eb3-45ab-bd1f-bfafd18ce2f8-kube-api-access-dqrwn\") on node \"ip-10-0-128-166.ec2.internal\" DevicePath \"\"" Sep 11 08:53:44.021348 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:53:44.021151 2572 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bfb07a08-7eb3-45ab-bd1f-bfafd18ce2f8-oauth-serving-cert\") on node \"ip-10-0-128-166.ec2.internal\" DevicePath \"\"" Sep 11 08:53:44.021348 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:53:44.021159 2572 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bfb07a08-7eb3-45ab-bd1f-bfafd18ce2f8-service-ca\") on node \"ip-10-0-128-166.ec2.internal\" DevicePath \"\"" Sep 11 08:53:44.021348 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:53:44.021167 2572 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bfb07a08-7eb3-45ab-bd1f-bfafd18ce2f8-console-serving-cert\") on node \"ip-10-0-128-166.ec2.internal\" DevicePath \"\"" Sep 11 08:53:44.021348 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:53:44.021177 2572 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bfb07a08-7eb3-45ab-bd1f-bfafd18ce2f8-trusted-ca-bundle\") on node \"ip-10-0-128-166.ec2.internal\" DevicePath \"\"" Sep 11 08:53:44.166320 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:53:44.166283 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-m2vv2" event={"ID":"4007c1a5-f195-449d-a4fd-f008a1d9f10f","Type":"ContainerStarted","Data":"2bb8d71d7d0614ab973bf5875b66176141a97ba0588798a9d8834f56d4cb11d0"} Sep 11 08:53:44.167682 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:53:44.167647 2572 generic.go:356] "Generic (PLEG): container finished" podID="bfb07a08-7eb3-45ab-bd1f-bfafd18ce2f8" containerID="693a13b90b1bf8edcaec1e94e701b229f7ca0186f46141ce3bcec4220be1238b" exitCode=0 Sep 11 08:53:44.167803 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:53:44.167685 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7dbc8d665-fsk26" Sep 11 08:53:44.167803 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:53:44.167691 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7dbc8d665-fsk26" event={"ID":"bfb07a08-7eb3-45ab-bd1f-bfafd18ce2f8","Type":"ContainerDied","Data":"693a13b90b1bf8edcaec1e94e701b229f7ca0186f46141ce3bcec4220be1238b"} Sep 11 08:53:44.167803 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:53:44.167734 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7dbc8d665-fsk26" event={"ID":"bfb07a08-7eb3-45ab-bd1f-bfafd18ce2f8","Type":"ContainerDied","Data":"146aa7633cab900c5e4c2b178e890f583a9c9880215461ee91bc68af2d2394fa"} Sep 11 08:53:44.167803 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:53:44.167761 2572 scope.go:117] "RemoveContainer" containerID="693a13b90b1bf8edcaec1e94e701b229f7ca0186f46141ce3bcec4220be1238b" Sep 11 08:53:44.177336 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:53:44.177318 2572 scope.go:117] "RemoveContainer" containerID="693a13b90b1bf8edcaec1e94e701b229f7ca0186f46141ce3bcec4220be1238b" Sep 11 08:53:44.177654 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:53:44.177631 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"693a13b90b1bf8edcaec1e94e701b229f7ca0186f46141ce3bcec4220be1238b\": container with ID starting with 693a13b90b1bf8edcaec1e94e701b229f7ca0186f46141ce3bcec4220be1238b not found: ID does not exist" containerID="693a13b90b1bf8edcaec1e94e701b229f7ca0186f46141ce3bcec4220be1238b" Sep 11 08:53:44.177722 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:53:44.177666 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"693a13b90b1bf8edcaec1e94e701b229f7ca0186f46141ce3bcec4220be1238b"} err="failed to get container status \"693a13b90b1bf8edcaec1e94e701b229f7ca0186f46141ce3bcec4220be1238b\": rpc error: code = NotFound desc = could not find container \"693a13b90b1bf8edcaec1e94e701b229f7ca0186f46141ce3bcec4220be1238b\": container with ID starting with 693a13b90b1bf8edcaec1e94e701b229f7ca0186f46141ce3bcec4220be1238b not found: ID does not exist" Sep 11 08:53:44.201786 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:53:44.201758 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7dbc8d665-fsk26"] Sep 11 08:53:44.208252 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:53:44.208206 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7dbc8d665-fsk26"] Sep 11 08:53:44.252489 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:53:44.252460 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bfb07a08-7eb3-45ab-bd1f-bfafd18ce2f8" path="/var/lib/kubelet/pods/bfb07a08-7eb3-45ab-bd1f-bfafd18ce2f8/volumes" Sep 11 08:53:49.190578 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:53:49.190535 2572 generic.go:356] "Generic (PLEG): container finished" podID="4007c1a5-f195-449d-a4fd-f008a1d9f10f" containerID="2bb8d71d7d0614ab973bf5875b66176141a97ba0588798a9d8834f56d4cb11d0" exitCode=0 Sep 11 08:53:49.191000 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:53:49.190615 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-m2vv2" event={"ID":"4007c1a5-f195-449d-a4fd-f008a1d9f10f","Type":"ContainerDied","Data":"2bb8d71d7d0614ab973bf5875b66176141a97ba0588798a9d8834f56d4cb11d0"} Sep 11 08:53:50.195905 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:53:50.195872 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-m2vv2" event={"ID":"4007c1a5-f195-449d-a4fd-f008a1d9f10f","Type":"ContainerStarted","Data":"19bea6a445857c37b2f2532720c59e25a5db7e6364f2111e8fb08536a9cf646d"} Sep 11 08:53:55.218578 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:53:55.218542 2572 generic.go:356] "Generic (PLEG): container finished" podID="4007c1a5-f195-449d-a4fd-f008a1d9f10f" containerID="19bea6a445857c37b2f2532720c59e25a5db7e6364f2111e8fb08536a9cf646d" exitCode=0 Sep 11 08:53:55.218978 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:53:55.218611 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-m2vv2" event={"ID":"4007c1a5-f195-449d-a4fd-f008a1d9f10f","Type":"ContainerDied","Data":"19bea6a445857c37b2f2532720c59e25a5db7e6364f2111e8fb08536a9cf646d"} Sep 11 08:53:55.227149 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:53:55.227132 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-intra-comm\"" Sep 11 08:53:56.235189 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:53:56.235134 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-m2vv2" event={"ID":"4007c1a5-f195-449d-a4fd-f008a1d9f10f","Type":"ContainerStarted","Data":"b99c72727fba9f43a1f52cc9d6baddf707784fe95b5ddce899ec8e749b052427"} Sep 11 08:53:56.235666 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:53:56.235209 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-m2vv2" event={"ID":"4007c1a5-f195-449d-a4fd-f008a1d9f10f","Type":"ContainerStarted","Data":"7e547a0ae3fd7bf76ac9c2b4f276238cbdeb06c03396ef3f924339df39a38167"} Sep 11 08:53:56.235666 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:53:56.235260 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-m2vv2" event={"ID":"4007c1a5-f195-449d-a4fd-f008a1d9f10f","Type":"ContainerStarted","Data":"61f9b7753f7e4ee8b9788378232df4e3ddf3b99cd4e9088547300514a15dde6a"} Sep 11 08:53:56.235666 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:53:56.235280 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-m2vv2" event={"ID":"4007c1a5-f195-449d-a4fd-f008a1d9f10f","Type":"ContainerStarted","Data":"4ca60992fd73ba43c3b4b153be993ad5ef2d238a146ceb618db39496ff5e1b21"} Sep 11 08:53:56.260261 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:53:56.260119 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-m2vv2" podStartSLOduration=14.622386292 podStartE2EDuration="38.260092702s" totalImagesPullingTime="23.63770641s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 08:53:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 08:53:56.257660479 +0000 UTC m=+456.532236186" watchObservedRunningTime="2026-09-11 08:53:56.260092702 +0000 UTC m=+456.534668413" Sep 11 08:53:58.811363 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:53:58.811195 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-m2vv2" Sep 11 08:53:58.811363 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:53:58.811280 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-m2vv2" Sep 11 08:53:58.811363 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:53:58.811300 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-m2vv2" Sep 11 08:53:58.811363 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:53:58.811321 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-m2vv2" Sep 11 08:53:58.813065 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:53:58.813024 2572 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-m2vv2" podUID="4007c1a5-f195-449d-a4fd-f008a1d9f10f" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.29:8443: connect: connection refused" Sep 11 08:53:58.813267 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:53:58.813046 2572 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-m2vv2" podUID="4007c1a5-f195-449d-a4fd-f008a1d9f10f" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.29:8816: connect: connection refused" Sep 11 08:53:58.813401 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:53:58.813069 2572 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-m2vv2" podUID="4007c1a5-f195-449d-a4fd-f008a1d9f10f" containerName="online" probeResult="failure" output="Get \"https://10.132.0.29:6567/health\": dial tcp 10.132.0.29:6567: connect: connection refused" Sep 11 08:53:58.813401 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:53:58.813085 2572 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-m2vv2" podUID="4007c1a5-f195-449d-a4fd-f008a1d9f10f" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.29:6571: connect: connection refused" Sep 11 08:53:59.811534 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:53:59.811496 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-m2vv2" Sep 11 08:53:59.812100 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:53:59.812074 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-m2vv2" Sep 11 08:53:59.812269 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:53:59.812112 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-m2vv2" Sep 11 08:53:59.812269 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:53:59.812132 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-m2vv2" Sep 11 08:54:01.811257 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:54:01.811187 2572 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-m2vv2" podUID="4007c1a5-f195-449d-a4fd-f008a1d9f10f" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.29:8443: connect: connection refused" Sep 11 08:54:01.811852 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:54:01.811242 2572 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-m2vv2" podUID="4007c1a5-f195-449d-a4fd-f008a1d9f10f" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.29:8816: connect: connection refused" Sep 11 08:54:01.812069 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:54:01.811241 2572 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-m2vv2" podUID="4007c1a5-f195-449d-a4fd-f008a1d9f10f" containerName="online" probeResult="failure" output="Get \"https://10.132.0.29:6567/health\": dial tcp 10.132.0.29:6567: connect: connection refused" Sep 11 08:54:01.812181 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:54:01.811266 2572 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-m2vv2" podUID="4007c1a5-f195-449d-a4fd-f008a1d9f10f" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.29:6571: connect: connection refused" Sep 11 08:54:04.812601 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:54:04.812546 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-m2vv2" Sep 11 08:54:04.813089 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:54:04.812719 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-m2vv2" Sep 11 08:54:04.813089 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:54:04.812845 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-m2vv2" Sep 11 08:54:04.813558 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:54:04.813532 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-m2vv2" Sep 11 08:54:04.813708 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:54:04.813685 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-m2vv2" Sep 11 08:54:04.813800 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:54:04.813781 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-m2vv2" Sep 11 08:54:04.818015 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:54:04.817999 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-m2vv2" Sep 11 08:54:05.286241 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:54:05.286201 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-m2vv2" Sep 11 08:55:07.157099 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:55:07.157059 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-79d9f545c-m2vv2"] Sep 11 08:55:07.157676 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:55:07.157412 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-m2vv2" podUID="4007c1a5-f195-449d-a4fd-f008a1d9f10f" containerName="registry" containerID="cri-o://4ca60992fd73ba43c3b4b153be993ad5ef2d238a146ceb618db39496ff5e1b21" gracePeriod=30 Sep 11 08:55:07.157749 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:55:07.157668 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-m2vv2" podUID="4007c1a5-f195-449d-a4fd-f008a1d9f10f" containerName="ui" containerID="cri-o://b99c72727fba9f43a1f52cc9d6baddf707784fe95b5ddce899ec8e749b052427" gracePeriod=30 Sep 11 08:55:07.157802 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:55:07.157759 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-m2vv2" podUID="4007c1a5-f195-449d-a4fd-f008a1d9f10f" containerName="offline" containerID="cri-o://7e547a0ae3fd7bf76ac9c2b4f276238cbdeb06c03396ef3f924339df39a38167" gracePeriod=30 Sep 11 08:55:07.157856 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:55:07.157833 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-m2vv2" podUID="4007c1a5-f195-449d-a4fd-f008a1d9f10f" containerName="online" containerID="cri-o://61f9b7753f7e4ee8b9788378232df4e3ddf3b99cd4e9088547300514a15dde6a" gracePeriod=30 Sep 11 08:55:07.525064 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:55:07.525021 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-79d9f545c-wtnvv"] Sep 11 08:55:07.525409 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:55:07.525365 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfb07a08-7eb3-45ab-bd1f-bfafd18ce2f8" containerName="console" Sep 11 08:55:07.525409 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:55:07.525409 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfb07a08-7eb3-45ab-bd1f-bfafd18ce2f8" containerName="console" Sep 11 08:55:07.525554 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:55:07.525472 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="bfb07a08-7eb3-45ab-bd1f-bfafd18ce2f8" containerName="console" Sep 11 08:55:07.528950 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:55:07.528926 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-wtnvv" Sep 11 08:55:07.530879 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:55:07.530856 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-pqgfh\"" Sep 11 08:55:07.540325 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:55:07.540299 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-79d9f545c-wtnvv"] Sep 11 08:55:07.690654 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:55:07.690618 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/83c2cfc4-c365-4ca4-884c-014de928568f-feast-data\") pod \"feast-simple-feast-setup-79d9f545c-wtnvv\" (UID: \"83c2cfc4-c365-4ca4-884c-014de928568f\") " pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-wtnvv" Sep 11 08:55:07.690654 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:55:07.690653 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/83c2cfc4-c365-4ca4-884c-014de928568f-offline-tls\") pod \"feast-simple-feast-setup-79d9f545c-wtnvv\" (UID: \"83c2cfc4-c365-4ca4-884c-014de928568f\") " pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-wtnvv" Sep 11 08:55:07.690917 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:55:07.690669 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/83c2cfc4-c365-4ca4-884c-014de928568f-online-tls\") pod \"feast-simple-feast-setup-79d9f545c-wtnvv\" (UID: \"83c2cfc4-c365-4ca4-884c-014de928568f\") " pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-wtnvv" Sep 11 08:55:07.690917 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:55:07.690706 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/83c2cfc4-c365-4ca4-884c-014de928568f-registry-tls\") pod \"feast-simple-feast-setup-79d9f545c-wtnvv\" (UID: \"83c2cfc4-c365-4ca4-884c-014de928568f\") " pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-wtnvv" Sep 11 08:55:07.690917 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:55:07.690837 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5zmn\" (UniqueName: \"kubernetes.io/projected/83c2cfc4-c365-4ca4-884c-014de928568f-kube-api-access-z5zmn\") pod \"feast-simple-feast-setup-79d9f545c-wtnvv\" (UID: \"83c2cfc4-c365-4ca4-884c-014de928568f\") " pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-wtnvv" Sep 11 08:55:07.690917 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:55:07.690885 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/83c2cfc4-c365-4ca4-884c-014de928568f-ui-tls\") pod \"feast-simple-feast-setup-79d9f545c-wtnvv\" (UID: \"83c2cfc4-c365-4ca4-884c-014de928568f\") " pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-wtnvv" Sep 11 08:55:07.792046 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:55:07.791949 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/83c2cfc4-c365-4ca4-884c-014de928568f-feast-data\") pod \"feast-simple-feast-setup-79d9f545c-wtnvv\" (UID: \"83c2cfc4-c365-4ca4-884c-014de928568f\") " pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-wtnvv" Sep 11 08:55:07.792046 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:55:07.791984 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/83c2cfc4-c365-4ca4-884c-014de928568f-offline-tls\") pod \"feast-simple-feast-setup-79d9f545c-wtnvv\" (UID: \"83c2cfc4-c365-4ca4-884c-014de928568f\") " pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-wtnvv" Sep 11 08:55:07.792046 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:55:07.792002 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/83c2cfc4-c365-4ca4-884c-014de928568f-online-tls\") pod \"feast-simple-feast-setup-79d9f545c-wtnvv\" (UID: \"83c2cfc4-c365-4ca4-884c-014de928568f\") " pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-wtnvv" Sep 11 08:55:07.792046 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:55:07.792038 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/83c2cfc4-c365-4ca4-884c-014de928568f-registry-tls\") pod \"feast-simple-feast-setup-79d9f545c-wtnvv\" (UID: \"83c2cfc4-c365-4ca4-884c-014de928568f\") " pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-wtnvv" Sep 11 08:55:07.792403 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:55:07.792133 2572 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-online-tls: secret "feast-simple-feast-setup-online-tls" not found Sep 11 08:55:07.792403 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:55:07.792264 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83c2cfc4-c365-4ca4-884c-014de928568f-online-tls podName:83c2cfc4-c365-4ca4-884c-014de928568f nodeName:}" failed. No retries permitted until 2026-09-11 08:55:08.292188752 +0000 UTC m=+528.566764427 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/83c2cfc4-c365-4ca4-884c-014de928568f-online-tls") pod "feast-simple-feast-setup-79d9f545c-wtnvv" (UID: "83c2cfc4-c365-4ca4-884c-014de928568f") : secret "feast-simple-feast-setup-online-tls" not found Sep 11 08:55:07.792403 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:55:07.792286 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5zmn\" (UniqueName: \"kubernetes.io/projected/83c2cfc4-c365-4ca4-884c-014de928568f-kube-api-access-z5zmn\") pod \"feast-simple-feast-setup-79d9f545c-wtnvv\" (UID: \"83c2cfc4-c365-4ca4-884c-014de928568f\") " pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-wtnvv" Sep 11 08:55:07.792403 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:55:07.792321 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/83c2cfc4-c365-4ca4-884c-014de928568f-ui-tls\") pod \"feast-simple-feast-setup-79d9f545c-wtnvv\" (UID: \"83c2cfc4-c365-4ca4-884c-014de928568f\") " pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-wtnvv" Sep 11 08:55:07.792592 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:55:07.792442 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/83c2cfc4-c365-4ca4-884c-014de928568f-feast-data\") pod \"feast-simple-feast-setup-79d9f545c-wtnvv\" (UID: \"83c2cfc4-c365-4ca4-884c-014de928568f\") " pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-wtnvv" Sep 11 08:55:07.794879 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:55:07.794848 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/83c2cfc4-c365-4ca4-884c-014de928568f-ui-tls\") pod \"feast-simple-feast-setup-79d9f545c-wtnvv\" (UID: \"83c2cfc4-c365-4ca4-884c-014de928568f\") " pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-wtnvv" Sep 11 08:55:07.795030 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:55:07.795009 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/83c2cfc4-c365-4ca4-884c-014de928568f-registry-tls\") pod \"feast-simple-feast-setup-79d9f545c-wtnvv\" (UID: \"83c2cfc4-c365-4ca4-884c-014de928568f\") " pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-wtnvv" Sep 11 08:55:07.795182 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:55:07.795163 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/83c2cfc4-c365-4ca4-884c-014de928568f-offline-tls\") pod \"feast-simple-feast-setup-79d9f545c-wtnvv\" (UID: \"83c2cfc4-c365-4ca4-884c-014de928568f\") " pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-wtnvv" Sep 11 08:55:07.802141 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:55:07.802115 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5zmn\" (UniqueName: \"kubernetes.io/projected/83c2cfc4-c365-4ca4-884c-014de928568f-kube-api-access-z5zmn\") pod \"feast-simple-feast-setup-79d9f545c-wtnvv\" (UID: \"83c2cfc4-c365-4ca4-884c-014de928568f\") " pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-wtnvv" Sep 11 08:55:08.297373 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:55:08.297343 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/83c2cfc4-c365-4ca4-884c-014de928568f-online-tls\") pod \"feast-simple-feast-setup-79d9f545c-wtnvv\" (UID: \"83c2cfc4-c365-4ca4-884c-014de928568f\") " pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-wtnvv" Sep 11 08:55:08.299918 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:55:08.299874 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/83c2cfc4-c365-4ca4-884c-014de928568f-online-tls\") pod \"feast-simple-feast-setup-79d9f545c-wtnvv\" (UID: \"83c2cfc4-c365-4ca4-884c-014de928568f\") " pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-wtnvv" Sep 11 08:55:08.438975 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:55:08.438940 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-wtnvv" Sep 11 08:55:08.531258 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:55:08.531210 2572 generic.go:356] "Generic (PLEG): container finished" podID="4007c1a5-f195-449d-a4fd-f008a1d9f10f" containerID="b99c72727fba9f43a1f52cc9d6baddf707784fe95b5ddce899ec8e749b052427" exitCode=0 Sep 11 08:55:08.531258 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:55:08.531253 2572 generic.go:356] "Generic (PLEG): container finished" podID="4007c1a5-f195-449d-a4fd-f008a1d9f10f" containerID="61f9b7753f7e4ee8b9788378232df4e3ddf3b99cd4e9088547300514a15dde6a" exitCode=0 Sep 11 08:55:08.531258 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:55:08.531258 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-m2vv2" event={"ID":"4007c1a5-f195-449d-a4fd-f008a1d9f10f","Type":"ContainerDied","Data":"b99c72727fba9f43a1f52cc9d6baddf707784fe95b5ddce899ec8e749b052427"} Sep 11 08:55:08.531466 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:55:08.531303 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-m2vv2" event={"ID":"4007c1a5-f195-449d-a4fd-f008a1d9f10f","Type":"ContainerDied","Data":"61f9b7753f7e4ee8b9788378232df4e3ddf3b99cd4e9088547300514a15dde6a"} Sep 11 08:55:08.713248 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:55:08.713211 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-79d9f545c-wtnvv"] Sep 11 08:55:08.714575 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:55:08.714547 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod83c2cfc4_c365_4ca4_884c_014de928568f.slice/crio-3ead0556c5f1481710976395d9865804caa621ed409e50cc885ec8a2a9b3fe9a WatchSource:0}: Error finding container 3ead0556c5f1481710976395d9865804caa621ed409e50cc885ec8a2a9b3fe9a: Status 404 returned error can't find the container with id 3ead0556c5f1481710976395d9865804caa621ed409e50cc885ec8a2a9b3fe9a Sep 11 08:55:09.537428 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:55:09.537398 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-wtnvv" event={"ID":"83c2cfc4-c365-4ca4-884c-014de928568f","Type":"ContainerStarted","Data":"baddd8757743006abced413a448ccd5200f06a1108ce7fb55cb6d7d27cc7c438"} Sep 11 08:55:09.537428 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:55:09.537440 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-wtnvv" event={"ID":"83c2cfc4-c365-4ca4-884c-014de928568f","Type":"ContainerStarted","Data":"3ead0556c5f1481710976395d9865804caa621ed409e50cc885ec8a2a9b3fe9a"} Sep 11 08:55:13.554020 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:55:13.553989 2572 generic.go:356] "Generic (PLEG): container finished" podID="83c2cfc4-c365-4ca4-884c-014de928568f" containerID="baddd8757743006abced413a448ccd5200f06a1108ce7fb55cb6d7d27cc7c438" exitCode=0 Sep 11 08:55:13.554410 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:55:13.554066 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-wtnvv" event={"ID":"83c2cfc4-c365-4ca4-884c-014de928568f","Type":"ContainerDied","Data":"baddd8757743006abced413a448ccd5200f06a1108ce7fb55cb6d7d27cc7c438"} Sep 11 08:55:14.559825 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:55:14.559792 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-wtnvv" event={"ID":"83c2cfc4-c365-4ca4-884c-014de928568f","Type":"ContainerStarted","Data":"8fa23ba46ec7eb09c69e03b6073ec3d2793c180b84cdac4992f5cf60941eba3e"} Sep 11 08:55:14.814327 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:55:14.814203 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-m2vv2" podUID="4007c1a5-f195-449d-a4fd-f008a1d9f10f" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.29:8443: connect: connection refused" Sep 11 08:55:15.283951 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:55:15.283906 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-m2vv2" podUID="4007c1a5-f195-449d-a4fd-f008a1d9f10f" containerName="online" probeResult="failure" output="Get \"https://10.132.0.29:6567/health\": dial tcp 10.132.0.29:6567: connect: connection refused" Sep 11 08:55:19.587314 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:55:19.587205 2572 generic.go:356] "Generic (PLEG): container finished" podID="83c2cfc4-c365-4ca4-884c-014de928568f" containerID="8fa23ba46ec7eb09c69e03b6073ec3d2793c180b84cdac4992f5cf60941eba3e" exitCode=0 Sep 11 08:55:19.587314 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:55:19.587282 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-wtnvv" event={"ID":"83c2cfc4-c365-4ca4-884c-014de928568f","Type":"ContainerDied","Data":"8fa23ba46ec7eb09c69e03b6073ec3d2793c180b84cdac4992f5cf60941eba3e"} Sep 11 08:55:20.598479 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:55:20.598438 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-wtnvv" event={"ID":"83c2cfc4-c365-4ca4-884c-014de928568f","Type":"ContainerStarted","Data":"1a524a716cf166d79c54284dfe6ea512aaf1c7abebcae5a8b4d5114694444805"} Sep 11 08:55:20.598957 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:55:20.598511 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-wtnvv" event={"ID":"83c2cfc4-c365-4ca4-884c-014de928568f","Type":"ContainerStarted","Data":"e423eab19850169d4765213500ad41b98cbdc142af9228a5419683b084899da7"} Sep 11 08:55:20.598957 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:55:20.598533 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-wtnvv" event={"ID":"83c2cfc4-c365-4ca4-884c-014de928568f","Type":"ContainerStarted","Data":"e108953a85c248e247bab59cd720c292ec9900b699d41640875920e8b60d51a9"} Sep 11 08:55:20.598957 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:55:20.598550 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-wtnvv" event={"ID":"83c2cfc4-c365-4ca4-884c-014de928568f","Type":"ContainerStarted","Data":"cda100dcc261f070b6a8ef44416d414f1ad7c69bdc569dc1c5dd323457c92729"} Sep 11 08:55:20.619964 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:55:20.619903 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-wtnvv" podStartSLOduration=13.619882063 podStartE2EDuration="13.619882063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 08:55:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 08:55:20.619640869 +0000 UTC m=+540.894216568" watchObservedRunningTime="2026-09-11 08:55:20.619882063 +0000 UTC m=+540.894457760" Sep 11 08:55:23.440127 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:55:23.440090 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-wtnvv" Sep 11 08:55:23.440722 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:55:23.440692 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-wtnvv" Sep 11 08:55:23.440722 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:55:23.440723 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-wtnvv" Sep 11 08:55:23.440906 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:55:23.440741 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-wtnvv" Sep 11 08:55:23.442491 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:55:23.442462 2572 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-wtnvv" podUID="83c2cfc4-c365-4ca4-884c-014de928568f" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.30:8816: connect: connection refused" Sep 11 08:55:23.442691 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:55:23.442487 2572 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-wtnvv" podUID="83c2cfc4-c365-4ca4-884c-014de928568f" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.30:6571: connect: connection refused" Sep 11 08:55:23.442765 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:55:23.442548 2572 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-wtnvv" podUID="83c2cfc4-c365-4ca4-884c-014de928568f" containerName="online" probeResult="failure" output="Get \"https://10.132.0.30:6567/health\": dial tcp 10.132.0.30:6567: connect: connection refused" Sep 11 08:55:23.442944 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:55:23.442914 2572 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-wtnvv" podUID="83c2cfc4-c365-4ca4-884c-014de928568f" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.30:8443: connect: connection refused" Sep 11 08:55:24.814425 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:55:24.814372 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-m2vv2" podUID="4007c1a5-f195-449d-a4fd-f008a1d9f10f" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.29:8443: connect: connection refused" Sep 11 08:55:25.283711 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:55:25.283662 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-m2vv2" podUID="4007c1a5-f195-449d-a4fd-f008a1d9f10f" containerName="online" probeResult="failure" output="Get \"https://10.132.0.29:6567/health\": dial tcp 10.132.0.29:6567: connect: connection refused" Sep 11 08:55:26.440053 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:55:26.440005 2572 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-wtnvv" podUID="83c2cfc4-c365-4ca4-884c-014de928568f" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.30:8816: connect: connection refused" Sep 11 08:55:26.440557 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:55:26.440006 2572 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-wtnvv" podUID="83c2cfc4-c365-4ca4-884c-014de928568f" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.30:8443: connect: connection refused" Sep 11 08:55:26.440557 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:55:26.440006 2572 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-wtnvv" podUID="83c2cfc4-c365-4ca4-884c-014de928568f" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.30:6571: connect: connection refused" Sep 11 08:55:26.440810 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:55:26.440731 2572 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-wtnvv" podUID="83c2cfc4-c365-4ca4-884c-014de928568f" containerName="online" probeResult="failure" output="Get \"https://10.132.0.30:6567/health\": dial tcp 10.132.0.30:6567: connect: connection refused" Sep 11 08:55:28.440039 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:55:28.440005 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-wtnvv" Sep 11 08:55:28.440427 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:55:28.440084 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-wtnvv" Sep 11 08:55:28.440427 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:55:28.440109 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-wtnvv" Sep 11 08:55:28.440427 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:55:28.440126 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-wtnvv" Sep 11 08:55:29.440209 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:55:29.440178 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-wtnvv" Sep 11 08:55:29.440646 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:55:29.440300 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-wtnvv" Sep 11 08:55:29.440646 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:55:29.440525 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-wtnvv" Sep 11 08:55:29.445727 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:55:29.445703 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-wtnvv" Sep 11 08:55:29.648450 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:55:29.648423 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-wtnvv" Sep 11 08:55:29.648640 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:55:29.648582 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-wtnvv" Sep 11 08:55:29.648640 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:55:29.648634 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-wtnvv" Sep 11 08:55:29.651127 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:55:29.651101 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-wtnvv" Sep 11 08:55:34.813618 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:55:34.813571 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-m2vv2" podUID="4007c1a5-f195-449d-a4fd-f008a1d9f10f" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.29:8443: connect: connection refused" Sep 11 08:55:34.814067 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:55:34.813738 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-m2vv2" Sep 11 08:55:35.283790 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:55:35.283755 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-m2vv2" podUID="4007c1a5-f195-449d-a4fd-f008a1d9f10f" containerName="online" probeResult="failure" output="Get \"https://10.132.0.29:6567/health\": dial tcp 10.132.0.29:6567: connect: connection refused" Sep 11 08:55:35.283964 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:55:35.283876 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-m2vv2" Sep 11 08:55:37.682062 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:55:37.682033 2572 generic.go:356] "Generic (PLEG): container finished" podID="4007c1a5-f195-449d-a4fd-f008a1d9f10f" containerID="7e547a0ae3fd7bf76ac9c2b4f276238cbdeb06c03396ef3f924339df39a38167" exitCode=137 Sep 11 08:55:37.682062 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:55:37.682058 2572 generic.go:356] "Generic (PLEG): container finished" podID="4007c1a5-f195-449d-a4fd-f008a1d9f10f" containerID="4ca60992fd73ba43c3b4b153be993ad5ef2d238a146ceb618db39496ff5e1b21" exitCode=137 Sep 11 08:55:37.682455 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:55:37.682093 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-m2vv2" event={"ID":"4007c1a5-f195-449d-a4fd-f008a1d9f10f","Type":"ContainerDied","Data":"7e547a0ae3fd7bf76ac9c2b4f276238cbdeb06c03396ef3f924339df39a38167"} Sep 11 08:55:37.682455 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:55:37.682136 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-m2vv2" event={"ID":"4007c1a5-f195-449d-a4fd-f008a1d9f10f","Type":"ContainerDied","Data":"4ca60992fd73ba43c3b4b153be993ad5ef2d238a146ceb618db39496ff5e1b21"} Sep 11 08:55:37.809741 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:55:37.809719 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-m2vv2" Sep 11 08:55:37.872422 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:55:37.872395 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/4007c1a5-f195-449d-a4fd-f008a1d9f10f-registry-tls\") pod \"4007c1a5-f195-449d-a4fd-f008a1d9f10f\" (UID: \"4007c1a5-f195-449d-a4fd-f008a1d9f10f\") " Sep 11 08:55:37.872422 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:55:37.872423 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/4007c1a5-f195-449d-a4fd-f008a1d9f10f-ui-tls\") pod \"4007c1a5-f195-449d-a4fd-f008a1d9f10f\" (UID: \"4007c1a5-f195-449d-a4fd-f008a1d9f10f\") " Sep 11 08:55:37.872621 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:55:37.872439 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/4007c1a5-f195-449d-a4fd-f008a1d9f10f-offline-tls\") pod \"4007c1a5-f195-449d-a4fd-f008a1d9f10f\" (UID: \"4007c1a5-f195-449d-a4fd-f008a1d9f10f\") " Sep 11 08:55:37.872621 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:55:37.872475 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/4007c1a5-f195-449d-a4fd-f008a1d9f10f-online-tls\") pod \"4007c1a5-f195-449d-a4fd-f008a1d9f10f\" (UID: \"4007c1a5-f195-449d-a4fd-f008a1d9f10f\") " Sep 11 08:55:37.872621 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:55:37.872531 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/4007c1a5-f195-449d-a4fd-f008a1d9f10f-feast-data\") pod \"4007c1a5-f195-449d-a4fd-f008a1d9f10f\" (UID: \"4007c1a5-f195-449d-a4fd-f008a1d9f10f\") " Sep 11 08:55:37.872621 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:55:37.872582 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wjlkk\" (UniqueName: \"kubernetes.io/projected/4007c1a5-f195-449d-a4fd-f008a1d9f10f-kube-api-access-wjlkk\") pod \"4007c1a5-f195-449d-a4fd-f008a1d9f10f\" (UID: \"4007c1a5-f195-449d-a4fd-f008a1d9f10f\") " Sep 11 08:55:37.873387 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:55:37.873327 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4007c1a5-f195-449d-a4fd-f008a1d9f10f-feast-data" (OuterVolumeSpecName: "feast-data") pod "4007c1a5-f195-449d-a4fd-f008a1d9f10f" (UID: "4007c1a5-f195-449d-a4fd-f008a1d9f10f"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 08:55:37.874896 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:55:37.874868 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4007c1a5-f195-449d-a4fd-f008a1d9f10f-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "4007c1a5-f195-449d-a4fd-f008a1d9f10f" (UID: "4007c1a5-f195-449d-a4fd-f008a1d9f10f"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 08:55:37.875024 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:55:37.874905 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4007c1a5-f195-449d-a4fd-f008a1d9f10f-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "4007c1a5-f195-449d-a4fd-f008a1d9f10f" (UID: "4007c1a5-f195-449d-a4fd-f008a1d9f10f"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 08:55:37.875024 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:55:37.874950 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4007c1a5-f195-449d-a4fd-f008a1d9f10f-online-tls" (OuterVolumeSpecName: "online-tls") pod "4007c1a5-f195-449d-a4fd-f008a1d9f10f" (UID: "4007c1a5-f195-449d-a4fd-f008a1d9f10f"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 08:55:37.875212 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:55:37.875192 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4007c1a5-f195-449d-a4fd-f008a1d9f10f-kube-api-access-wjlkk" (OuterVolumeSpecName: "kube-api-access-wjlkk") pod "4007c1a5-f195-449d-a4fd-f008a1d9f10f" (UID: "4007c1a5-f195-449d-a4fd-f008a1d9f10f"). InnerVolumeSpecName "kube-api-access-wjlkk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 08:55:37.875339 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:55:37.875318 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4007c1a5-f195-449d-a4fd-f008a1d9f10f-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "4007c1a5-f195-449d-a4fd-f008a1d9f10f" (UID: "4007c1a5-f195-449d-a4fd-f008a1d9f10f"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 08:55:37.973198 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:55:37.973176 2572 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/4007c1a5-f195-449d-a4fd-f008a1d9f10f-online-tls\") on node \"ip-10-0-128-166.ec2.internal\" DevicePath \"\"" Sep 11 08:55:37.973198 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:55:37.973196 2572 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/4007c1a5-f195-449d-a4fd-f008a1d9f10f-feast-data\") on node \"ip-10-0-128-166.ec2.internal\" DevicePath \"\"" Sep 11 08:55:37.973358 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:55:37.973206 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjlkk\" (UniqueName: \"kubernetes.io/projected/4007c1a5-f195-449d-a4fd-f008a1d9f10f-kube-api-access-wjlkk\") on node \"ip-10-0-128-166.ec2.internal\" DevicePath \"\"" Sep 11 08:55:37.973358 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:55:37.973214 2572 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/4007c1a5-f195-449d-a4fd-f008a1d9f10f-registry-tls\") on node \"ip-10-0-128-166.ec2.internal\" DevicePath \"\"" Sep 11 08:55:37.973358 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:55:37.973240 2572 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/4007c1a5-f195-449d-a4fd-f008a1d9f10f-ui-tls\") on node \"ip-10-0-128-166.ec2.internal\" DevicePath \"\"" Sep 11 08:55:37.973358 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:55:37.973248 2572 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/4007c1a5-f195-449d-a4fd-f008a1d9f10f-offline-tls\") on node \"ip-10-0-128-166.ec2.internal\" DevicePath \"\"" Sep 11 08:55:38.688747 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:55:38.688722 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-m2vv2" Sep 11 08:55:38.688747 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:55:38.688730 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-m2vv2" event={"ID":"4007c1a5-f195-449d-a4fd-f008a1d9f10f","Type":"ContainerDied","Data":"0399207d9e9357d107397154d5f676552e6d81ade0dc92a36452c5903eb5aedf"} Sep 11 08:55:38.689203 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:55:38.688786 2572 scope.go:117] "RemoveContainer" containerID="b99c72727fba9f43a1f52cc9d6baddf707784fe95b5ddce899ec8e749b052427" Sep 11 08:55:38.697326 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:55:38.697309 2572 scope.go:117] "RemoveContainer" containerID="7e547a0ae3fd7bf76ac9c2b4f276238cbdeb06c03396ef3f924339df39a38167" Sep 11 08:55:38.705086 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:55:38.705068 2572 scope.go:117] "RemoveContainer" containerID="61f9b7753f7e4ee8b9788378232df4e3ddf3b99cd4e9088547300514a15dde6a" Sep 11 08:55:38.705659 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:55:38.705638 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-79d9f545c-m2vv2"] Sep 11 08:55:38.709101 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:55:38.709081 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-79d9f545c-m2vv2"] Sep 11 08:55:38.712848 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:55:38.712827 2572 scope.go:117] "RemoveContainer" containerID="4ca60992fd73ba43c3b4b153be993ad5ef2d238a146ceb618db39496ff5e1b21" Sep 11 08:55:38.719920 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:55:38.719906 2572 scope.go:117] "RemoveContainer" containerID="19bea6a445857c37b2f2532720c59e25a5db7e6364f2111e8fb08536a9cf646d" Sep 11 08:55:38.726928 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:55:38.726914 2572 scope.go:117] "RemoveContainer" containerID="2bb8d71d7d0614ab973bf5875b66176141a97ba0588798a9d8834f56d4cb11d0" Sep 11 08:55:40.252736 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:55:40.252693 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4007c1a5-f195-449d-a4fd-f008a1d9f10f" path="/var/lib/kubelet/pods/4007c1a5-f195-449d-a4fd-f008a1d9f10f/volumes" Sep 11 08:56:20.186335 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:56:20.186310 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-xjs9g_546705a6-1aaf-49fe-9f7f-3fc9e566437b/console-operator/2.log" Sep 11 08:56:20.186858 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:56:20.186390 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-xjs9g_546705a6-1aaf-49fe-9f7f-3fc9e566437b/console-operator/2.log" Sep 11 08:56:32.489857 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:56:32.489823 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-cc6567fcd-7z8l4"] Sep 11 08:56:32.492192 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:56:32.490117 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4007c1a5-f195-449d-a4fd-f008a1d9f10f" containerName="registry" Sep 11 08:56:32.492192 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:56:32.490128 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="4007c1a5-f195-449d-a4fd-f008a1d9f10f" containerName="registry" Sep 11 08:56:32.492192 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:56:32.490142 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4007c1a5-f195-449d-a4fd-f008a1d9f10f" containerName="online" Sep 11 08:56:32.492192 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:56:32.490147 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="4007c1a5-f195-449d-a4fd-f008a1d9f10f" containerName="online" Sep 11 08:56:32.492192 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:56:32.490152 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4007c1a5-f195-449d-a4fd-f008a1d9f10f" containerName="ui" Sep 11 08:56:32.492192 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:56:32.490157 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="4007c1a5-f195-449d-a4fd-f008a1d9f10f" containerName="ui" Sep 11 08:56:32.492192 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:56:32.490168 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4007c1a5-f195-449d-a4fd-f008a1d9f10f" containerName="feast-init" Sep 11 08:56:32.492192 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:56:32.490173 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="4007c1a5-f195-449d-a4fd-f008a1d9f10f" containerName="feast-init" Sep 11 08:56:32.492192 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:56:32.490180 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4007c1a5-f195-449d-a4fd-f008a1d9f10f" containerName="feast-apply" Sep 11 08:56:32.492192 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:56:32.490185 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="4007c1a5-f195-449d-a4fd-f008a1d9f10f" containerName="feast-apply" Sep 11 08:56:32.492192 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:56:32.490190 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4007c1a5-f195-449d-a4fd-f008a1d9f10f" containerName="offline" Sep 11 08:56:32.492192 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:56:32.490195 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="4007c1a5-f195-449d-a4fd-f008a1d9f10f" containerName="offline" Sep 11 08:56:32.492192 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:56:32.490267 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="4007c1a5-f195-449d-a4fd-f008a1d9f10f" containerName="registry" Sep 11 08:56:32.492192 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:56:32.490276 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="4007c1a5-f195-449d-a4fd-f008a1d9f10f" containerName="offline" Sep 11 08:56:32.492192 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:56:32.490282 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="4007c1a5-f195-449d-a4fd-f008a1d9f10f" containerName="ui" Sep 11 08:56:32.492192 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:56:32.490288 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="4007c1a5-f195-449d-a4fd-f008a1d9f10f" containerName="online" Sep 11 08:56:32.493306 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:56:32.493291 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-cc6567fcd-7z8l4" Sep 11 08:56:32.495265 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:56:32.495244 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-ui-tls\"" Sep 11 08:56:32.495393 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:56:32.495322 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-dockercfg-j6xn5\"" Sep 11 08:56:32.495457 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:56:32.495409 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-intra-comm\"" Sep 11 08:56:32.495457 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:56:32.495418 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-offline-tls\"" Sep 11 08:56:32.495635 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:56:32.495623 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"kube-root-ca.crt\"" Sep 11 08:56:32.495870 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:56:32.495821 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"openshift-service-ca.crt\"" Sep 11 08:56:32.496045 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:56:32.496029 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-client-ca\"" Sep 11 08:56:32.496109 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:56:32.496095 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-online-tls\"" Sep 11 08:56:32.505965 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:56:32.505945 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-cc6567fcd-7z8l4"] Sep 11 08:56:32.606583 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:56:32.606551 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vhzw\" (UniqueName: \"kubernetes.io/projected/88b2e8f2-799e-4e9e-8329-70969c9f7dce-kube-api-access-6vhzw\") pod \"feast-simple-feast-remote-setup-cc6567fcd-7z8l4\" (UID: \"88b2e8f2-799e-4e9e-8329-70969c9f7dce\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-cc6567fcd-7z8l4" Sep 11 08:56:32.606583 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:56:32.606583 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/88b2e8f2-799e-4e9e-8329-70969c9f7dce-ui-tls\") pod \"feast-simple-feast-remote-setup-cc6567fcd-7z8l4\" (UID: \"88b2e8f2-799e-4e9e-8329-70969c9f7dce\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-cc6567fcd-7z8l4" Sep 11 08:56:32.606760 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:56:32.606607 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/88b2e8f2-799e-4e9e-8329-70969c9f7dce-online-tls\") pod \"feast-simple-feast-remote-setup-cc6567fcd-7z8l4\" (UID: \"88b2e8f2-799e-4e9e-8329-70969c9f7dce\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-cc6567fcd-7z8l4" Sep 11 08:56:32.606760 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:56:32.606728 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/88b2e8f2-799e-4e9e-8329-70969c9f7dce-registry-tls\") pod \"feast-simple-feast-remote-setup-cc6567fcd-7z8l4\" (UID: \"88b2e8f2-799e-4e9e-8329-70969c9f7dce\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-cc6567fcd-7z8l4" Sep 11 08:56:32.606760 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:56:32.606753 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/88b2e8f2-799e-4e9e-8329-70969c9f7dce-offline-tls\") pod \"feast-simple-feast-remote-setup-cc6567fcd-7z8l4\" (UID: \"88b2e8f2-799e-4e9e-8329-70969c9f7dce\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-cc6567fcd-7z8l4" Sep 11 08:56:32.606857 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:56:32.606770 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/88b2e8f2-799e-4e9e-8329-70969c9f7dce-feast-data\") pod \"feast-simple-feast-remote-setup-cc6567fcd-7z8l4\" (UID: \"88b2e8f2-799e-4e9e-8329-70969c9f7dce\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-cc6567fcd-7z8l4" Sep 11 08:56:32.708002 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:56:32.707969 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/88b2e8f2-799e-4e9e-8329-70969c9f7dce-registry-tls\") pod \"feast-simple-feast-remote-setup-cc6567fcd-7z8l4\" (UID: \"88b2e8f2-799e-4e9e-8329-70969c9f7dce\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-cc6567fcd-7z8l4" Sep 11 08:56:32.708002 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:56:32.708000 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/88b2e8f2-799e-4e9e-8329-70969c9f7dce-offline-tls\") pod \"feast-simple-feast-remote-setup-cc6567fcd-7z8l4\" (UID: \"88b2e8f2-799e-4e9e-8329-70969c9f7dce\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-cc6567fcd-7z8l4" Sep 11 08:56:32.708220 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:56:32.708015 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/88b2e8f2-799e-4e9e-8329-70969c9f7dce-feast-data\") pod \"feast-simple-feast-remote-setup-cc6567fcd-7z8l4\" (UID: \"88b2e8f2-799e-4e9e-8329-70969c9f7dce\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-cc6567fcd-7z8l4" Sep 11 08:56:32.708220 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:56:32.708056 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vhzw\" (UniqueName: \"kubernetes.io/projected/88b2e8f2-799e-4e9e-8329-70969c9f7dce-kube-api-access-6vhzw\") pod \"feast-simple-feast-remote-setup-cc6567fcd-7z8l4\" (UID: \"88b2e8f2-799e-4e9e-8329-70969c9f7dce\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-cc6567fcd-7z8l4" Sep 11 08:56:32.708220 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:56:32.708082 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/88b2e8f2-799e-4e9e-8329-70969c9f7dce-ui-tls\") pod \"feast-simple-feast-remote-setup-cc6567fcd-7z8l4\" (UID: \"88b2e8f2-799e-4e9e-8329-70969c9f7dce\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-cc6567fcd-7z8l4" Sep 11 08:56:32.708220 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:56:32.708120 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/88b2e8f2-799e-4e9e-8329-70969c9f7dce-online-tls\") pod \"feast-simple-feast-remote-setup-cc6567fcd-7z8l4\" (UID: \"88b2e8f2-799e-4e9e-8329-70969c9f7dce\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-cc6567fcd-7z8l4" Sep 11 08:56:32.708463 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:56:32.708272 2572 secret.go:189] Couldn't get secret test-ns-remote-registry/feast-simple-feast-remote-setup-online-tls: secret "feast-simple-feast-remote-setup-online-tls" not found Sep 11 08:56:32.708463 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:56:32.708339 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88b2e8f2-799e-4e9e-8329-70969c9f7dce-online-tls podName:88b2e8f2-799e-4e9e-8329-70969c9f7dce nodeName:}" failed. No retries permitted until 2026-09-11 08:56:33.208315627 +0000 UTC m=+613.482891302 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/88b2e8f2-799e-4e9e-8329-70969c9f7dce-online-tls") pod "feast-simple-feast-remote-setup-cc6567fcd-7z8l4" (UID: "88b2e8f2-799e-4e9e-8329-70969c9f7dce") : secret "feast-simple-feast-remote-setup-online-tls" not found Sep 11 08:56:32.708463 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:56:32.708447 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/88b2e8f2-799e-4e9e-8329-70969c9f7dce-feast-data\") pod \"feast-simple-feast-remote-setup-cc6567fcd-7z8l4\" (UID: \"88b2e8f2-799e-4e9e-8329-70969c9f7dce\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-cc6567fcd-7z8l4" Sep 11 08:56:32.708666 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:56:32.708646 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/88b2e8f2-799e-4e9e-8329-70969c9f7dce-registry-tls\") pod \"feast-simple-feast-remote-setup-cc6567fcd-7z8l4\" (UID: \"88b2e8f2-799e-4e9e-8329-70969c9f7dce\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-cc6567fcd-7z8l4" Sep 11 08:56:32.710683 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:56:32.710662 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/88b2e8f2-799e-4e9e-8329-70969c9f7dce-ui-tls\") pod \"feast-simple-feast-remote-setup-cc6567fcd-7z8l4\" (UID: \"88b2e8f2-799e-4e9e-8329-70969c9f7dce\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-cc6567fcd-7z8l4" Sep 11 08:56:32.710783 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:56:32.710705 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/88b2e8f2-799e-4e9e-8329-70969c9f7dce-offline-tls\") pod \"feast-simple-feast-remote-setup-cc6567fcd-7z8l4\" (UID: \"88b2e8f2-799e-4e9e-8329-70969c9f7dce\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-cc6567fcd-7z8l4" Sep 11 08:56:32.715801 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:56:32.715777 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vhzw\" (UniqueName: \"kubernetes.io/projected/88b2e8f2-799e-4e9e-8329-70969c9f7dce-kube-api-access-6vhzw\") pod \"feast-simple-feast-remote-setup-cc6567fcd-7z8l4\" (UID: \"88b2e8f2-799e-4e9e-8329-70969c9f7dce\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-cc6567fcd-7z8l4" Sep 11 08:56:33.211470 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:56:33.211438 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/88b2e8f2-799e-4e9e-8329-70969c9f7dce-online-tls\") pod \"feast-simple-feast-remote-setup-cc6567fcd-7z8l4\" (UID: \"88b2e8f2-799e-4e9e-8329-70969c9f7dce\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-cc6567fcd-7z8l4" Sep 11 08:56:33.213890 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:56:33.213868 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/88b2e8f2-799e-4e9e-8329-70969c9f7dce-online-tls\") pod \"feast-simple-feast-remote-setup-cc6567fcd-7z8l4\" (UID: \"88b2e8f2-799e-4e9e-8329-70969c9f7dce\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-cc6567fcd-7z8l4" Sep 11 08:56:33.401975 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:56:33.401941 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-cc6567fcd-7z8l4" Sep 11 08:56:33.675623 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:56:33.675543 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-cc6567fcd-7z8l4"] Sep 11 08:56:33.678367 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:56:33.678342 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod88b2e8f2_799e_4e9e_8329_70969c9f7dce.slice/crio-097eb44f6998a3fb937c5e0e9626cc0b084ad5d21c6f7c51c89eeb811ad9a81b WatchSource:0}: Error finding container 097eb44f6998a3fb937c5e0e9626cc0b084ad5d21c6f7c51c89eeb811ad9a81b: Status 404 returned error can't find the container with id 097eb44f6998a3fb937c5e0e9626cc0b084ad5d21c6f7c51c89eeb811ad9a81b Sep 11 08:56:33.915662 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:56:33.915620 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-cc6567fcd-7z8l4" event={"ID":"88b2e8f2-799e-4e9e-8329-70969c9f7dce","Type":"ContainerStarted","Data":"b4ce236e2c2829dd7eb611386e27baa9353022168490c32c44e2000ce73a1ed3"} Sep 11 08:56:33.915819 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:56:33.915674 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-cc6567fcd-7z8l4" event={"ID":"88b2e8f2-799e-4e9e-8329-70969c9f7dce","Type":"ContainerStarted","Data":"097eb44f6998a3fb937c5e0e9626cc0b084ad5d21c6f7c51c89eeb811ad9a81b"} Sep 11 08:56:38.937824 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:56:38.937793 2572 generic.go:356] "Generic (PLEG): container finished" podID="88b2e8f2-799e-4e9e-8329-70969c9f7dce" containerID="b4ce236e2c2829dd7eb611386e27baa9353022168490c32c44e2000ce73a1ed3" exitCode=0 Sep 11 08:56:38.938276 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:56:38.937842 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-cc6567fcd-7z8l4" event={"ID":"88b2e8f2-799e-4e9e-8329-70969c9f7dce","Type":"ContainerDied","Data":"b4ce236e2c2829dd7eb611386e27baa9353022168490c32c44e2000ce73a1ed3"} Sep 11 08:56:39.943801 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:56:39.943762 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-cc6567fcd-7z8l4" event={"ID":"88b2e8f2-799e-4e9e-8329-70969c9f7dce","Type":"ContainerStarted","Data":"966794ad63e7cd88c8bfb6d62976d3a787bdeba1db8247bc48b85db54a0af45c"} Sep 11 08:57:01.527139 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:57:01.527110 2572 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod88b2e8f2_799e_4e9e_8329_70969c9f7dce.slice/crio-conmon-966794ad63e7cd88c8bfb6d62976d3a787bdeba1db8247bc48b85db54a0af45c.scope\": RecentStats: unable to find data in memory cache]" Sep 11 08:57:02.039879 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:57:02.039843 2572 generic.go:356] "Generic (PLEG): container finished" podID="88b2e8f2-799e-4e9e-8329-70969c9f7dce" containerID="966794ad63e7cd88c8bfb6d62976d3a787bdeba1db8247bc48b85db54a0af45c" exitCode=0 Sep 11 08:57:02.040040 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:57:02.039893 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-cc6567fcd-7z8l4" event={"ID":"88b2e8f2-799e-4e9e-8329-70969c9f7dce","Type":"ContainerDied","Data":"966794ad63e7cd88c8bfb6d62976d3a787bdeba1db8247bc48b85db54a0af45c"} Sep 11 08:57:03.048840 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:57:03.048800 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-cc6567fcd-7z8l4" event={"ID":"88b2e8f2-799e-4e9e-8329-70969c9f7dce","Type":"ContainerStarted","Data":"7b212132d644e6cd7a85256c6e6a9f02c5a7942a4a6f0dcf126a641476ce2f6e"} Sep 11 08:57:03.049274 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:57:03.048861 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-cc6567fcd-7z8l4" event={"ID":"88b2e8f2-799e-4e9e-8329-70969c9f7dce","Type":"ContainerStarted","Data":"400ac9d353dc88188dce4896c7166692565e6bad1d2a329d3a60137b932faeb2"} Sep 11 08:57:03.049274 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:57:03.048881 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-cc6567fcd-7z8l4" event={"ID":"88b2e8f2-799e-4e9e-8329-70969c9f7dce","Type":"ContainerStarted","Data":"523e46bd77f02ea198cca91406afe6d89d03676c6920db11f266e4572e1cf24f"} Sep 11 08:57:03.074013 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:57:03.073953 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-cc6567fcd-7z8l4" podStartSLOduration=31.073933402 podStartE2EDuration="31.073933402s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 08:56:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 08:57:03.071639645 +0000 UTC m=+643.346215345" watchObservedRunningTime="2026-09-11 08:57:03.073933402 +0000 UTC m=+643.348509099" Sep 11 08:57:03.402463 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:57:03.402372 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-cc6567fcd-7z8l4" Sep 11 08:57:03.402634 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:57:03.402595 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-cc6567fcd-7z8l4" Sep 11 08:57:03.402634 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:57:03.402618 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-cc6567fcd-7z8l4" Sep 11 08:57:03.402634 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:57:03.402633 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-cc6567fcd-7z8l4" Sep 11 08:57:03.402769 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:57:03.402650 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-cc6567fcd-7z8l4" Sep 11 08:57:03.402769 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:57:03.402665 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-cc6567fcd-7z8l4" Sep 11 08:57:03.404784 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:57:03.404744 2572 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-cc6567fcd-7z8l4" podUID="88b2e8f2-799e-4e9e-8329-70969c9f7dce" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.31:8443: connect: connection refused" Sep 11 08:57:03.404784 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:57:03.404772 2572 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-cc6567fcd-7z8l4" podUID="88b2e8f2-799e-4e9e-8329-70969c9f7dce" containerName="online" probeResult="failure" output="Get \"https://10.132.0.31:6567/health\": dial tcp 10.132.0.31:6567: connect: connection refused" Sep 11 08:57:03.404987 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:57:03.404748 2572 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-cc6567fcd-7z8l4" podUID="88b2e8f2-799e-4e9e-8329-70969c9f7dce" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.31:8816: connect: connection refused" Sep 11 08:57:06.402508 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:57:06.402465 2572 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-cc6567fcd-7z8l4" podUID="88b2e8f2-799e-4e9e-8329-70969c9f7dce" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.31:8816: connect: connection refused" Sep 11 08:57:06.402508 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:57:06.402496 2572 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-cc6567fcd-7z8l4" podUID="88b2e8f2-799e-4e9e-8329-70969c9f7dce" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.31:8443: connect: connection refused" Sep 11 08:57:06.403076 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:57:06.402575 2572 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-cc6567fcd-7z8l4" podUID="88b2e8f2-799e-4e9e-8329-70969c9f7dce" containerName="online" probeResult="failure" output="Get \"https://10.132.0.31:6567/health\": dial tcp 10.132.0.31:6567: connect: connection refused" Sep 11 08:57:09.403457 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:57:09.403298 2572 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-cc6567fcd-7z8l4" podUID="88b2e8f2-799e-4e9e-8329-70969c9f7dce" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.31:8816: connect: connection refused" Sep 11 08:57:09.403457 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:57:09.403298 2572 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-cc6567fcd-7z8l4" podUID="88b2e8f2-799e-4e9e-8329-70969c9f7dce" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.31:8443: connect: connection refused" Sep 11 08:57:09.403984 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:57:09.403866 2572 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-cc6567fcd-7z8l4" podUID="88b2e8f2-799e-4e9e-8329-70969c9f7dce" containerName="online" probeResult="failure" output="Get \"https://10.132.0.31:6567/health\": dial tcp 10.132.0.31:6567: connect: connection refused" Sep 11 08:57:12.403551 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:57:12.403517 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-cc6567fcd-7z8l4" Sep 11 08:57:12.404112 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:57:12.403741 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-cc6567fcd-7z8l4" Sep 11 08:57:12.404378 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:57:12.404354 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-cc6567fcd-7z8l4" Sep 11 08:57:12.404685 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:57:12.404666 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-cc6567fcd-7z8l4" Sep 11 08:57:12.428944 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:57:12.428917 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-cc6567fcd-7z8l4" Sep 11 08:57:13.115715 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:57:13.115690 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-cc6567fcd-7z8l4" Sep 11 08:58:14.804873 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:14.804788 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-cc6567fcd-7z8l4"] Sep 11 08:58:14.805427 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:14.805108 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-cc6567fcd-7z8l4" podUID="88b2e8f2-799e-4e9e-8329-70969c9f7dce" containerName="online" containerID="cri-o://523e46bd77f02ea198cca91406afe6d89d03676c6920db11f266e4572e1cf24f" gracePeriod=30 Sep 11 08:58:14.805427 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:14.805180 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-cc6567fcd-7z8l4" podUID="88b2e8f2-799e-4e9e-8329-70969c9f7dce" containerName="ui" containerID="cri-o://7b212132d644e6cd7a85256c6e6a9f02c5a7942a4a6f0dcf126a641476ce2f6e" gracePeriod=30 Sep 11 08:58:14.805427 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:14.805249 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-cc6567fcd-7z8l4" podUID="88b2e8f2-799e-4e9e-8329-70969c9f7dce" containerName="offline" containerID="cri-o://400ac9d353dc88188dce4896c7166692565e6bad1d2a329d3a60137b932faeb2" gracePeriod=30 Sep 11 08:58:14.991300 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:14.991268 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-79d9f545c-wtnvv"] Sep 11 08:58:14.991524 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:14.991499 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-wtnvv" podUID="83c2cfc4-c365-4ca4-884c-014de928568f" containerName="registry" containerID="cri-o://cda100dcc261f070b6a8ef44416d414f1ad7c69bdc569dc1c5dd323457c92729" gracePeriod=30 Sep 11 08:58:14.991608 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:14.991570 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-wtnvv" podUID="83c2cfc4-c365-4ca4-884c-014de928568f" containerName="online" containerID="cri-o://e108953a85c248e247bab59cd720c292ec9900b699d41640875920e8b60d51a9" gracePeriod=30 Sep 11 08:58:14.991667 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:14.991598 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-wtnvv" podUID="83c2cfc4-c365-4ca4-884c-014de928568f" containerName="offline" containerID="cri-o://e423eab19850169d4765213500ad41b98cbdc142af9228a5419683b084899da7" gracePeriod=30 Sep 11 08:58:14.991725 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:14.991570 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-wtnvv" podUID="83c2cfc4-c365-4ca4-884c-014de928568f" containerName="ui" containerID="cri-o://1a524a716cf166d79c54284dfe6ea512aaf1c7abebcae5a8b4d5114694444805" gracePeriod=30 Sep 11 08:58:16.365906 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:16.365867 2572 generic.go:356] "Generic (PLEG): container finished" podID="83c2cfc4-c365-4ca4-884c-014de928568f" containerID="1a524a716cf166d79c54284dfe6ea512aaf1c7abebcae5a8b4d5114694444805" exitCode=0 Sep 11 08:58:16.365906 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:16.365899 2572 generic.go:356] "Generic (PLEG): container finished" podID="83c2cfc4-c365-4ca4-884c-014de928568f" containerID="e108953a85c248e247bab59cd720c292ec9900b699d41640875920e8b60d51a9" exitCode=0 Sep 11 08:58:16.366365 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:16.365929 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-wtnvv" event={"ID":"83c2cfc4-c365-4ca4-884c-014de928568f","Type":"ContainerDied","Data":"1a524a716cf166d79c54284dfe6ea512aaf1c7abebcae5a8b4d5114694444805"} Sep 11 08:58:16.366365 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:16.365972 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-wtnvv" event={"ID":"83c2cfc4-c365-4ca4-884c-014de928568f","Type":"ContainerDied","Data":"e108953a85c248e247bab59cd720c292ec9900b699d41640875920e8b60d51a9"} Sep 11 08:58:16.368708 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:16.368688 2572 generic.go:356] "Generic (PLEG): container finished" podID="88b2e8f2-799e-4e9e-8329-70969c9f7dce" containerID="7b212132d644e6cd7a85256c6e6a9f02c5a7942a4a6f0dcf126a641476ce2f6e" exitCode=0 Sep 11 08:58:16.368708 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:16.368706 2572 generic.go:356] "Generic (PLEG): container finished" podID="88b2e8f2-799e-4e9e-8329-70969c9f7dce" containerID="523e46bd77f02ea198cca91406afe6d89d03676c6920db11f266e4572e1cf24f" exitCode=0 Sep 11 08:58:16.368848 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:16.368753 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-cc6567fcd-7z8l4" event={"ID":"88b2e8f2-799e-4e9e-8329-70969c9f7dce","Type":"ContainerDied","Data":"7b212132d644e6cd7a85256c6e6a9f02c5a7942a4a6f0dcf126a641476ce2f6e"} Sep 11 08:58:16.368848 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:16.368783 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-cc6567fcd-7z8l4" event={"ID":"88b2e8f2-799e-4e9e-8329-70969c9f7dce","Type":"ContainerDied","Data":"523e46bd77f02ea198cca91406afe6d89d03676c6920db11f266e4572e1cf24f"} Sep 11 08:58:19.648291 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:19.648205 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-wtnvv" podUID="83c2cfc4-c365-4ca4-884c-014de928568f" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.30:8443: connect: connection refused" Sep 11 08:58:19.648731 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:19.648292 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-wtnvv" podUID="83c2cfc4-c365-4ca4-884c-014de928568f" containerName="online" probeResult="failure" output="Get \"https://10.132.0.30:6567/health\": dial tcp 10.132.0.30:6567: connect: connection refused" Sep 11 08:58:22.404870 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:22.404825 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-cc6567fcd-7z8l4" podUID="88b2e8f2-799e-4e9e-8329-70969c9f7dce" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.31:8443: connect: connection refused" Sep 11 08:58:23.096349 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:23.096304 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-cc6567fcd-7z8l4" podUID="88b2e8f2-799e-4e9e-8329-70969c9f7dce" containerName="online" probeResult="failure" output="Get \"https://10.132.0.31:6567/health\": dial tcp 10.132.0.31:6567: connect: connection refused" Sep 11 08:58:29.648443 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:29.648391 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-wtnvv" podUID="83c2cfc4-c365-4ca4-884c-014de928568f" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.30:8443: connect: connection refused" Sep 11 08:58:29.648890 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:29.648392 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-wtnvv" podUID="83c2cfc4-c365-4ca4-884c-014de928568f" containerName="online" probeResult="failure" output="Get \"https://10.132.0.30:6567/health\": dial tcp 10.132.0.30:6567: connect: connection refused" Sep 11 08:58:32.404446 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:32.404404 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-cc6567fcd-7z8l4" podUID="88b2e8f2-799e-4e9e-8329-70969c9f7dce" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.31:8443: connect: connection refused" Sep 11 08:58:33.097327 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:33.097281 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-cc6567fcd-7z8l4" podUID="88b2e8f2-799e-4e9e-8329-70969c9f7dce" containerName="online" probeResult="failure" output="Get \"https://10.132.0.31:6567/health\": dial tcp 10.132.0.31:6567: connect: connection refused" Sep 11 08:58:39.648468 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:39.648418 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-wtnvv" podUID="83c2cfc4-c365-4ca4-884c-014de928568f" containerName="online" probeResult="failure" output="Get \"https://10.132.0.30:6567/health\": dial tcp 10.132.0.30:6567: connect: connection refused" Sep 11 08:58:39.648468 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:39.648431 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-wtnvv" podUID="83c2cfc4-c365-4ca4-884c-014de928568f" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.30:8443: connect: connection refused" Sep 11 08:58:39.649021 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:39.648553 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-wtnvv" Sep 11 08:58:39.649021 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:39.648629 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-wtnvv" Sep 11 08:58:42.405051 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:42.404998 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-cc6567fcd-7z8l4" podUID="88b2e8f2-799e-4e9e-8329-70969c9f7dce" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.31:8443: connect: connection refused" Sep 11 08:58:42.405495 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:42.405133 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-cc6567fcd-7z8l4" Sep 11 08:58:43.097364 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:43.097325 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-cc6567fcd-7z8l4" podUID="88b2e8f2-799e-4e9e-8329-70969c9f7dce" containerName="online" probeResult="failure" output="Get \"https://10.132.0.31:6567/health\": dial tcp 10.132.0.31:6567: connect: connection refused" Sep 11 08:58:43.097537 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:43.097442 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-cc6567fcd-7z8l4" Sep 11 08:58:45.451152 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:45.451128 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-cc6567fcd-7z8l4" Sep 11 08:58:45.487919 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:45.487892 2572 generic.go:356] "Generic (PLEG): container finished" podID="83c2cfc4-c365-4ca4-884c-014de928568f" containerID="e423eab19850169d4765213500ad41b98cbdc142af9228a5419683b084899da7" exitCode=137 Sep 11 08:58:45.487919 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:45.487917 2572 generic.go:356] "Generic (PLEG): container finished" podID="83c2cfc4-c365-4ca4-884c-014de928568f" containerID="cda100dcc261f070b6a8ef44416d414f1ad7c69bdc569dc1c5dd323457c92729" exitCode=137 Sep 11 08:58:45.488105 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:45.487974 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-wtnvv" event={"ID":"83c2cfc4-c365-4ca4-884c-014de928568f","Type":"ContainerDied","Data":"e423eab19850169d4765213500ad41b98cbdc142af9228a5419683b084899da7"} Sep 11 08:58:45.488105 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:45.488029 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-wtnvv" event={"ID":"83c2cfc4-c365-4ca4-884c-014de928568f","Type":"ContainerDied","Data":"cda100dcc261f070b6a8ef44416d414f1ad7c69bdc569dc1c5dd323457c92729"} Sep 11 08:58:45.490456 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:45.490432 2572 generic.go:356] "Generic (PLEG): container finished" podID="88b2e8f2-799e-4e9e-8329-70969c9f7dce" containerID="400ac9d353dc88188dce4896c7166692565e6bad1d2a329d3a60137b932faeb2" exitCode=137 Sep 11 08:58:45.490561 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:45.490475 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-cc6567fcd-7z8l4" event={"ID":"88b2e8f2-799e-4e9e-8329-70969c9f7dce","Type":"ContainerDied","Data":"400ac9d353dc88188dce4896c7166692565e6bad1d2a329d3a60137b932faeb2"} Sep 11 08:58:45.490561 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:45.490505 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-cc6567fcd-7z8l4" event={"ID":"88b2e8f2-799e-4e9e-8329-70969c9f7dce","Type":"ContainerDied","Data":"097eb44f6998a3fb937c5e0e9626cc0b084ad5d21c6f7c51c89eeb811ad9a81b"} Sep 11 08:58:45.490561 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:45.490523 2572 scope.go:117] "RemoveContainer" containerID="7b212132d644e6cd7a85256c6e6a9f02c5a7942a4a6f0dcf126a641476ce2f6e" Sep 11 08:58:45.490561 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:45.490524 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-cc6567fcd-7z8l4" Sep 11 08:58:45.498539 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:45.498523 2572 scope.go:117] "RemoveContainer" containerID="400ac9d353dc88188dce4896c7166692565e6bad1d2a329d3a60137b932faeb2" Sep 11 08:58:45.506621 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:45.506605 2572 scope.go:117] "RemoveContainer" containerID="523e46bd77f02ea198cca91406afe6d89d03676c6920db11f266e4572e1cf24f" Sep 11 08:58:45.515267 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:45.515249 2572 scope.go:117] "RemoveContainer" containerID="966794ad63e7cd88c8bfb6d62976d3a787bdeba1db8247bc48b85db54a0af45c" Sep 11 08:58:45.522448 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:45.522422 2572 scope.go:117] "RemoveContainer" containerID="b4ce236e2c2829dd7eb611386e27baa9353022168490c32c44e2000ce73a1ed3" Sep 11 08:58:45.531496 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:45.531475 2572 scope.go:117] "RemoveContainer" containerID="7b212132d644e6cd7a85256c6e6a9f02c5a7942a4a6f0dcf126a641476ce2f6e" Sep 11 08:58:45.531774 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:58:45.531757 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b212132d644e6cd7a85256c6e6a9f02c5a7942a4a6f0dcf126a641476ce2f6e\": container with ID starting with 7b212132d644e6cd7a85256c6e6a9f02c5a7942a4a6f0dcf126a641476ce2f6e not found: ID does not exist" containerID="7b212132d644e6cd7a85256c6e6a9f02c5a7942a4a6f0dcf126a641476ce2f6e" Sep 11 08:58:45.531830 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:45.531784 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b212132d644e6cd7a85256c6e6a9f02c5a7942a4a6f0dcf126a641476ce2f6e"} err="failed to get container status \"7b212132d644e6cd7a85256c6e6a9f02c5a7942a4a6f0dcf126a641476ce2f6e\": rpc error: code = NotFound desc = could not find container \"7b212132d644e6cd7a85256c6e6a9f02c5a7942a4a6f0dcf126a641476ce2f6e\": container with ID starting with 7b212132d644e6cd7a85256c6e6a9f02c5a7942a4a6f0dcf126a641476ce2f6e not found: ID does not exist" Sep 11 08:58:45.531830 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:45.531801 2572 scope.go:117] "RemoveContainer" containerID="400ac9d353dc88188dce4896c7166692565e6bad1d2a329d3a60137b932faeb2" Sep 11 08:58:45.532152 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:58:45.532132 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"400ac9d353dc88188dce4896c7166692565e6bad1d2a329d3a60137b932faeb2\": container with ID starting with 400ac9d353dc88188dce4896c7166692565e6bad1d2a329d3a60137b932faeb2 not found: ID does not exist" containerID="400ac9d353dc88188dce4896c7166692565e6bad1d2a329d3a60137b932faeb2" Sep 11 08:58:45.532256 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:45.532157 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"400ac9d353dc88188dce4896c7166692565e6bad1d2a329d3a60137b932faeb2"} err="failed to get container status \"400ac9d353dc88188dce4896c7166692565e6bad1d2a329d3a60137b932faeb2\": rpc error: code = NotFound desc = could not find container \"400ac9d353dc88188dce4896c7166692565e6bad1d2a329d3a60137b932faeb2\": container with ID starting with 400ac9d353dc88188dce4896c7166692565e6bad1d2a329d3a60137b932faeb2 not found: ID does not exist" Sep 11 08:58:45.532256 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:45.532173 2572 scope.go:117] "RemoveContainer" containerID="523e46bd77f02ea198cca91406afe6d89d03676c6920db11f266e4572e1cf24f" Sep 11 08:58:45.532464 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:58:45.532446 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"523e46bd77f02ea198cca91406afe6d89d03676c6920db11f266e4572e1cf24f\": container with ID starting with 523e46bd77f02ea198cca91406afe6d89d03676c6920db11f266e4572e1cf24f not found: ID does not exist" containerID="523e46bd77f02ea198cca91406afe6d89d03676c6920db11f266e4572e1cf24f" Sep 11 08:58:45.532511 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:45.532467 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"523e46bd77f02ea198cca91406afe6d89d03676c6920db11f266e4572e1cf24f"} err="failed to get container status \"523e46bd77f02ea198cca91406afe6d89d03676c6920db11f266e4572e1cf24f\": rpc error: code = NotFound desc = could not find container \"523e46bd77f02ea198cca91406afe6d89d03676c6920db11f266e4572e1cf24f\": container with ID starting with 523e46bd77f02ea198cca91406afe6d89d03676c6920db11f266e4572e1cf24f not found: ID does not exist" Sep 11 08:58:45.532511 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:45.532487 2572 scope.go:117] "RemoveContainer" containerID="966794ad63e7cd88c8bfb6d62976d3a787bdeba1db8247bc48b85db54a0af45c" Sep 11 08:58:45.532768 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:58:45.532752 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"966794ad63e7cd88c8bfb6d62976d3a787bdeba1db8247bc48b85db54a0af45c\": container with ID starting with 966794ad63e7cd88c8bfb6d62976d3a787bdeba1db8247bc48b85db54a0af45c not found: ID does not exist" containerID="966794ad63e7cd88c8bfb6d62976d3a787bdeba1db8247bc48b85db54a0af45c" Sep 11 08:58:45.532805 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:45.532772 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"966794ad63e7cd88c8bfb6d62976d3a787bdeba1db8247bc48b85db54a0af45c"} err="failed to get container status \"966794ad63e7cd88c8bfb6d62976d3a787bdeba1db8247bc48b85db54a0af45c\": rpc error: code = NotFound desc = could not find container \"966794ad63e7cd88c8bfb6d62976d3a787bdeba1db8247bc48b85db54a0af45c\": container with ID starting with 966794ad63e7cd88c8bfb6d62976d3a787bdeba1db8247bc48b85db54a0af45c not found: ID does not exist" Sep 11 08:58:45.532805 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:45.532785 2572 scope.go:117] "RemoveContainer" containerID="b4ce236e2c2829dd7eb611386e27baa9353022168490c32c44e2000ce73a1ed3" Sep 11 08:58:45.533019 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:58:45.533003 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4ce236e2c2829dd7eb611386e27baa9353022168490c32c44e2000ce73a1ed3\": container with ID starting with b4ce236e2c2829dd7eb611386e27baa9353022168490c32c44e2000ce73a1ed3 not found: ID does not exist" containerID="b4ce236e2c2829dd7eb611386e27baa9353022168490c32c44e2000ce73a1ed3" Sep 11 08:58:45.533071 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:45.533023 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4ce236e2c2829dd7eb611386e27baa9353022168490c32c44e2000ce73a1ed3"} err="failed to get container status \"b4ce236e2c2829dd7eb611386e27baa9353022168490c32c44e2000ce73a1ed3\": rpc error: code = NotFound desc = could not find container \"b4ce236e2c2829dd7eb611386e27baa9353022168490c32c44e2000ce73a1ed3\": container with ID starting with b4ce236e2c2829dd7eb611386e27baa9353022168490c32c44e2000ce73a1ed3 not found: ID does not exist" Sep 11 08:58:45.594837 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:45.594777 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6vhzw\" (UniqueName: \"kubernetes.io/projected/88b2e8f2-799e-4e9e-8329-70969c9f7dce-kube-api-access-6vhzw\") pod \"88b2e8f2-799e-4e9e-8329-70969c9f7dce\" (UID: \"88b2e8f2-799e-4e9e-8329-70969c9f7dce\") " Sep 11 08:58:45.594947 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:45.594846 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/88b2e8f2-799e-4e9e-8329-70969c9f7dce-registry-tls\") pod \"88b2e8f2-799e-4e9e-8329-70969c9f7dce\" (UID: \"88b2e8f2-799e-4e9e-8329-70969c9f7dce\") " Sep 11 08:58:45.594947 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:45.594902 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/88b2e8f2-799e-4e9e-8329-70969c9f7dce-online-tls\") pod \"88b2e8f2-799e-4e9e-8329-70969c9f7dce\" (UID: \"88b2e8f2-799e-4e9e-8329-70969c9f7dce\") " Sep 11 08:58:45.594947 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:45.594922 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/88b2e8f2-799e-4e9e-8329-70969c9f7dce-offline-tls\") pod \"88b2e8f2-799e-4e9e-8329-70969c9f7dce\" (UID: \"88b2e8f2-799e-4e9e-8329-70969c9f7dce\") " Sep 11 08:58:45.594947 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:45.594938 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/88b2e8f2-799e-4e9e-8329-70969c9f7dce-feast-data\") pod \"88b2e8f2-799e-4e9e-8329-70969c9f7dce\" (UID: \"88b2e8f2-799e-4e9e-8329-70969c9f7dce\") " Sep 11 08:58:45.595154 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:45.594994 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/88b2e8f2-799e-4e9e-8329-70969c9f7dce-ui-tls\") pod \"88b2e8f2-799e-4e9e-8329-70969c9f7dce\" (UID: \"88b2e8f2-799e-4e9e-8329-70969c9f7dce\") " Sep 11 08:58:45.595243 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:45.595200 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88b2e8f2-799e-4e9e-8329-70969c9f7dce-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "88b2e8f2-799e-4e9e-8329-70969c9f7dce" (UID: "88b2e8f2-799e-4e9e-8329-70969c9f7dce"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 08:58:45.595483 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:45.595460 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88b2e8f2-799e-4e9e-8329-70969c9f7dce-feast-data" (OuterVolumeSpecName: "feast-data") pod "88b2e8f2-799e-4e9e-8329-70969c9f7dce" (UID: "88b2e8f2-799e-4e9e-8329-70969c9f7dce"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 08:58:45.597836 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:45.597809 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88b2e8f2-799e-4e9e-8329-70969c9f7dce-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "88b2e8f2-799e-4e9e-8329-70969c9f7dce" (UID: "88b2e8f2-799e-4e9e-8329-70969c9f7dce"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 08:58:45.597941 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:45.597853 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88b2e8f2-799e-4e9e-8329-70969c9f7dce-online-tls" (OuterVolumeSpecName: "online-tls") pod "88b2e8f2-799e-4e9e-8329-70969c9f7dce" (UID: "88b2e8f2-799e-4e9e-8329-70969c9f7dce"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 08:58:45.597941 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:45.597908 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88b2e8f2-799e-4e9e-8329-70969c9f7dce-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "88b2e8f2-799e-4e9e-8329-70969c9f7dce" (UID: "88b2e8f2-799e-4e9e-8329-70969c9f7dce"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 08:58:45.598026 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:45.597935 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88b2e8f2-799e-4e9e-8329-70969c9f7dce-kube-api-access-6vhzw" (OuterVolumeSpecName: "kube-api-access-6vhzw") pod "88b2e8f2-799e-4e9e-8329-70969c9f7dce" (UID: "88b2e8f2-799e-4e9e-8329-70969c9f7dce"). InnerVolumeSpecName "kube-api-access-6vhzw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 08:58:45.627615 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:45.627588 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-wtnvv" Sep 11 08:58:45.696115 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:45.696089 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/83c2cfc4-c365-4ca4-884c-014de928568f-registry-tls\") pod \"83c2cfc4-c365-4ca4-884c-014de928568f\" (UID: \"83c2cfc4-c365-4ca4-884c-014de928568f\") " Sep 11 08:58:45.696115 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:45.696120 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/83c2cfc4-c365-4ca4-884c-014de928568f-online-tls\") pod \"83c2cfc4-c365-4ca4-884c-014de928568f\" (UID: \"83c2cfc4-c365-4ca4-884c-014de928568f\") " Sep 11 08:58:45.696297 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:45.696155 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/83c2cfc4-c365-4ca4-884c-014de928568f-offline-tls\") pod \"83c2cfc4-c365-4ca4-884c-014de928568f\" (UID: \"83c2cfc4-c365-4ca4-884c-014de928568f\") " Sep 11 08:58:45.696297 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:45.696195 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/83c2cfc4-c365-4ca4-884c-014de928568f-ui-tls\") pod \"83c2cfc4-c365-4ca4-884c-014de928568f\" (UID: \"83c2cfc4-c365-4ca4-884c-014de928568f\") " Sep 11 08:58:45.696297 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:45.696257 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/83c2cfc4-c365-4ca4-884c-014de928568f-feast-data\") pod \"83c2cfc4-c365-4ca4-884c-014de928568f\" (UID: \"83c2cfc4-c365-4ca4-884c-014de928568f\") " Sep 11 08:58:45.696297 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:45.696289 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z5zmn\" (UniqueName: \"kubernetes.io/projected/83c2cfc4-c365-4ca4-884c-014de928568f-kube-api-access-z5zmn\") pod \"83c2cfc4-c365-4ca4-884c-014de928568f\" (UID: \"83c2cfc4-c365-4ca4-884c-014de928568f\") " Sep 11 08:58:45.696598 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:45.696571 2572 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/88b2e8f2-799e-4e9e-8329-70969c9f7dce-online-tls\") on node \"ip-10-0-128-166.ec2.internal\" DevicePath \"\"" Sep 11 08:58:45.696672 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:45.696600 2572 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/88b2e8f2-799e-4e9e-8329-70969c9f7dce-offline-tls\") on node \"ip-10-0-128-166.ec2.internal\" DevicePath \"\"" Sep 11 08:58:45.696672 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:45.696613 2572 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/88b2e8f2-799e-4e9e-8329-70969c9f7dce-feast-data\") on node \"ip-10-0-128-166.ec2.internal\" DevicePath \"\"" Sep 11 08:58:45.696672 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:45.696625 2572 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/88b2e8f2-799e-4e9e-8329-70969c9f7dce-ui-tls\") on node \"ip-10-0-128-166.ec2.internal\" DevicePath \"\"" Sep 11 08:58:45.696672 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:45.696637 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vhzw\" (UniqueName: \"kubernetes.io/projected/88b2e8f2-799e-4e9e-8329-70969c9f7dce-kube-api-access-6vhzw\") on node \"ip-10-0-128-166.ec2.internal\" DevicePath \"\"" Sep 11 08:58:45.696672 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:45.696650 2572 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/88b2e8f2-799e-4e9e-8329-70969c9f7dce-registry-tls\") on node \"ip-10-0-128-166.ec2.internal\" DevicePath \"\"" Sep 11 08:58:45.697084 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:45.696996 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83c2cfc4-c365-4ca4-884c-014de928568f-feast-data" (OuterVolumeSpecName: "feast-data") pod "83c2cfc4-c365-4ca4-884c-014de928568f" (UID: "83c2cfc4-c365-4ca4-884c-014de928568f"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 08:58:45.698400 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:45.698374 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83c2cfc4-c365-4ca4-884c-014de928568f-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "83c2cfc4-c365-4ca4-884c-014de928568f" (UID: "83c2cfc4-c365-4ca4-884c-014de928568f"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 08:58:45.698400 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:45.698387 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83c2cfc4-c365-4ca4-884c-014de928568f-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "83c2cfc4-c365-4ca4-884c-014de928568f" (UID: "83c2cfc4-c365-4ca4-884c-014de928568f"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 08:58:45.698536 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:45.698410 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83c2cfc4-c365-4ca4-884c-014de928568f-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "83c2cfc4-c365-4ca4-884c-014de928568f" (UID: "83c2cfc4-c365-4ca4-884c-014de928568f"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 08:58:45.698808 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:45.698796 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83c2cfc4-c365-4ca4-884c-014de928568f-online-tls" (OuterVolumeSpecName: "online-tls") pod "83c2cfc4-c365-4ca4-884c-014de928568f" (UID: "83c2cfc4-c365-4ca4-884c-014de928568f"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 08:58:45.698847 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:45.698819 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83c2cfc4-c365-4ca4-884c-014de928568f-kube-api-access-z5zmn" (OuterVolumeSpecName: "kube-api-access-z5zmn") pod "83c2cfc4-c365-4ca4-884c-014de928568f" (UID: "83c2cfc4-c365-4ca4-884c-014de928568f"). InnerVolumeSpecName "kube-api-access-z5zmn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 08:58:45.797170 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:45.797147 2572 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/83c2cfc4-c365-4ca4-884c-014de928568f-feast-data\") on node \"ip-10-0-128-166.ec2.internal\" DevicePath \"\"" Sep 11 08:58:45.797304 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:45.797170 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5zmn\" (UniqueName: \"kubernetes.io/projected/83c2cfc4-c365-4ca4-884c-014de928568f-kube-api-access-z5zmn\") on node \"ip-10-0-128-166.ec2.internal\" DevicePath \"\"" Sep 11 08:58:45.797304 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:45.797184 2572 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/83c2cfc4-c365-4ca4-884c-014de928568f-registry-tls\") on node \"ip-10-0-128-166.ec2.internal\" DevicePath \"\"" Sep 11 08:58:45.797304 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:45.797195 2572 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/83c2cfc4-c365-4ca4-884c-014de928568f-online-tls\") on node \"ip-10-0-128-166.ec2.internal\" DevicePath \"\"" Sep 11 08:58:45.797304 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:45.797207 2572 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/83c2cfc4-c365-4ca4-884c-014de928568f-offline-tls\") on node \"ip-10-0-128-166.ec2.internal\" DevicePath \"\"" Sep 11 08:58:45.797304 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:45.797217 2572 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/83c2cfc4-c365-4ca4-884c-014de928568f-ui-tls\") on node \"ip-10-0-128-166.ec2.internal\" DevicePath \"\"" Sep 11 08:58:45.811621 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:45.811600 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-cc6567fcd-7z8l4"] Sep 11 08:58:45.815427 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:45.815406 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-cc6567fcd-7z8l4"] Sep 11 08:58:46.253362 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:46.253318 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88b2e8f2-799e-4e9e-8329-70969c9f7dce" path="/var/lib/kubelet/pods/88b2e8f2-799e-4e9e-8329-70969c9f7dce/volumes" Sep 11 08:58:46.499113 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:46.499069 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-wtnvv" event={"ID":"83c2cfc4-c365-4ca4-884c-014de928568f","Type":"ContainerDied","Data":"3ead0556c5f1481710976395d9865804caa621ed409e50cc885ec8a2a9b3fe9a"} Sep 11 08:58:46.499554 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:46.499135 2572 scope.go:117] "RemoveContainer" containerID="1a524a716cf166d79c54284dfe6ea512aaf1c7abebcae5a8b4d5114694444805" Sep 11 08:58:46.499554 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:46.499088 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-79d9f545c-wtnvv" Sep 11 08:58:46.507605 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:46.507592 2572 scope.go:117] "RemoveContainer" containerID="e423eab19850169d4765213500ad41b98cbdc142af9228a5419683b084899da7" Sep 11 08:58:46.515095 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:46.515076 2572 scope.go:117] "RemoveContainer" containerID="e108953a85c248e247bab59cd720c292ec9900b699d41640875920e8b60d51a9" Sep 11 08:58:46.516601 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:46.516580 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-79d9f545c-wtnvv"] Sep 11 08:58:46.520267 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:46.520247 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-79d9f545c-wtnvv"] Sep 11 08:58:46.523182 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:46.523168 2572 scope.go:117] "RemoveContainer" containerID="cda100dcc261f070b6a8ef44416d414f1ad7c69bdc569dc1c5dd323457c92729" Sep 11 08:58:46.531998 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:46.531480 2572 scope.go:117] "RemoveContainer" containerID="8fa23ba46ec7eb09c69e03b6073ec3d2793c180b84cdac4992f5cf60941eba3e" Sep 11 08:58:46.539426 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:46.539410 2572 scope.go:117] "RemoveContainer" containerID="baddd8757743006abced413a448ccd5200f06a1108ce7fb55cb6d7d27cc7c438" Sep 11 08:58:48.252316 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:48.252283 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="83c2cfc4-c365-4ca4-884c-014de928568f" path="/var/lib/kubelet/pods/83c2cfc4-c365-4ca4-884c-014de928568f/volumes" Sep 11 08:58:57.767941 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:57.767900 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/postgres-864846d4f5-tjzg9"] Sep 11 08:58:57.768417 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:57.768397 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83c2cfc4-c365-4ca4-884c-014de928568f" containerName="feast-apply" Sep 11 08:58:57.768498 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:57.768420 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="83c2cfc4-c365-4ca4-884c-014de928568f" containerName="feast-apply" Sep 11 08:58:57.768498 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:57.768432 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83c2cfc4-c365-4ca4-884c-014de928568f" containerName="ui" Sep 11 08:58:57.768498 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:57.768441 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="83c2cfc4-c365-4ca4-884c-014de928568f" containerName="ui" Sep 11 08:58:57.768498 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:57.768459 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88b2e8f2-799e-4e9e-8329-70969c9f7dce" containerName="online" Sep 11 08:58:57.768498 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:57.768467 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="88b2e8f2-799e-4e9e-8329-70969c9f7dce" containerName="online" Sep 11 08:58:57.768498 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:57.768478 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83c2cfc4-c365-4ca4-884c-014de928568f" containerName="registry" Sep 11 08:58:57.768498 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:57.768486 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="83c2cfc4-c365-4ca4-884c-014de928568f" containerName="registry" Sep 11 08:58:57.768498 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:57.768495 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83c2cfc4-c365-4ca4-884c-014de928568f" containerName="online" Sep 11 08:58:57.768867 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:57.768503 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="83c2cfc4-c365-4ca4-884c-014de928568f" containerName="online" Sep 11 08:58:57.768867 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:57.768515 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88b2e8f2-799e-4e9e-8329-70969c9f7dce" containerName="ui" Sep 11 08:58:57.768867 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:57.768523 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="88b2e8f2-799e-4e9e-8329-70969c9f7dce" containerName="ui" Sep 11 08:58:57.768867 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:57.768536 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88b2e8f2-799e-4e9e-8329-70969c9f7dce" containerName="feast-apply" Sep 11 08:58:57.768867 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:57.768544 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="88b2e8f2-799e-4e9e-8329-70969c9f7dce" containerName="feast-apply" Sep 11 08:58:57.768867 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:57.768553 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88b2e8f2-799e-4e9e-8329-70969c9f7dce" containerName="offline" Sep 11 08:58:57.768867 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:57.768561 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="88b2e8f2-799e-4e9e-8329-70969c9f7dce" containerName="offline" Sep 11 08:58:57.768867 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:57.768572 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83c2cfc4-c365-4ca4-884c-014de928568f" containerName="feast-init" Sep 11 08:58:57.768867 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:57.768581 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="83c2cfc4-c365-4ca4-884c-014de928568f" containerName="feast-init" Sep 11 08:58:57.768867 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:57.768597 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88b2e8f2-799e-4e9e-8329-70969c9f7dce" containerName="feast-init" Sep 11 08:58:57.768867 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:57.768605 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="88b2e8f2-799e-4e9e-8329-70969c9f7dce" containerName="feast-init" Sep 11 08:58:57.768867 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:57.768617 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83c2cfc4-c365-4ca4-884c-014de928568f" containerName="offline" Sep 11 08:58:57.768867 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:57.768625 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="83c2cfc4-c365-4ca4-884c-014de928568f" containerName="offline" Sep 11 08:58:57.768867 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:57.768708 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="88b2e8f2-799e-4e9e-8329-70969c9f7dce" containerName="ui" Sep 11 08:58:57.768867 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:57.768719 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="83c2cfc4-c365-4ca4-884c-014de928568f" containerName="offline" Sep 11 08:58:57.768867 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:57.768727 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="83c2cfc4-c365-4ca4-884c-014de928568f" containerName="ui" Sep 11 08:58:57.768867 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:57.768737 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="83c2cfc4-c365-4ca4-884c-014de928568f" containerName="online" Sep 11 08:58:57.768867 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:57.768747 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="88b2e8f2-799e-4e9e-8329-70969c9f7dce" containerName="online" Sep 11 08:58:57.768867 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:57.768756 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="83c2cfc4-c365-4ca4-884c-014de928568f" containerName="registry" Sep 11 08:58:57.768867 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:57.768764 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="88b2e8f2-799e-4e9e-8329-70969c9f7dce" containerName="offline" Sep 11 08:58:57.773584 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:57.773566 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-864846d4f5-tjzg9" Sep 11 08:58:57.775521 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:57.775500 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"postgres-secret\"" Sep 11 08:58:57.775640 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:57.775621 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"default-dockercfg-kqd8r\"" Sep 11 08:58:57.775640 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:57.775635 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\"" Sep 11 08:58:57.776027 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:57.776011 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\"" Sep 11 08:58:57.779016 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:57.778991 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/postgres-864846d4f5-tjzg9"] Sep 11 08:58:57.887045 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:57.887007 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/efacc0be-de7d-49b9-b5be-c527de35e4f0-postgresdata\") pod \"postgres-864846d4f5-tjzg9\" (UID: \"efacc0be-de7d-49b9-b5be-c527de35e4f0\") " pod="test-ns-feast/postgres-864846d4f5-tjzg9" Sep 11 08:58:57.887195 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:57.887053 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvkvr\" (UniqueName: \"kubernetes.io/projected/efacc0be-de7d-49b9-b5be-c527de35e4f0-kube-api-access-fvkvr\") pod \"postgres-864846d4f5-tjzg9\" (UID: \"efacc0be-de7d-49b9-b5be-c527de35e4f0\") " pod="test-ns-feast/postgres-864846d4f5-tjzg9" Sep 11 08:58:57.988281 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:57.988252 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvkvr\" (UniqueName: \"kubernetes.io/projected/efacc0be-de7d-49b9-b5be-c527de35e4f0-kube-api-access-fvkvr\") pod \"postgres-864846d4f5-tjzg9\" (UID: \"efacc0be-de7d-49b9-b5be-c527de35e4f0\") " pod="test-ns-feast/postgres-864846d4f5-tjzg9" Sep 11 08:58:57.988407 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:57.988327 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/efacc0be-de7d-49b9-b5be-c527de35e4f0-postgresdata\") pod \"postgres-864846d4f5-tjzg9\" (UID: \"efacc0be-de7d-49b9-b5be-c527de35e4f0\") " pod="test-ns-feast/postgres-864846d4f5-tjzg9" Sep 11 08:58:57.988650 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:57.988635 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/efacc0be-de7d-49b9-b5be-c527de35e4f0-postgresdata\") pod \"postgres-864846d4f5-tjzg9\" (UID: \"efacc0be-de7d-49b9-b5be-c527de35e4f0\") " pod="test-ns-feast/postgres-864846d4f5-tjzg9" Sep 11 08:58:57.996618 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:57.996596 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvkvr\" (UniqueName: \"kubernetes.io/projected/efacc0be-de7d-49b9-b5be-c527de35e4f0-kube-api-access-fvkvr\") pod \"postgres-864846d4f5-tjzg9\" (UID: \"efacc0be-de7d-49b9-b5be-c527de35e4f0\") " pod="test-ns-feast/postgres-864846d4f5-tjzg9" Sep 11 08:58:58.091483 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:58.091421 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-864846d4f5-tjzg9" Sep 11 08:58:58.356820 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:58.356797 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/postgres-864846d4f5-tjzg9"] Sep 11 08:58:58.359193 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:58:58.359164 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podefacc0be_de7d_49b9_b5be_c527de35e4f0.slice/crio-63dd22de875fb01e3b00d2f7a00396b9c37ddf201d3b7ce42351df06b0b5091e WatchSource:0}: Error finding container 63dd22de875fb01e3b00d2f7a00396b9c37ddf201d3b7ce42351df06b0b5091e: Status 404 returned error can't find the container with id 63dd22de875fb01e3b00d2f7a00396b9c37ddf201d3b7ce42351df06b0b5091e Sep 11 08:58:58.361151 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:58.361132 2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 08:58:58.552527 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:58:58.552497 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-864846d4f5-tjzg9" event={"ID":"efacc0be-de7d-49b9-b5be-c527de35e4f0","Type":"ContainerStarted","Data":"63dd22de875fb01e3b00d2f7a00396b9c37ddf201d3b7ce42351df06b0b5091e"} Sep 11 08:59:03.576185 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:59:03.576149 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-864846d4f5-tjzg9" event={"ID":"efacc0be-de7d-49b9-b5be-c527de35e4f0","Type":"ContainerStarted","Data":"6579e9d66f5aa2c62728006c9dffd530aa50cbb86973a4eee6286949d397b7c5"} Sep 11 08:59:03.590087 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:59:03.590032 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/postgres-864846d4f5-tjzg9" podStartSLOduration=2.021803628 podStartE2EDuration="6.590017588s" totalImagesPullingTime="4.56821396s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 08:58:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 08:59:03.588999349 +0000 UTC m=+763.863575045" watchObservedRunningTime="2026-09-11 08:59:03.590017588 +0000 UTC m=+763.864593284" Sep 11 08:59:06.590865 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:59:06.590834 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-credit-scoring-797c5c65c5-g6nnz"] Sep 11 08:59:06.594559 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:59:06.594535 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-797c5c65c5-g6nnz" Sep 11 08:59:06.596861 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:59:06.596832 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-offline-tls\"" Sep 11 08:59:06.596861 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:59:06.596833 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-online-tls\"" Sep 11 08:59:06.597084 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:59:06.596865 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-intra-comm\"" Sep 11 08:59:06.597084 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:59:06.596880 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-dockercfg-t4pnn\"" Sep 11 08:59:06.597084 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:59:06.596880 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-registry-tls\"" Sep 11 08:59:06.606874 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:59:06.606856 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-credit-scoring-797c5c65c5-g6nnz"] Sep 11 08:59:06.664807 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:59:06.664780 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkqkp\" (UniqueName: \"kubernetes.io/projected/d169a9e0-7689-4170-96fa-f59f171484dd-kube-api-access-bkqkp\") pod \"feast-credit-scoring-797c5c65c5-g6nnz\" (UID: \"d169a9e0-7689-4170-96fa-f59f171484dd\") " pod="test-ns-feast/feast-credit-scoring-797c5c65c5-g6nnz" Sep 11 08:59:06.664910 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:59:06.664820 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/d169a9e0-7689-4170-96fa-f59f171484dd-online-tls\") pod \"feast-credit-scoring-797c5c65c5-g6nnz\" (UID: \"d169a9e0-7689-4170-96fa-f59f171484dd\") " pod="test-ns-feast/feast-credit-scoring-797c5c65c5-g6nnz" Sep 11 08:59:06.664910 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:59:06.664842 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/d169a9e0-7689-4170-96fa-f59f171484dd-offline-tls\") pod \"feast-credit-scoring-797c5c65c5-g6nnz\" (UID: \"d169a9e0-7689-4170-96fa-f59f171484dd\") " pod="test-ns-feast/feast-credit-scoring-797c5c65c5-g6nnz" Sep 11 08:59:06.664910 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:59:06.664897 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/d169a9e0-7689-4170-96fa-f59f171484dd-registry-tls\") pod \"feast-credit-scoring-797c5c65c5-g6nnz\" (UID: \"d169a9e0-7689-4170-96fa-f59f171484dd\") " pod="test-ns-feast/feast-credit-scoring-797c5c65c5-g6nnz" Sep 11 08:59:06.665017 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:59:06.664914 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/d169a9e0-7689-4170-96fa-f59f171484dd-feast-data\") pod \"feast-credit-scoring-797c5c65c5-g6nnz\" (UID: \"d169a9e0-7689-4170-96fa-f59f171484dd\") " pod="test-ns-feast/feast-credit-scoring-797c5c65c5-g6nnz" Sep 11 08:59:06.766173 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:59:06.766146 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkqkp\" (UniqueName: \"kubernetes.io/projected/d169a9e0-7689-4170-96fa-f59f171484dd-kube-api-access-bkqkp\") pod \"feast-credit-scoring-797c5c65c5-g6nnz\" (UID: \"d169a9e0-7689-4170-96fa-f59f171484dd\") " pod="test-ns-feast/feast-credit-scoring-797c5c65c5-g6nnz" Sep 11 08:59:06.766338 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:59:06.766193 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/d169a9e0-7689-4170-96fa-f59f171484dd-online-tls\") pod \"feast-credit-scoring-797c5c65c5-g6nnz\" (UID: \"d169a9e0-7689-4170-96fa-f59f171484dd\") " pod="test-ns-feast/feast-credit-scoring-797c5c65c5-g6nnz" Sep 11 08:59:06.766338 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:59:06.766214 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/d169a9e0-7689-4170-96fa-f59f171484dd-offline-tls\") pod \"feast-credit-scoring-797c5c65c5-g6nnz\" (UID: \"d169a9e0-7689-4170-96fa-f59f171484dd\") " pod="test-ns-feast/feast-credit-scoring-797c5c65c5-g6nnz" Sep 11 08:59:06.766338 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:59:06.766269 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/d169a9e0-7689-4170-96fa-f59f171484dd-registry-tls\") pod \"feast-credit-scoring-797c5c65c5-g6nnz\" (UID: \"d169a9e0-7689-4170-96fa-f59f171484dd\") " pod="test-ns-feast/feast-credit-scoring-797c5c65c5-g6nnz" Sep 11 08:59:06.766338 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:59:06.766286 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/d169a9e0-7689-4170-96fa-f59f171484dd-feast-data\") pod \"feast-credit-scoring-797c5c65c5-g6nnz\" (UID: \"d169a9e0-7689-4170-96fa-f59f171484dd\") " pod="test-ns-feast/feast-credit-scoring-797c5c65c5-g6nnz" Sep 11 08:59:06.766490 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:59:06.766374 2572 secret.go:189] Couldn't get secret test-ns-feast/feast-credit-scoring-offline-tls: secret "feast-credit-scoring-offline-tls" not found Sep 11 08:59:06.766490 ip-10-0-128-166 kubenswrapper[2572]: E0911 08:59:06.766437 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d169a9e0-7689-4170-96fa-f59f171484dd-offline-tls podName:d169a9e0-7689-4170-96fa-f59f171484dd nodeName:}" failed. No retries permitted until 2026-09-11 08:59:07.266418595 +0000 UTC m=+767.540994286 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/d169a9e0-7689-4170-96fa-f59f171484dd-offline-tls") pod "feast-credit-scoring-797c5c65c5-g6nnz" (UID: "d169a9e0-7689-4170-96fa-f59f171484dd") : secret "feast-credit-scoring-offline-tls" not found Sep 11 08:59:06.766750 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:59:06.766722 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/d169a9e0-7689-4170-96fa-f59f171484dd-feast-data\") pod \"feast-credit-scoring-797c5c65c5-g6nnz\" (UID: \"d169a9e0-7689-4170-96fa-f59f171484dd\") " pod="test-ns-feast/feast-credit-scoring-797c5c65c5-g6nnz" Sep 11 08:59:06.768772 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:59:06.768750 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/d169a9e0-7689-4170-96fa-f59f171484dd-online-tls\") pod \"feast-credit-scoring-797c5c65c5-g6nnz\" (UID: \"d169a9e0-7689-4170-96fa-f59f171484dd\") " pod="test-ns-feast/feast-credit-scoring-797c5c65c5-g6nnz" Sep 11 08:59:06.768848 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:59:06.768809 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/d169a9e0-7689-4170-96fa-f59f171484dd-registry-tls\") pod \"feast-credit-scoring-797c5c65c5-g6nnz\" (UID: \"d169a9e0-7689-4170-96fa-f59f171484dd\") " pod="test-ns-feast/feast-credit-scoring-797c5c65c5-g6nnz" Sep 11 08:59:06.776617 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:59:06.776596 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkqkp\" (UniqueName: \"kubernetes.io/projected/d169a9e0-7689-4170-96fa-f59f171484dd-kube-api-access-bkqkp\") pod \"feast-credit-scoring-797c5c65c5-g6nnz\" (UID: \"d169a9e0-7689-4170-96fa-f59f171484dd\") " pod="test-ns-feast/feast-credit-scoring-797c5c65c5-g6nnz" Sep 11 08:59:07.270914 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:59:07.270874 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/d169a9e0-7689-4170-96fa-f59f171484dd-offline-tls\") pod \"feast-credit-scoring-797c5c65c5-g6nnz\" (UID: \"d169a9e0-7689-4170-96fa-f59f171484dd\") " pod="test-ns-feast/feast-credit-scoring-797c5c65c5-g6nnz" Sep 11 08:59:07.273514 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:59:07.273494 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/d169a9e0-7689-4170-96fa-f59f171484dd-offline-tls\") pod \"feast-credit-scoring-797c5c65c5-g6nnz\" (UID: \"d169a9e0-7689-4170-96fa-f59f171484dd\") " pod="test-ns-feast/feast-credit-scoring-797c5c65c5-g6nnz" Sep 11 08:59:07.504126 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:59:07.504091 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-797c5c65c5-g6nnz" Sep 11 08:59:07.780250 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:59:07.780206 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-credit-scoring-797c5c65c5-g6nnz"] Sep 11 08:59:07.781850 ip-10-0-128-166 kubenswrapper[2572]: W0911 08:59:07.781825 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd169a9e0_7689_4170_96fa_f59f171484dd.slice/crio-73e9accb532d7606500416254e3451b167b97fc496b91a0080e0a52e11f9ebb5 WatchSource:0}: Error finding container 73e9accb532d7606500416254e3451b167b97fc496b91a0080e0a52e11f9ebb5: Status 404 returned error can't find the container with id 73e9accb532d7606500416254e3451b167b97fc496b91a0080e0a52e11f9ebb5 Sep 11 08:59:08.598372 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:59:08.598334 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-797c5c65c5-g6nnz" event={"ID":"d169a9e0-7689-4170-96fa-f59f171484dd","Type":"ContainerStarted","Data":"0f61c6e76c03cef07f124af6c3614e5a7dbe56e28c3ed642d0b2376d3d9e7d69"} Sep 11 08:59:08.598541 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:59:08.598422 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-797c5c65c5-g6nnz" event={"ID":"d169a9e0-7689-4170-96fa-f59f171484dd","Type":"ContainerStarted","Data":"73e9accb532d7606500416254e3451b167b97fc496b91a0080e0a52e11f9ebb5"} Sep 11 08:59:09.604264 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:59:09.604205 2572 generic.go:356] "Generic (PLEG): container finished" podID="d169a9e0-7689-4170-96fa-f59f171484dd" containerID="0f61c6e76c03cef07f124af6c3614e5a7dbe56e28c3ed642d0b2376d3d9e7d69" exitCode=0 Sep 11 08:59:09.604658 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:59:09.604292 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-797c5c65c5-g6nnz" event={"ID":"d169a9e0-7689-4170-96fa-f59f171484dd","Type":"ContainerDied","Data":"0f61c6e76c03cef07f124af6c3614e5a7dbe56e28c3ed642d0b2376d3d9e7d69"} Sep 11 08:59:10.610321 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:59:10.610284 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-797c5c65c5-g6nnz" event={"ID":"d169a9e0-7689-4170-96fa-f59f171484dd","Type":"ContainerStarted","Data":"b8a1163aa800f7a5612d35ee3a1adb085c275d427ecbddcc3fc73f185d8e88e9"} Sep 11 08:59:16.639048 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:59:16.639014 2572 generic.go:356] "Generic (PLEG): container finished" podID="d169a9e0-7689-4170-96fa-f59f171484dd" containerID="b8a1163aa800f7a5612d35ee3a1adb085c275d427ecbddcc3fc73f185d8e88e9" exitCode=0 Sep 11 08:59:16.639469 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:59:16.639084 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-797c5c65c5-g6nnz" event={"ID":"d169a9e0-7689-4170-96fa-f59f171484dd","Type":"ContainerDied","Data":"b8a1163aa800f7a5612d35ee3a1adb085c275d427ecbddcc3fc73f185d8e88e9"} Sep 11 08:59:17.648715 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:59:17.648667 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-797c5c65c5-g6nnz" event={"ID":"d169a9e0-7689-4170-96fa-f59f171484dd","Type":"ContainerStarted","Data":"cb4806ae0f11e6318bdbb09bcfe72b645be509133c3fe09df519c27970df3498"} Sep 11 08:59:17.649159 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:59:17.648791 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-797c5c65c5-g6nnz" event={"ID":"d169a9e0-7689-4170-96fa-f59f171484dd","Type":"ContainerStarted","Data":"ac4be6248302ff451fa131d39fae0bb56a1c61018f257523cf3b5f6bdd37826f"} Sep 11 08:59:17.649159 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:59:17.648824 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-797c5c65c5-g6nnz" event={"ID":"d169a9e0-7689-4170-96fa-f59f171484dd","Type":"ContainerStarted","Data":"473f855dadd3db9c5daa5ea6843af1f8692f17a2615c7ad0b0c9d3c778ec5d87"} Sep 11 08:59:17.670918 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:59:17.670849 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-credit-scoring-797c5c65c5-g6nnz" podStartSLOduration=11.67083006 podStartE2EDuration="11.67083006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 08:59:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 08:59:17.670438917 +0000 UTC m=+777.945014614" watchObservedRunningTime="2026-09-11 08:59:17.67083006 +0000 UTC m=+777.945405769" Sep 11 08:59:19.504430 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:59:19.504384 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-credit-scoring-797c5c65c5-g6nnz" Sep 11 08:59:19.504916 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:59:19.504467 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-credit-scoring-797c5c65c5-g6nnz" Sep 11 08:59:19.504916 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:59:19.504495 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-credit-scoring-797c5c65c5-g6nnz" Sep 11 08:59:19.506043 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:59:19.506003 2572 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-797c5c65c5-g6nnz" podUID="d169a9e0-7689-4170-96fa-f59f171484dd" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.33:8816: connect: connection refused" Sep 11 08:59:19.506190 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:59:19.506037 2572 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-797c5c65c5-g6nnz" podUID="d169a9e0-7689-4170-96fa-f59f171484dd" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.33:6571: connect: connection refused" Sep 11 08:59:19.506190 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:59:19.506071 2572 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-797c5c65c5-g6nnz" podUID="d169a9e0-7689-4170-96fa-f59f171484dd" containerName="online" probeResult="failure" output="Get \"https://10.132.0.33:6567/health\": dial tcp 10.132.0.33:6567: connect: connection refused" Sep 11 08:59:22.505373 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:59:22.505323 2572 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-797c5c65c5-g6nnz" podUID="d169a9e0-7689-4170-96fa-f59f171484dd" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.33:8816: connect: connection refused" Sep 11 08:59:22.505858 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:59:22.505429 2572 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-797c5c65c5-g6nnz" podUID="d169a9e0-7689-4170-96fa-f59f171484dd" containerName="online" probeResult="failure" output="Get \"https://10.132.0.33:6567/health\": dial tcp 10.132.0.33:6567: connect: connection refused" Sep 11 08:59:22.505858 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:59:22.505327 2572 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-797c5c65c5-g6nnz" podUID="d169a9e0-7689-4170-96fa-f59f171484dd" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.33:6571: connect: connection refused" Sep 11 08:59:25.505409 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:59:25.505356 2572 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-797c5c65c5-g6nnz" podUID="d169a9e0-7689-4170-96fa-f59f171484dd" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.33:6571: connect: connection refused" Sep 11 08:59:25.506357 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:59:25.506335 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-credit-scoring-797c5c65c5-g6nnz" Sep 11 08:59:25.506796 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:59:25.506769 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-797c5c65c5-g6nnz" Sep 11 08:59:25.507176 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:59:25.507158 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-credit-scoring-797c5c65c5-g6nnz" Sep 11 08:59:25.511786 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:59:25.511767 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-credit-scoring-797c5c65c5-g6nnz" Sep 11 08:59:25.694567 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:59:25.694536 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-797c5c65c5-g6nnz" Sep 11 08:59:25.697669 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:59:25.697649 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-credit-scoring-797c5c65c5-g6nnz" Sep 11 08:59:27.504826 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:59:27.504781 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-797c5c65c5-g6nnz" Sep 11 08:59:28.504538 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:59:28.504497 2572 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-797c5c65c5-g6nnz" podUID="d169a9e0-7689-4170-96fa-f59f171484dd" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.33:6571: connect: connection refused" Sep 11 08:59:31.505498 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:59:31.505461 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-credit-scoring-797c5c65c5-g6nnz" Sep 11 08:59:31.506352 ip-10-0-128-166 kubenswrapper[2572]: I0911 08:59:31.506334 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-credit-scoring-797c5c65c5-g6nnz" Sep 11 09:00:34.438448 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:00:34.438416 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-credit-scoring-797c5c65c5-g6nnz"] Sep 11 09:00:34.438998 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:00:34.438638 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-credit-scoring-797c5c65c5-g6nnz" podUID="d169a9e0-7689-4170-96fa-f59f171484dd" containerName="registry" containerID="cri-o://473f855dadd3db9c5daa5ea6843af1f8692f17a2615c7ad0b0c9d3c778ec5d87" gracePeriod=30 Sep 11 09:00:34.438998 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:00:34.438695 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-credit-scoring-797c5c65c5-g6nnz" podUID="d169a9e0-7689-4170-96fa-f59f171484dd" containerName="offline" containerID="cri-o://cb4806ae0f11e6318bdbb09bcfe72b645be509133c3fe09df519c27970df3498" gracePeriod=30 Sep 11 09:00:34.438998 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:00:34.438738 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-credit-scoring-797c5c65c5-g6nnz" podUID="d169a9e0-7689-4170-96fa-f59f171484dd" containerName="online" containerID="cri-o://ac4be6248302ff451fa131d39fae0bb56a1c61018f257523cf3b5f6bdd37826f" gracePeriod=30 Sep 11 09:00:34.455169 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:00:34.455140 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/postgres-864846d4f5-tjzg9"] Sep 11 09:00:34.455361 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:00:34.455329 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/postgres-864846d4f5-tjzg9" podUID="efacc0be-de7d-49b9-b5be-c527de35e4f0" containerName="postgres" containerID="cri-o://6579e9d66f5aa2c62728006c9dffd530aa50cbb86973a4eee6286949d397b7c5" gracePeriod=30 Sep 11 09:00:34.699392 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:00:34.699360 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-864846d4f5-tjzg9" Sep 11 09:00:34.812376 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:00:34.812342 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fvkvr\" (UniqueName: \"kubernetes.io/projected/efacc0be-de7d-49b9-b5be-c527de35e4f0-kube-api-access-fvkvr\") pod \"efacc0be-de7d-49b9-b5be-c527de35e4f0\" (UID: \"efacc0be-de7d-49b9-b5be-c527de35e4f0\") " Sep 11 09:00:34.812576 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:00:34.812408 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/efacc0be-de7d-49b9-b5be-c527de35e4f0-postgresdata\") pod \"efacc0be-de7d-49b9-b5be-c527de35e4f0\" (UID: \"efacc0be-de7d-49b9-b5be-c527de35e4f0\") " Sep 11 09:00:34.814605 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:00:34.814573 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efacc0be-de7d-49b9-b5be-c527de35e4f0-kube-api-access-fvkvr" (OuterVolumeSpecName: "kube-api-access-fvkvr") pod "efacc0be-de7d-49b9-b5be-c527de35e4f0" (UID: "efacc0be-de7d-49b9-b5be-c527de35e4f0"). InnerVolumeSpecName "kube-api-access-fvkvr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 09:00:34.841869 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:00:34.841832 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efacc0be-de7d-49b9-b5be-c527de35e4f0-postgresdata" (OuterVolumeSpecName: "postgresdata") pod "efacc0be-de7d-49b9-b5be-c527de35e4f0" (UID: "efacc0be-de7d-49b9-b5be-c527de35e4f0"). InnerVolumeSpecName "postgresdata". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 09:00:34.913137 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:00:34.913103 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvkvr\" (UniqueName: \"kubernetes.io/projected/efacc0be-de7d-49b9-b5be-c527de35e4f0-kube-api-access-fvkvr\") on node \"ip-10-0-128-166.ec2.internal\" DevicePath \"\"" Sep 11 09:00:34.913137 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:00:34.913129 2572 reconciler_common.go:299] "Volume detached for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/efacc0be-de7d-49b9-b5be-c527de35e4f0-postgresdata\") on node \"ip-10-0-128-166.ec2.internal\" DevicePath \"\"" Sep 11 09:00:35.008582 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:00:35.008483 2572 generic.go:356] "Generic (PLEG): container finished" podID="efacc0be-de7d-49b9-b5be-c527de35e4f0" containerID="6579e9d66f5aa2c62728006c9dffd530aa50cbb86973a4eee6286949d397b7c5" exitCode=0 Sep 11 09:00:35.008582 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:00:35.008530 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-864846d4f5-tjzg9" Sep 11 09:00:35.008810 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:00:35.008576 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-864846d4f5-tjzg9" event={"ID":"efacc0be-de7d-49b9-b5be-c527de35e4f0","Type":"ContainerDied","Data":"6579e9d66f5aa2c62728006c9dffd530aa50cbb86973a4eee6286949d397b7c5"} Sep 11 09:00:35.008810 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:00:35.008619 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-864846d4f5-tjzg9" event={"ID":"efacc0be-de7d-49b9-b5be-c527de35e4f0","Type":"ContainerDied","Data":"63dd22de875fb01e3b00d2f7a00396b9c37ddf201d3b7ce42351df06b0b5091e"} Sep 11 09:00:35.008810 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:00:35.008637 2572 scope.go:117] "RemoveContainer" containerID="6579e9d66f5aa2c62728006c9dffd530aa50cbb86973a4eee6286949d397b7c5" Sep 11 09:00:35.017549 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:00:35.017527 2572 scope.go:117] "RemoveContainer" containerID="6579e9d66f5aa2c62728006c9dffd530aa50cbb86973a4eee6286949d397b7c5" Sep 11 09:00:35.017811 ip-10-0-128-166 kubenswrapper[2572]: E0911 09:00:35.017793 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6579e9d66f5aa2c62728006c9dffd530aa50cbb86973a4eee6286949d397b7c5\": container with ID starting with 6579e9d66f5aa2c62728006c9dffd530aa50cbb86973a4eee6286949d397b7c5 not found: ID does not exist" containerID="6579e9d66f5aa2c62728006c9dffd530aa50cbb86973a4eee6286949d397b7c5" Sep 11 09:00:35.017858 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:00:35.017818 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6579e9d66f5aa2c62728006c9dffd530aa50cbb86973a4eee6286949d397b7c5"} err="failed to get container status \"6579e9d66f5aa2c62728006c9dffd530aa50cbb86973a4eee6286949d397b7c5\": rpc error: code = NotFound desc = could not find container \"6579e9d66f5aa2c62728006c9dffd530aa50cbb86973a4eee6286949d397b7c5\": container with ID starting with 6579e9d66f5aa2c62728006c9dffd530aa50cbb86973a4eee6286949d397b7c5 not found: ID does not exist" Sep 11 09:00:35.027395 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:00:35.027374 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/postgres-864846d4f5-tjzg9"] Sep 11 09:00:35.030456 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:00:35.030438 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/postgres-864846d4f5-tjzg9"] Sep 11 09:00:35.694666 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:00:35.694628 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-797c5c65c5-g6nnz" podUID="d169a9e0-7689-4170-96fa-f59f171484dd" containerName="online" probeResult="failure" output="Get \"https://10.132.0.33:6567/health\": dial tcp 10.132.0.33:6567: connect: connection refused" Sep 11 09:00:36.015626 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:00:36.015539 2572 generic.go:356] "Generic (PLEG): container finished" podID="d169a9e0-7689-4170-96fa-f59f171484dd" containerID="ac4be6248302ff451fa131d39fae0bb56a1c61018f257523cf3b5f6bdd37826f" exitCode=0 Sep 11 09:00:36.015626 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:00:36.015611 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-797c5c65c5-g6nnz" event={"ID":"d169a9e0-7689-4170-96fa-f59f171484dd","Type":"ContainerDied","Data":"ac4be6248302ff451fa131d39fae0bb56a1c61018f257523cf3b5f6bdd37826f"} Sep 11 09:00:36.256040 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:00:36.256008 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="efacc0be-de7d-49b9-b5be-c527de35e4f0" path="/var/lib/kubelet/pods/efacc0be-de7d-49b9-b5be-c527de35e4f0/volumes" Sep 11 09:00:45.694841 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:00:45.694797 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-797c5c65c5-g6nnz" podUID="d169a9e0-7689-4170-96fa-f59f171484dd" containerName="online" probeResult="failure" output="Get \"https://10.132.0.33:6567/health\": dial tcp 10.132.0.33:6567: connect: connection refused" Sep 11 09:00:55.694820 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:00:55.694776 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-797c5c65c5-g6nnz" podUID="d169a9e0-7689-4170-96fa-f59f171484dd" containerName="online" probeResult="failure" output="Get \"https://10.132.0.33:6567/health\": dial tcp 10.132.0.33:6567: connect: connection refused" Sep 11 09:00:55.695366 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:00:55.694893 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-797c5c65c5-g6nnz" Sep 11 09:01:05.085384 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:01:05.085358 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-797c5c65c5-g6nnz" Sep 11 09:01:05.132501 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:01:05.132470 2572 generic.go:356] "Generic (PLEG): container finished" podID="d169a9e0-7689-4170-96fa-f59f171484dd" containerID="cb4806ae0f11e6318bdbb09bcfe72b645be509133c3fe09df519c27970df3498" exitCode=137 Sep 11 09:01:05.132501 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:01:05.132501 2572 generic.go:356] "Generic (PLEG): container finished" podID="d169a9e0-7689-4170-96fa-f59f171484dd" containerID="473f855dadd3db9c5daa5ea6843af1f8692f17a2615c7ad0b0c9d3c778ec5d87" exitCode=137 Sep 11 09:01:05.132675 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:01:05.132531 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-797c5c65c5-g6nnz" Sep 11 09:01:05.132675 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:01:05.132536 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-797c5c65c5-g6nnz" event={"ID":"d169a9e0-7689-4170-96fa-f59f171484dd","Type":"ContainerDied","Data":"cb4806ae0f11e6318bdbb09bcfe72b645be509133c3fe09df519c27970df3498"} Sep 11 09:01:05.132675 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:01:05.132596 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-797c5c65c5-g6nnz" event={"ID":"d169a9e0-7689-4170-96fa-f59f171484dd","Type":"ContainerDied","Data":"473f855dadd3db9c5daa5ea6843af1f8692f17a2615c7ad0b0c9d3c778ec5d87"} Sep 11 09:01:05.132675 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:01:05.132620 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-797c5c65c5-g6nnz" event={"ID":"d169a9e0-7689-4170-96fa-f59f171484dd","Type":"ContainerDied","Data":"73e9accb532d7606500416254e3451b167b97fc496b91a0080e0a52e11f9ebb5"} Sep 11 09:01:05.132675 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:01:05.132643 2572 scope.go:117] "RemoveContainer" containerID="cb4806ae0f11e6318bdbb09bcfe72b645be509133c3fe09df519c27970df3498" Sep 11 09:01:05.140942 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:01:05.140924 2572 scope.go:117] "RemoveContainer" containerID="ac4be6248302ff451fa131d39fae0bb56a1c61018f257523cf3b5f6bdd37826f" Sep 11 09:01:05.145048 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:01:05.145032 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bkqkp\" (UniqueName: \"kubernetes.io/projected/d169a9e0-7689-4170-96fa-f59f171484dd-kube-api-access-bkqkp\") pod \"d169a9e0-7689-4170-96fa-f59f171484dd\" (UID: \"d169a9e0-7689-4170-96fa-f59f171484dd\") " Sep 11 09:01:05.145140 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:01:05.145069 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/d169a9e0-7689-4170-96fa-f59f171484dd-registry-tls\") pod \"d169a9e0-7689-4170-96fa-f59f171484dd\" (UID: \"d169a9e0-7689-4170-96fa-f59f171484dd\") " Sep 11 09:01:05.145140 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:01:05.145109 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/d169a9e0-7689-4170-96fa-f59f171484dd-online-tls\") pod \"d169a9e0-7689-4170-96fa-f59f171484dd\" (UID: \"d169a9e0-7689-4170-96fa-f59f171484dd\") " Sep 11 09:01:05.145140 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:01:05.145140 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/d169a9e0-7689-4170-96fa-f59f171484dd-feast-data\") pod \"d169a9e0-7689-4170-96fa-f59f171484dd\" (UID: \"d169a9e0-7689-4170-96fa-f59f171484dd\") " Sep 11 09:01:05.145340 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:01:05.145174 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/d169a9e0-7689-4170-96fa-f59f171484dd-offline-tls\") pod \"d169a9e0-7689-4170-96fa-f59f171484dd\" (UID: \"d169a9e0-7689-4170-96fa-f59f171484dd\") " Sep 11 09:01:05.147388 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:01:05.147357 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d169a9e0-7689-4170-96fa-f59f171484dd-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "d169a9e0-7689-4170-96fa-f59f171484dd" (UID: "d169a9e0-7689-4170-96fa-f59f171484dd"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 09:01:05.147578 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:01:05.147559 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d169a9e0-7689-4170-96fa-f59f171484dd-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "d169a9e0-7689-4170-96fa-f59f171484dd" (UID: "d169a9e0-7689-4170-96fa-f59f171484dd"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 09:01:05.147722 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:01:05.147704 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d169a9e0-7689-4170-96fa-f59f171484dd-online-tls" (OuterVolumeSpecName: "online-tls") pod "d169a9e0-7689-4170-96fa-f59f171484dd" (UID: "d169a9e0-7689-4170-96fa-f59f171484dd"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 09:01:05.147795 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:01:05.147774 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d169a9e0-7689-4170-96fa-f59f171484dd-kube-api-access-bkqkp" (OuterVolumeSpecName: "kube-api-access-bkqkp") pod "d169a9e0-7689-4170-96fa-f59f171484dd" (UID: "d169a9e0-7689-4170-96fa-f59f171484dd"). InnerVolumeSpecName "kube-api-access-bkqkp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 09:01:05.148987 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:01:05.148967 2572 scope.go:117] "RemoveContainer" containerID="473f855dadd3db9c5daa5ea6843af1f8692f17a2615c7ad0b0c9d3c778ec5d87" Sep 11 09:01:05.154170 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:01:05.154141 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d169a9e0-7689-4170-96fa-f59f171484dd-feast-data" (OuterVolumeSpecName: "feast-data") pod "d169a9e0-7689-4170-96fa-f59f171484dd" (UID: "d169a9e0-7689-4170-96fa-f59f171484dd"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 09:01:05.156614 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:01:05.156596 2572 scope.go:117] "RemoveContainer" containerID="b8a1163aa800f7a5612d35ee3a1adb085c275d427ecbddcc3fc73f185d8e88e9" Sep 11 09:01:05.164993 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:01:05.164975 2572 scope.go:117] "RemoveContainer" containerID="0f61c6e76c03cef07f124af6c3614e5a7dbe56e28c3ed642d0b2376d3d9e7d69" Sep 11 09:01:05.171972 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:01:05.171957 2572 scope.go:117] "RemoveContainer" containerID="cb4806ae0f11e6318bdbb09bcfe72b645be509133c3fe09df519c27970df3498" Sep 11 09:01:05.172251 ip-10-0-128-166 kubenswrapper[2572]: E0911 09:01:05.172210 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb4806ae0f11e6318bdbb09bcfe72b645be509133c3fe09df519c27970df3498\": container with ID starting with cb4806ae0f11e6318bdbb09bcfe72b645be509133c3fe09df519c27970df3498 not found: ID does not exist" containerID="cb4806ae0f11e6318bdbb09bcfe72b645be509133c3fe09df519c27970df3498" Sep 11 09:01:05.172426 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:01:05.172317 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb4806ae0f11e6318bdbb09bcfe72b645be509133c3fe09df519c27970df3498"} err="failed to get container status \"cb4806ae0f11e6318bdbb09bcfe72b645be509133c3fe09df519c27970df3498\": rpc error: code = NotFound desc = could not find container \"cb4806ae0f11e6318bdbb09bcfe72b645be509133c3fe09df519c27970df3498\": container with ID starting with cb4806ae0f11e6318bdbb09bcfe72b645be509133c3fe09df519c27970df3498 not found: ID does not exist" Sep 11 09:01:05.172426 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:01:05.172337 2572 scope.go:117] "RemoveContainer" containerID="ac4be6248302ff451fa131d39fae0bb56a1c61018f257523cf3b5f6bdd37826f" Sep 11 09:01:05.172601 ip-10-0-128-166 kubenswrapper[2572]: E0911 09:01:05.172583 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac4be6248302ff451fa131d39fae0bb56a1c61018f257523cf3b5f6bdd37826f\": container with ID starting with ac4be6248302ff451fa131d39fae0bb56a1c61018f257523cf3b5f6bdd37826f not found: ID does not exist" containerID="ac4be6248302ff451fa131d39fae0bb56a1c61018f257523cf3b5f6bdd37826f" Sep 11 09:01:05.172639 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:01:05.172607 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac4be6248302ff451fa131d39fae0bb56a1c61018f257523cf3b5f6bdd37826f"} err="failed to get container status \"ac4be6248302ff451fa131d39fae0bb56a1c61018f257523cf3b5f6bdd37826f\": rpc error: code = NotFound desc = could not find container \"ac4be6248302ff451fa131d39fae0bb56a1c61018f257523cf3b5f6bdd37826f\": container with ID starting with ac4be6248302ff451fa131d39fae0bb56a1c61018f257523cf3b5f6bdd37826f not found: ID does not exist" Sep 11 09:01:05.172639 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:01:05.172622 2572 scope.go:117] "RemoveContainer" containerID="473f855dadd3db9c5daa5ea6843af1f8692f17a2615c7ad0b0c9d3c778ec5d87" Sep 11 09:01:05.172876 ip-10-0-128-166 kubenswrapper[2572]: E0911 09:01:05.172860 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"473f855dadd3db9c5daa5ea6843af1f8692f17a2615c7ad0b0c9d3c778ec5d87\": container with ID starting with 473f855dadd3db9c5daa5ea6843af1f8692f17a2615c7ad0b0c9d3c778ec5d87 not found: ID does not exist" containerID="473f855dadd3db9c5daa5ea6843af1f8692f17a2615c7ad0b0c9d3c778ec5d87" Sep 11 09:01:05.172922 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:01:05.172881 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"473f855dadd3db9c5daa5ea6843af1f8692f17a2615c7ad0b0c9d3c778ec5d87"} err="failed to get container status \"473f855dadd3db9c5daa5ea6843af1f8692f17a2615c7ad0b0c9d3c778ec5d87\": rpc error: code = NotFound desc = could not find container \"473f855dadd3db9c5daa5ea6843af1f8692f17a2615c7ad0b0c9d3c778ec5d87\": container with ID starting with 473f855dadd3db9c5daa5ea6843af1f8692f17a2615c7ad0b0c9d3c778ec5d87 not found: ID does not exist" Sep 11 09:01:05.172922 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:01:05.172894 2572 scope.go:117] "RemoveContainer" containerID="b8a1163aa800f7a5612d35ee3a1adb085c275d427ecbddcc3fc73f185d8e88e9" Sep 11 09:01:05.173150 ip-10-0-128-166 kubenswrapper[2572]: E0911 09:01:05.173120 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8a1163aa800f7a5612d35ee3a1adb085c275d427ecbddcc3fc73f185d8e88e9\": container with ID starting with b8a1163aa800f7a5612d35ee3a1adb085c275d427ecbddcc3fc73f185d8e88e9 not found: ID does not exist" containerID="b8a1163aa800f7a5612d35ee3a1adb085c275d427ecbddcc3fc73f185d8e88e9" Sep 11 09:01:05.173192 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:01:05.173155 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8a1163aa800f7a5612d35ee3a1adb085c275d427ecbddcc3fc73f185d8e88e9"} err="failed to get container status \"b8a1163aa800f7a5612d35ee3a1adb085c275d427ecbddcc3fc73f185d8e88e9\": rpc error: code = NotFound desc = could not find container \"b8a1163aa800f7a5612d35ee3a1adb085c275d427ecbddcc3fc73f185d8e88e9\": container with ID starting with b8a1163aa800f7a5612d35ee3a1adb085c275d427ecbddcc3fc73f185d8e88e9 not found: ID does not exist" Sep 11 09:01:05.173192 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:01:05.173168 2572 scope.go:117] "RemoveContainer" containerID="0f61c6e76c03cef07f124af6c3614e5a7dbe56e28c3ed642d0b2376d3d9e7d69" Sep 11 09:01:05.173448 ip-10-0-128-166 kubenswrapper[2572]: E0911 09:01:05.173433 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f61c6e76c03cef07f124af6c3614e5a7dbe56e28c3ed642d0b2376d3d9e7d69\": container with ID starting with 0f61c6e76c03cef07f124af6c3614e5a7dbe56e28c3ed642d0b2376d3d9e7d69 not found: ID does not exist" containerID="0f61c6e76c03cef07f124af6c3614e5a7dbe56e28c3ed642d0b2376d3d9e7d69" Sep 11 09:01:05.173512 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:01:05.173450 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f61c6e76c03cef07f124af6c3614e5a7dbe56e28c3ed642d0b2376d3d9e7d69"} err="failed to get container status \"0f61c6e76c03cef07f124af6c3614e5a7dbe56e28c3ed642d0b2376d3d9e7d69\": rpc error: code = NotFound desc = could not find container \"0f61c6e76c03cef07f124af6c3614e5a7dbe56e28c3ed642d0b2376d3d9e7d69\": container with ID starting with 0f61c6e76c03cef07f124af6c3614e5a7dbe56e28c3ed642d0b2376d3d9e7d69 not found: ID does not exist" Sep 11 09:01:05.173512 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:01:05.173463 2572 scope.go:117] "RemoveContainer" containerID="cb4806ae0f11e6318bdbb09bcfe72b645be509133c3fe09df519c27970df3498" Sep 11 09:01:05.173691 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:01:05.173675 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb4806ae0f11e6318bdbb09bcfe72b645be509133c3fe09df519c27970df3498"} err="failed to get container status \"cb4806ae0f11e6318bdbb09bcfe72b645be509133c3fe09df519c27970df3498\": rpc error: code = NotFound desc = could not find container \"cb4806ae0f11e6318bdbb09bcfe72b645be509133c3fe09df519c27970df3498\": container with ID starting with cb4806ae0f11e6318bdbb09bcfe72b645be509133c3fe09df519c27970df3498 not found: ID does not exist" Sep 11 09:01:05.173732 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:01:05.173691 2572 scope.go:117] "RemoveContainer" containerID="ac4be6248302ff451fa131d39fae0bb56a1c61018f257523cf3b5f6bdd37826f" Sep 11 09:01:05.173921 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:01:05.173902 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac4be6248302ff451fa131d39fae0bb56a1c61018f257523cf3b5f6bdd37826f"} err="failed to get container status \"ac4be6248302ff451fa131d39fae0bb56a1c61018f257523cf3b5f6bdd37826f\": rpc error: code = NotFound desc = could not find container \"ac4be6248302ff451fa131d39fae0bb56a1c61018f257523cf3b5f6bdd37826f\": container with ID starting with ac4be6248302ff451fa131d39fae0bb56a1c61018f257523cf3b5f6bdd37826f not found: ID does not exist" Sep 11 09:01:05.173974 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:01:05.173921 2572 scope.go:117] "RemoveContainer" containerID="473f855dadd3db9c5daa5ea6843af1f8692f17a2615c7ad0b0c9d3c778ec5d87" Sep 11 09:01:05.174119 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:01:05.174104 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"473f855dadd3db9c5daa5ea6843af1f8692f17a2615c7ad0b0c9d3c778ec5d87"} err="failed to get container status \"473f855dadd3db9c5daa5ea6843af1f8692f17a2615c7ad0b0c9d3c778ec5d87\": rpc error: code = NotFound desc = could not find container \"473f855dadd3db9c5daa5ea6843af1f8692f17a2615c7ad0b0c9d3c778ec5d87\": container with ID starting with 473f855dadd3db9c5daa5ea6843af1f8692f17a2615c7ad0b0c9d3c778ec5d87 not found: ID does not exist" Sep 11 09:01:05.174163 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:01:05.174119 2572 scope.go:117] "RemoveContainer" containerID="b8a1163aa800f7a5612d35ee3a1adb085c275d427ecbddcc3fc73f185d8e88e9" Sep 11 09:01:05.174350 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:01:05.174331 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8a1163aa800f7a5612d35ee3a1adb085c275d427ecbddcc3fc73f185d8e88e9"} err="failed to get container status \"b8a1163aa800f7a5612d35ee3a1adb085c275d427ecbddcc3fc73f185d8e88e9\": rpc error: code = NotFound desc = could not find container \"b8a1163aa800f7a5612d35ee3a1adb085c275d427ecbddcc3fc73f185d8e88e9\": container with ID starting with b8a1163aa800f7a5612d35ee3a1adb085c275d427ecbddcc3fc73f185d8e88e9 not found: ID does not exist" Sep 11 09:01:05.174350 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:01:05.174349 2572 scope.go:117] "RemoveContainer" containerID="0f61c6e76c03cef07f124af6c3614e5a7dbe56e28c3ed642d0b2376d3d9e7d69" Sep 11 09:01:05.174575 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:01:05.174561 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f61c6e76c03cef07f124af6c3614e5a7dbe56e28c3ed642d0b2376d3d9e7d69"} err="failed to get container status \"0f61c6e76c03cef07f124af6c3614e5a7dbe56e28c3ed642d0b2376d3d9e7d69\": rpc error: code = NotFound desc = could not find container \"0f61c6e76c03cef07f124af6c3614e5a7dbe56e28c3ed642d0b2376d3d9e7d69\": container with ID starting with 0f61c6e76c03cef07f124af6c3614e5a7dbe56e28c3ed642d0b2376d3d9e7d69 not found: ID does not exist" Sep 11 09:01:05.246063 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:01:05.246004 2572 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/d169a9e0-7689-4170-96fa-f59f171484dd-offline-tls\") on node \"ip-10-0-128-166.ec2.internal\" DevicePath \"\"" Sep 11 09:01:05.246063 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:01:05.246029 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkqkp\" (UniqueName: \"kubernetes.io/projected/d169a9e0-7689-4170-96fa-f59f171484dd-kube-api-access-bkqkp\") on node \"ip-10-0-128-166.ec2.internal\" DevicePath \"\"" Sep 11 09:01:05.246063 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:01:05.246039 2572 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/d169a9e0-7689-4170-96fa-f59f171484dd-registry-tls\") on node \"ip-10-0-128-166.ec2.internal\" DevicePath \"\"" Sep 11 09:01:05.246218 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:01:05.246067 2572 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/d169a9e0-7689-4170-96fa-f59f171484dd-online-tls\") on node \"ip-10-0-128-166.ec2.internal\" DevicePath \"\"" Sep 11 09:01:05.246218 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:01:05.246078 2572 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/d169a9e0-7689-4170-96fa-f59f171484dd-feast-data\") on node \"ip-10-0-128-166.ec2.internal\" DevicePath \"\"" Sep 11 09:01:05.452881 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:01:05.452860 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-credit-scoring-797c5c65c5-g6nnz"] Sep 11 09:01:05.456153 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:01:05.456130 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-credit-scoring-797c5c65c5-g6nnz"] Sep 11 09:01:06.251529 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:01:06.251497 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d169a9e0-7689-4170-96fa-f59f171484dd" path="/var/lib/kubelet/pods/d169a9e0-7689-4170-96fa-f59f171484dd/volumes" Sep 11 09:01:20.210445 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:01:20.210369 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-xjs9g_546705a6-1aaf-49fe-9f7f-3fc9e566437b/console-operator/2.log" Sep 11 09:01:20.211723 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:01:20.211702 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-xjs9g_546705a6-1aaf-49fe-9f7f-3fc9e566437b/console-operator/2.log" Sep 11 09:03:17.317781 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:03:17.317750 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-4xqss"] Sep 11 09:03:17.318270 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:03:17.318098 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d169a9e0-7689-4170-96fa-f59f171484dd" containerName="offline" Sep 11 09:03:17.318270 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:03:17.318112 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="d169a9e0-7689-4170-96fa-f59f171484dd" containerName="offline" Sep 11 09:03:17.318270 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:03:17.318128 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d169a9e0-7689-4170-96fa-f59f171484dd" containerName="feast-init" Sep 11 09:03:17.318270 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:03:17.318133 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="d169a9e0-7689-4170-96fa-f59f171484dd" containerName="feast-init" Sep 11 09:03:17.318270 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:03:17.318142 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d169a9e0-7689-4170-96fa-f59f171484dd" containerName="registry" Sep 11 09:03:17.318270 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:03:17.318147 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="d169a9e0-7689-4170-96fa-f59f171484dd" containerName="registry" Sep 11 09:03:17.318270 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:03:17.318154 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="efacc0be-de7d-49b9-b5be-c527de35e4f0" containerName="postgres" Sep 11 09:03:17.318270 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:03:17.318158 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="efacc0be-de7d-49b9-b5be-c527de35e4f0" containerName="postgres" Sep 11 09:03:17.318270 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:03:17.318182 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d169a9e0-7689-4170-96fa-f59f171484dd" containerName="online" Sep 11 09:03:17.318270 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:03:17.318188 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="d169a9e0-7689-4170-96fa-f59f171484dd" containerName="online" Sep 11 09:03:17.318270 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:03:17.318195 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d169a9e0-7689-4170-96fa-f59f171484dd" containerName="feast-apply" Sep 11 09:03:17.318270 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:03:17.318199 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="d169a9e0-7689-4170-96fa-f59f171484dd" containerName="feast-apply" Sep 11 09:03:17.318270 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:03:17.318269 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="d169a9e0-7689-4170-96fa-f59f171484dd" containerName="online" Sep 11 09:03:17.318270 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:03:17.318277 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="d169a9e0-7689-4170-96fa-f59f171484dd" containerName="registry" Sep 11 09:03:17.318694 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:03:17.318283 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="d169a9e0-7689-4170-96fa-f59f171484dd" containerName="offline" Sep 11 09:03:17.318694 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:03:17.318291 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="efacc0be-de7d-49b9-b5be-c527de35e4f0" containerName="postgres" Sep 11 09:03:17.321020 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:03:17.321005 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-69bbd798b6-4xqss" Sep 11 09:03:17.322836 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:03:17.322816 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"kube-root-ca.crt\"" Sep 11 09:03:17.322965 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:03:17.322816 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"openshift-service-ca.crt\"" Sep 11 09:03:17.323436 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:03:17.323417 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"default-dockercfg-fglf8\"" Sep 11 09:03:17.335125 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:03:17.335103 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-4xqss"] Sep 11 09:03:17.393697 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:03:17.393658 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbxpv\" (UniqueName: \"kubernetes.io/projected/b6c264c9-0874-4e91-a901-9bb0ab9a75c1-kube-api-access-lbxpv\") pod \"redis-69bbd798b6-4xqss\" (UID: \"b6c264c9-0874-4e91-a901-9bb0ab9a75c1\") " pod="test-ns-feast-rest/redis-69bbd798b6-4xqss" Sep 11 09:03:17.494935 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:03:17.494899 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbxpv\" (UniqueName: \"kubernetes.io/projected/b6c264c9-0874-4e91-a901-9bb0ab9a75c1-kube-api-access-lbxpv\") pod \"redis-69bbd798b6-4xqss\" (UID: \"b6c264c9-0874-4e91-a901-9bb0ab9a75c1\") " pod="test-ns-feast-rest/redis-69bbd798b6-4xqss" Sep 11 09:03:17.503942 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:03:17.503912 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbxpv\" (UniqueName: \"kubernetes.io/projected/b6c264c9-0874-4e91-a901-9bb0ab9a75c1-kube-api-access-lbxpv\") pod \"redis-69bbd798b6-4xqss\" (UID: \"b6c264c9-0874-4e91-a901-9bb0ab9a75c1\") " pod="test-ns-feast-rest/redis-69bbd798b6-4xqss" Sep 11 09:03:17.639114 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:03:17.639038 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-69bbd798b6-4xqss" Sep 11 09:03:17.907273 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:03:17.907247 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-4xqss"] Sep 11 09:03:17.909999 ip-10-0-128-166 kubenswrapper[2572]: W0911 09:03:17.909972 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb6c264c9_0874_4e91_a901_9bb0ab9a75c1.slice/crio-030c4bd278501a5ef39375e8b98e317d7c6cc5d18219a078cca1065c7e78a340 WatchSource:0}: Error finding container 030c4bd278501a5ef39375e8b98e317d7c6cc5d18219a078cca1065c7e78a340: Status 404 returned error can't find the container with id 030c4bd278501a5ef39375e8b98e317d7c6cc5d18219a078cca1065c7e78a340 Sep 11 09:03:18.640275 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:03:18.640213 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-69bbd798b6-4xqss" event={"ID":"b6c264c9-0874-4e91-a901-9bb0ab9a75c1","Type":"ContainerStarted","Data":"030c4bd278501a5ef39375e8b98e317d7c6cc5d18219a078cca1065c7e78a340"} Sep 11 09:03:19.646557 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:03:19.646527 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-69bbd798b6-4xqss" event={"ID":"b6c264c9-0874-4e91-a901-9bb0ab9a75c1","Type":"ContainerStarted","Data":"fcd9714cb43bb3f7c325ba3c5758adbe60c3d41656ae6cbc838f6c37e4a581a9"} Sep 11 09:03:19.660721 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:03:19.660677 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast-rest/redis-69bbd798b6-4xqss" podStartSLOduration=1.028213361 podStartE2EDuration="2.660656614s" totalImagesPullingTime="1.632443253s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 09:03:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 09:03:19.659677944 +0000 UTC m=+1019.934253642" watchObservedRunningTime="2026-09-11 09:03:19.660656614 +0000 UTC m=+1019.935232311" Sep 11 09:03:22.745201 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:03:22.745164 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-fb9h4"] Sep 11 09:03:22.754551 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:03:22.754514 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-864846d4f5-fb9h4" Sep 11 09:03:22.756807 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:03:22.756782 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"postgres-secret\"" Sep 11 09:03:22.757169 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:03:22.757151 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-fb9h4"] Sep 11 09:03:22.837358 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:03:22.837336 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7bbw\" (UniqueName: \"kubernetes.io/projected/d11d6929-fb34-47a2-8537-dedc99a3d93a-kube-api-access-r7bbw\") pod \"postgres-864846d4f5-fb9h4\" (UID: \"d11d6929-fb34-47a2-8537-dedc99a3d93a\") " pod="test-ns-feast-rest/postgres-864846d4f5-fb9h4" Sep 11 09:03:22.837493 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:03:22.837430 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/d11d6929-fb34-47a2-8537-dedc99a3d93a-postgresdata\") pod \"postgres-864846d4f5-fb9h4\" (UID: \"d11d6929-fb34-47a2-8537-dedc99a3d93a\") " pod="test-ns-feast-rest/postgres-864846d4f5-fb9h4" Sep 11 09:03:22.938606 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:03:22.938575 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/d11d6929-fb34-47a2-8537-dedc99a3d93a-postgresdata\") pod \"postgres-864846d4f5-fb9h4\" (UID: \"d11d6929-fb34-47a2-8537-dedc99a3d93a\") " pod="test-ns-feast-rest/postgres-864846d4f5-fb9h4" Sep 11 09:03:22.938606 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:03:22.938607 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7bbw\" (UniqueName: \"kubernetes.io/projected/d11d6929-fb34-47a2-8537-dedc99a3d93a-kube-api-access-r7bbw\") pod \"postgres-864846d4f5-fb9h4\" (UID: \"d11d6929-fb34-47a2-8537-dedc99a3d93a\") " pod="test-ns-feast-rest/postgres-864846d4f5-fb9h4" Sep 11 09:03:22.938968 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:03:22.938950 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/d11d6929-fb34-47a2-8537-dedc99a3d93a-postgresdata\") pod \"postgres-864846d4f5-fb9h4\" (UID: \"d11d6929-fb34-47a2-8537-dedc99a3d93a\") " pod="test-ns-feast-rest/postgres-864846d4f5-fb9h4" Sep 11 09:03:22.947723 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:03:22.947704 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7bbw\" (UniqueName: \"kubernetes.io/projected/d11d6929-fb34-47a2-8537-dedc99a3d93a-kube-api-access-r7bbw\") pod \"postgres-864846d4f5-fb9h4\" (UID: \"d11d6929-fb34-47a2-8537-dedc99a3d93a\") " pod="test-ns-feast-rest/postgres-864846d4f5-fb9h4" Sep 11 09:03:23.064792 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:03:23.064723 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-864846d4f5-fb9h4" Sep 11 09:03:23.333441 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:03:23.333416 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-fb9h4"] Sep 11 09:03:23.335496 ip-10-0-128-166 kubenswrapper[2572]: W0911 09:03:23.335474 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd11d6929_fb34_47a2_8537_dedc99a3d93a.slice/crio-237d7523e9c7abdadc61d4c5d3c6e141e235f29618357a61abb31c33c5264d4f WatchSource:0}: Error finding container 237d7523e9c7abdadc61d4c5d3c6e141e235f29618357a61abb31c33c5264d4f: Status 404 returned error can't find the container with id 237d7523e9c7abdadc61d4c5d3c6e141e235f29618357a61abb31c33c5264d4f Sep 11 09:03:23.664377 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:03:23.664298 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-864846d4f5-fb9h4" event={"ID":"d11d6929-fb34-47a2-8537-dedc99a3d93a","Type":"ContainerStarted","Data":"a641ae6c5b3f5f964dc589708c0491832614b3c8df2df2d425696fe75f0c387e"} Sep 11 09:03:23.664377 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:03:23.664346 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-864846d4f5-fb9h4" event={"ID":"d11d6929-fb34-47a2-8537-dedc99a3d93a","Type":"ContainerStarted","Data":"237d7523e9c7abdadc61d4c5d3c6e141e235f29618357a61abb31c33c5264d4f"} Sep 11 09:03:23.678810 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:03:23.678754 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast-rest/postgres-864846d4f5-fb9h4" podStartSLOduration=1.6787394089999998 podStartE2EDuration="1.678739409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 09:03:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 09:03:23.676857071 +0000 UTC m=+1023.951432777" watchObservedRunningTime="2026-09-11 09:03:23.678739409 +0000 UTC m=+1023.953315104" Sep 11 09:03:28.339078 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:03:28.339046 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-6fbf9ffc6f-v4jhk"] Sep 11 09:03:28.342717 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:03:28.342696 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-6fbf9ffc6f-v4jhk" Sep 11 09:03:28.369802 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:03:28.369775 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-online-tls\"" Sep 11 09:03:28.370461 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:03:28.370445 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-dockercfg-nr49w\"" Sep 11 09:03:28.370669 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:03:28.370653 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-registry-tls\"" Sep 11 09:03:28.371597 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:03:28.371572 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-intra-comm\"" Sep 11 09:03:28.408542 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:03:28.408516 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-6fbf9ffc6f-v4jhk"] Sep 11 09:03:28.487070 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:03:28.487035 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/aea9836a-fd24-40de-83f7-6a60dcef0c7e-registry-tls\") pod \"feast-credit-scoring-6fbf9ffc6f-v4jhk\" (UID: \"aea9836a-fd24-40de-83f7-6a60dcef0c7e\") " pod="test-ns-feast-rest/feast-credit-scoring-6fbf9ffc6f-v4jhk" Sep 11 09:03:28.487070 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:03:28.487070 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cddbn\" (UniqueName: \"kubernetes.io/projected/aea9836a-fd24-40de-83f7-6a60dcef0c7e-kube-api-access-cddbn\") pod \"feast-credit-scoring-6fbf9ffc6f-v4jhk\" (UID: \"aea9836a-fd24-40de-83f7-6a60dcef0c7e\") " pod="test-ns-feast-rest/feast-credit-scoring-6fbf9ffc6f-v4jhk" Sep 11 09:03:28.487284 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:03:28.487092 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/aea9836a-fd24-40de-83f7-6a60dcef0c7e-online-tls\") pod \"feast-credit-scoring-6fbf9ffc6f-v4jhk\" (UID: \"aea9836a-fd24-40de-83f7-6a60dcef0c7e\") " pod="test-ns-feast-rest/feast-credit-scoring-6fbf9ffc6f-v4jhk" Sep 11 09:03:28.487284 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:03:28.487126 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/aea9836a-fd24-40de-83f7-6a60dcef0c7e-feast-data\") pod \"feast-credit-scoring-6fbf9ffc6f-v4jhk\" (UID: \"aea9836a-fd24-40de-83f7-6a60dcef0c7e\") " pod="test-ns-feast-rest/feast-credit-scoring-6fbf9ffc6f-v4jhk" Sep 11 09:03:28.588035 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:03:28.587994 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/aea9836a-fd24-40de-83f7-6a60dcef0c7e-registry-tls\") pod \"feast-credit-scoring-6fbf9ffc6f-v4jhk\" (UID: \"aea9836a-fd24-40de-83f7-6a60dcef0c7e\") " pod="test-ns-feast-rest/feast-credit-scoring-6fbf9ffc6f-v4jhk" Sep 11 09:03:28.588035 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:03:28.588038 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cddbn\" (UniqueName: \"kubernetes.io/projected/aea9836a-fd24-40de-83f7-6a60dcef0c7e-kube-api-access-cddbn\") pod \"feast-credit-scoring-6fbf9ffc6f-v4jhk\" (UID: \"aea9836a-fd24-40de-83f7-6a60dcef0c7e\") " pod="test-ns-feast-rest/feast-credit-scoring-6fbf9ffc6f-v4jhk" Sep 11 09:03:28.588269 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:03:28.588158 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/aea9836a-fd24-40de-83f7-6a60dcef0c7e-online-tls\") pod \"feast-credit-scoring-6fbf9ffc6f-v4jhk\" (UID: \"aea9836a-fd24-40de-83f7-6a60dcef0c7e\") " pod="test-ns-feast-rest/feast-credit-scoring-6fbf9ffc6f-v4jhk" Sep 11 09:03:28.588269 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:03:28.588188 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/aea9836a-fd24-40de-83f7-6a60dcef0c7e-feast-data\") pod \"feast-credit-scoring-6fbf9ffc6f-v4jhk\" (UID: \"aea9836a-fd24-40de-83f7-6a60dcef0c7e\") " pod="test-ns-feast-rest/feast-credit-scoring-6fbf9ffc6f-v4jhk" Sep 11 09:03:28.588562 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:03:28.588535 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/aea9836a-fd24-40de-83f7-6a60dcef0c7e-feast-data\") pod \"feast-credit-scoring-6fbf9ffc6f-v4jhk\" (UID: \"aea9836a-fd24-40de-83f7-6a60dcef0c7e\") " pod="test-ns-feast-rest/feast-credit-scoring-6fbf9ffc6f-v4jhk" Sep 11 09:03:28.590568 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:03:28.590515 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/aea9836a-fd24-40de-83f7-6a60dcef0c7e-registry-tls\") pod \"feast-credit-scoring-6fbf9ffc6f-v4jhk\" (UID: \"aea9836a-fd24-40de-83f7-6a60dcef0c7e\") " pod="test-ns-feast-rest/feast-credit-scoring-6fbf9ffc6f-v4jhk" Sep 11 09:03:28.590688 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:03:28.590669 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/aea9836a-fd24-40de-83f7-6a60dcef0c7e-online-tls\") pod \"feast-credit-scoring-6fbf9ffc6f-v4jhk\" (UID: \"aea9836a-fd24-40de-83f7-6a60dcef0c7e\") " pod="test-ns-feast-rest/feast-credit-scoring-6fbf9ffc6f-v4jhk" Sep 11 09:03:28.596138 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:03:28.596119 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cddbn\" (UniqueName: \"kubernetes.io/projected/aea9836a-fd24-40de-83f7-6a60dcef0c7e-kube-api-access-cddbn\") pod \"feast-credit-scoring-6fbf9ffc6f-v4jhk\" (UID: \"aea9836a-fd24-40de-83f7-6a60dcef0c7e\") " pod="test-ns-feast-rest/feast-credit-scoring-6fbf9ffc6f-v4jhk" Sep 11 09:03:28.651321 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:03:28.651297 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-6fbf9ffc6f-v4jhk" Sep 11 09:03:28.924713 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:03:28.924622 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-6fbf9ffc6f-v4jhk"] Sep 11 09:03:28.927299 ip-10-0-128-166 kubenswrapper[2572]: W0911 09:03:28.927269 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaea9836a_fd24_40de_83f7_6a60dcef0c7e.slice/crio-3c2dc50c2e54bfdf298b08ce912ac03c1b9d40fa1fcb2201c150198864bac4a2 WatchSource:0}: Error finding container 3c2dc50c2e54bfdf298b08ce912ac03c1b9d40fa1fcb2201c150198864bac4a2: Status 404 returned error can't find the container with id 3c2dc50c2e54bfdf298b08ce912ac03c1b9d40fa1fcb2201c150198864bac4a2 Sep 11 09:03:29.690413 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:03:29.690376 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-6fbf9ffc6f-v4jhk" event={"ID":"aea9836a-fd24-40de-83f7-6a60dcef0c7e","Type":"ContainerStarted","Data":"ac092b1cc84a132e55f42f1c06252ad549e5542979ec2bbe73e22d5f46db1d86"} Sep 11 09:03:29.690784 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:03:29.690427 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-6fbf9ffc6f-v4jhk" event={"ID":"aea9836a-fd24-40de-83f7-6a60dcef0c7e","Type":"ContainerStarted","Data":"3c2dc50c2e54bfdf298b08ce912ac03c1b9d40fa1fcb2201c150198864bac4a2"} Sep 11 09:03:30.695733 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:03:30.695693 2572 generic.go:356] "Generic (PLEG): container finished" podID="aea9836a-fd24-40de-83f7-6a60dcef0c7e" containerID="ac092b1cc84a132e55f42f1c06252ad549e5542979ec2bbe73e22d5f46db1d86" exitCode=0 Sep 11 09:03:30.696108 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:03:30.695752 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-6fbf9ffc6f-v4jhk" event={"ID":"aea9836a-fd24-40de-83f7-6a60dcef0c7e","Type":"ContainerDied","Data":"ac092b1cc84a132e55f42f1c06252ad549e5542979ec2bbe73e22d5f46db1d86"} Sep 11 09:03:31.701534 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:03:31.701496 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-6fbf9ffc6f-v4jhk" event={"ID":"aea9836a-fd24-40de-83f7-6a60dcef0c7e","Type":"ContainerStarted","Data":"67b58f57c1a49e7fe70a2d9e59e2c8e6426e3fc7ebe2b8bea35fb3ebbdd5d664"} Sep 11 09:03:37.728068 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:03:37.728032 2572 generic.go:356] "Generic (PLEG): container finished" podID="aea9836a-fd24-40de-83f7-6a60dcef0c7e" containerID="67b58f57c1a49e7fe70a2d9e59e2c8e6426e3fc7ebe2b8bea35fb3ebbdd5d664" exitCode=0 Sep 11 09:03:37.728485 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:03:37.728087 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-6fbf9ffc6f-v4jhk" event={"ID":"aea9836a-fd24-40de-83f7-6a60dcef0c7e","Type":"ContainerDied","Data":"67b58f57c1a49e7fe70a2d9e59e2c8e6426e3fc7ebe2b8bea35fb3ebbdd5d664"} Sep 11 09:03:38.736786 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:03:38.736754 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-6fbf9ffc6f-v4jhk" event={"ID":"aea9836a-fd24-40de-83f7-6a60dcef0c7e","Type":"ContainerStarted","Data":"708210725af9835ff31aca4364897cc38755bc36ec0f0a111990a0c2d1f3ebaf"} Sep 11 09:03:38.737149 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:03:38.736807 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-6fbf9ffc6f-v4jhk" event={"ID":"aea9836a-fd24-40de-83f7-6a60dcef0c7e","Type":"ContainerStarted","Data":"1092dd438cfc1a67a42c272bde1211321486ed76c0c225e925de5a1f9eb81690"} Sep 11 09:03:38.759374 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:03:38.759308 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast-rest/feast-credit-scoring-6fbf9ffc6f-v4jhk" podStartSLOduration=10.759291946 podStartE2EDuration="10.759291946s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 09:03:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 09:03:38.757653612 +0000 UTC m=+1039.032229307" watchObservedRunningTime="2026-09-11 09:03:38.759291946 +0000 UTC m=+1039.033867642" Sep 11 09:03:40.652245 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:03:40.652186 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-credit-scoring-6fbf9ffc6f-v4jhk" Sep 11 09:03:40.652738 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:03:40.652278 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-credit-scoring-6fbf9ffc6f-v4jhk" Sep 11 09:03:40.654100 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:03:40.654070 2572 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-6fbf9ffc6f-v4jhk" podUID="aea9836a-fd24-40de-83f7-6a60dcef0c7e" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.36:6571: connect: connection refused" Sep 11 09:03:40.654257 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:03:40.654100 2572 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-6fbf9ffc6f-v4jhk" podUID="aea9836a-fd24-40de-83f7-6a60dcef0c7e" containerName="online" probeResult="failure" output="Get \"https://10.132.0.36:6567/health\": dial tcp 10.132.0.36:6567: connect: connection refused" Sep 11 09:03:43.652357 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:03:43.652309 2572 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-6fbf9ffc6f-v4jhk" podUID="aea9836a-fd24-40de-83f7-6a60dcef0c7e" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.36:6571: connect: connection refused" Sep 11 09:03:43.653385 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:03:43.652906 2572 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-6fbf9ffc6f-v4jhk" podUID="aea9836a-fd24-40de-83f7-6a60dcef0c7e" containerName="online" probeResult="failure" output="Get \"https://10.132.0.36:6567/health\": dial tcp 10.132.0.36:6567: connect: connection refused" Sep 11 09:03:46.652308 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:03:46.652254 2572 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-6fbf9ffc6f-v4jhk" podUID="aea9836a-fd24-40de-83f7-6a60dcef0c7e" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.36:6571: connect: connection refused" Sep 11 09:03:46.658383 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:03:46.658351 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-credit-scoring-6fbf9ffc6f-v4jhk" Sep 11 09:03:46.658832 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:03:46.658811 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-credit-scoring-6fbf9ffc6f-v4jhk" Sep 11 09:03:46.661607 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:03:46.661585 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-credit-scoring-6fbf9ffc6f-v4jhk" Sep 11 09:03:48.651917 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:03:48.651878 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-credit-scoring-6fbf9ffc6f-v4jhk" Sep 11 09:03:49.652926 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:03:49.652894 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-credit-scoring-6fbf9ffc6f-v4jhk" Sep 11 09:03:49.790848 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:03:49.790821 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-credit-scoring-6fbf9ffc6f-v4jhk" Sep 11 09:03:54.189951 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:03:54.189914 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-775c6d58bb-2kbhq"] Sep 11 09:03:54.193788 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:03:54.193766 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-775c6d58bb-2kbhq" Sep 11 09:03:54.195763 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:03:54.195729 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-online-tls\"" Sep 11 09:03:54.195763 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:03:54.195748 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-dockercfg-khrfc\"" Sep 11 09:03:54.195763 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:03:54.195734 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-registry-tls\"" Sep 11 09:03:54.196041 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:03:54.196028 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-intra-comm\"" Sep 11 09:03:54.206513 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:03:54.206493 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/76e60db5-7a8e-46ba-87a0-becb0dcb32a8-feast-data\") pod \"feast-driver-ranking-775c6d58bb-2kbhq\" (UID: \"76e60db5-7a8e-46ba-87a0-becb0dcb32a8\") " pod="test-ns-feast-rest/feast-driver-ranking-775c6d58bb-2kbhq" Sep 11 09:03:54.206639 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:03:54.206535 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/76e60db5-7a8e-46ba-87a0-becb0dcb32a8-online-tls\") pod \"feast-driver-ranking-775c6d58bb-2kbhq\" (UID: \"76e60db5-7a8e-46ba-87a0-becb0dcb32a8\") " pod="test-ns-feast-rest/feast-driver-ranking-775c6d58bb-2kbhq" Sep 11 09:03:54.206639 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:03:54.206563 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-775c6d58bb-2kbhq"] Sep 11 09:03:54.206639 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:03:54.206600 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/76e60db5-7a8e-46ba-87a0-becb0dcb32a8-registry-tls\") pod \"feast-driver-ranking-775c6d58bb-2kbhq\" (UID: \"76e60db5-7a8e-46ba-87a0-becb0dcb32a8\") " pod="test-ns-feast-rest/feast-driver-ranking-775c6d58bb-2kbhq" Sep 11 09:03:54.206781 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:03:54.206683 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zztmf\" (UniqueName: \"kubernetes.io/projected/76e60db5-7a8e-46ba-87a0-becb0dcb32a8-kube-api-access-zztmf\") pod \"feast-driver-ranking-775c6d58bb-2kbhq\" (UID: \"76e60db5-7a8e-46ba-87a0-becb0dcb32a8\") " pod="test-ns-feast-rest/feast-driver-ranking-775c6d58bb-2kbhq" Sep 11 09:03:54.307581 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:03:54.307545 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/76e60db5-7a8e-46ba-87a0-becb0dcb32a8-online-tls\") pod \"feast-driver-ranking-775c6d58bb-2kbhq\" (UID: \"76e60db5-7a8e-46ba-87a0-becb0dcb32a8\") " pod="test-ns-feast-rest/feast-driver-ranking-775c6d58bb-2kbhq" Sep 11 09:03:54.307769 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:03:54.307593 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/76e60db5-7a8e-46ba-87a0-becb0dcb32a8-registry-tls\") pod \"feast-driver-ranking-775c6d58bb-2kbhq\" (UID: \"76e60db5-7a8e-46ba-87a0-becb0dcb32a8\") " pod="test-ns-feast-rest/feast-driver-ranking-775c6d58bb-2kbhq" Sep 11 09:03:54.307769 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:03:54.307658 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zztmf\" (UniqueName: \"kubernetes.io/projected/76e60db5-7a8e-46ba-87a0-becb0dcb32a8-kube-api-access-zztmf\") pod \"feast-driver-ranking-775c6d58bb-2kbhq\" (UID: \"76e60db5-7a8e-46ba-87a0-becb0dcb32a8\") " pod="test-ns-feast-rest/feast-driver-ranking-775c6d58bb-2kbhq" Sep 11 09:03:54.307769 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:03:54.307698 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/76e60db5-7a8e-46ba-87a0-becb0dcb32a8-feast-data\") pod \"feast-driver-ranking-775c6d58bb-2kbhq\" (UID: \"76e60db5-7a8e-46ba-87a0-becb0dcb32a8\") " pod="test-ns-feast-rest/feast-driver-ranking-775c6d58bb-2kbhq" Sep 11 09:03:54.307769 ip-10-0-128-166 kubenswrapper[2572]: E0911 09:03:54.307710 2572 secret.go:189] Couldn't get secret test-ns-feast-rest/feast-driver-ranking-online-tls: secret "feast-driver-ranking-online-tls" not found Sep 11 09:03:54.308017 ip-10-0-128-166 kubenswrapper[2572]: E0911 09:03:54.307781 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76e60db5-7a8e-46ba-87a0-becb0dcb32a8-online-tls podName:76e60db5-7a8e-46ba-87a0-becb0dcb32a8 nodeName:}" failed. No retries permitted until 2026-09-11 09:03:54.807759736 +0000 UTC m=+1055.082335424 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/76e60db5-7a8e-46ba-87a0-becb0dcb32a8-online-tls") pod "feast-driver-ranking-775c6d58bb-2kbhq" (UID: "76e60db5-7a8e-46ba-87a0-becb0dcb32a8") : secret "feast-driver-ranking-online-tls" not found Sep 11 09:03:54.308090 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:03:54.308073 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/76e60db5-7a8e-46ba-87a0-becb0dcb32a8-feast-data\") pod \"feast-driver-ranking-775c6d58bb-2kbhq\" (UID: \"76e60db5-7a8e-46ba-87a0-becb0dcb32a8\") " pod="test-ns-feast-rest/feast-driver-ranking-775c6d58bb-2kbhq" Sep 11 09:03:54.310172 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:03:54.310156 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/76e60db5-7a8e-46ba-87a0-becb0dcb32a8-registry-tls\") pod \"feast-driver-ranking-775c6d58bb-2kbhq\" (UID: \"76e60db5-7a8e-46ba-87a0-becb0dcb32a8\") " pod="test-ns-feast-rest/feast-driver-ranking-775c6d58bb-2kbhq" Sep 11 09:03:54.316047 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:03:54.316022 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zztmf\" (UniqueName: \"kubernetes.io/projected/76e60db5-7a8e-46ba-87a0-becb0dcb32a8-kube-api-access-zztmf\") pod \"feast-driver-ranking-775c6d58bb-2kbhq\" (UID: \"76e60db5-7a8e-46ba-87a0-becb0dcb32a8\") " pod="test-ns-feast-rest/feast-driver-ranking-775c6d58bb-2kbhq" Sep 11 09:03:54.811011 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:03:54.810976 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/76e60db5-7a8e-46ba-87a0-becb0dcb32a8-online-tls\") pod \"feast-driver-ranking-775c6d58bb-2kbhq\" (UID: \"76e60db5-7a8e-46ba-87a0-becb0dcb32a8\") " pod="test-ns-feast-rest/feast-driver-ranking-775c6d58bb-2kbhq" Sep 11 09:03:54.813623 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:03:54.813600 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/76e60db5-7a8e-46ba-87a0-becb0dcb32a8-online-tls\") pod \"feast-driver-ranking-775c6d58bb-2kbhq\" (UID: \"76e60db5-7a8e-46ba-87a0-becb0dcb32a8\") " pod="test-ns-feast-rest/feast-driver-ranking-775c6d58bb-2kbhq" Sep 11 09:03:55.103136 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:03:55.103043 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-775c6d58bb-2kbhq" Sep 11 09:03:55.374729 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:03:55.374699 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-775c6d58bb-2kbhq"] Sep 11 09:03:55.375693 ip-10-0-128-166 kubenswrapper[2572]: W0911 09:03:55.375668 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76e60db5_7a8e_46ba_87a0_becb0dcb32a8.slice/crio-15dd4fecba6b7018389c677868e2b15ed622ac2af02325c7bf5dd7b1d31bd664 WatchSource:0}: Error finding container 15dd4fecba6b7018389c677868e2b15ed622ac2af02325c7bf5dd7b1d31bd664: Status 404 returned error can't find the container with id 15dd4fecba6b7018389c677868e2b15ed622ac2af02325c7bf5dd7b1d31bd664 Sep 11 09:03:55.816939 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:03:55.816903 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-775c6d58bb-2kbhq" event={"ID":"76e60db5-7a8e-46ba-87a0-becb0dcb32a8","Type":"ContainerStarted","Data":"a2258d21fa12bf64161b9354b1a712465d5335b4d68c38aa29766020e7de4cac"} Sep 11 09:03:55.817107 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:03:55.816953 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-775c6d58bb-2kbhq" event={"ID":"76e60db5-7a8e-46ba-87a0-becb0dcb32a8","Type":"ContainerStarted","Data":"15dd4fecba6b7018389c677868e2b15ed622ac2af02325c7bf5dd7b1d31bd664"} Sep 11 09:04:00.840105 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:04:00.840070 2572 generic.go:356] "Generic (PLEG): container finished" podID="76e60db5-7a8e-46ba-87a0-becb0dcb32a8" containerID="a2258d21fa12bf64161b9354b1a712465d5335b4d68c38aa29766020e7de4cac" exitCode=0 Sep 11 09:04:00.840105 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:04:00.840117 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-775c6d58bb-2kbhq" event={"ID":"76e60db5-7a8e-46ba-87a0-becb0dcb32a8","Type":"ContainerDied","Data":"a2258d21fa12bf64161b9354b1a712465d5335b4d68c38aa29766020e7de4cac"} Sep 11 09:04:01.847025 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:04:01.846992 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-775c6d58bb-2kbhq" event={"ID":"76e60db5-7a8e-46ba-87a0-becb0dcb32a8","Type":"ContainerStarted","Data":"260aa15790613111e946036d9be4504661a487b5a6f5aa8805818eb1b5355335"} Sep 11 09:04:07.877012 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:04:07.876920 2572 generic.go:356] "Generic (PLEG): container finished" podID="76e60db5-7a8e-46ba-87a0-becb0dcb32a8" containerID="260aa15790613111e946036d9be4504661a487b5a6f5aa8805818eb1b5355335" exitCode=0 Sep 11 09:04:07.877012 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:04:07.876992 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-775c6d58bb-2kbhq" event={"ID":"76e60db5-7a8e-46ba-87a0-becb0dcb32a8","Type":"ContainerDied","Data":"260aa15790613111e946036d9be4504661a487b5a6f5aa8805818eb1b5355335"} Sep 11 09:04:08.884279 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:04:08.884246 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-775c6d58bb-2kbhq" event={"ID":"76e60db5-7a8e-46ba-87a0-becb0dcb32a8","Type":"ContainerStarted","Data":"eb09066bd01d3c543918a263f9f84a71c5f1c98036c9ce69edbb6021ae25c88e"} Sep 11 09:04:08.884781 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:04:08.884302 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-775c6d58bb-2kbhq" event={"ID":"76e60db5-7a8e-46ba-87a0-becb0dcb32a8","Type":"ContainerStarted","Data":"0cdb8c0a2a9b834dff89429f7c85a43ae0c292de0c64800dd1e34defb20a2b1b"} Sep 11 09:04:08.904597 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:04:08.904535 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast-rest/feast-driver-ranking-775c6d58bb-2kbhq" podStartSLOduration=14.904514952 podStartE2EDuration="14.904514952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 09:03:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 09:04:08.902992842 +0000 UTC m=+1069.177568538" watchObservedRunningTime="2026-09-11 09:04:08.904514952 +0000 UTC m=+1069.179090650" Sep 11 09:04:10.104098 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:04:10.104054 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-driver-ranking-775c6d58bb-2kbhq" Sep 11 09:04:10.104595 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:04:10.104142 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-driver-ranking-775c6d58bb-2kbhq" Sep 11 09:04:10.105431 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:04:10.105399 2572 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-775c6d58bb-2kbhq" podUID="76e60db5-7a8e-46ba-87a0-becb0dcb32a8" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.37:6571: connect: connection refused" Sep 11 09:04:10.105523 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:04:10.105445 2572 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-775c6d58bb-2kbhq" podUID="76e60db5-7a8e-46ba-87a0-becb0dcb32a8" containerName="online" probeResult="failure" output="Get \"https://10.132.0.37:6567/health\": dial tcp 10.132.0.37:6567: connect: connection refused" Sep 11 09:04:13.103676 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:04:13.103575 2572 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-775c6d58bb-2kbhq" podUID="76e60db5-7a8e-46ba-87a0-becb0dcb32a8" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.37:6571: connect: connection refused" Sep 11 09:04:13.103676 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:04:13.103590 2572 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-775c6d58bb-2kbhq" podUID="76e60db5-7a8e-46ba-87a0-becb0dcb32a8" containerName="online" probeResult="failure" output="Get \"https://10.132.0.37:6567/health\": dial tcp 10.132.0.37:6567: connect: connection refused" Sep 11 09:04:15.104096 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:04:15.104060 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-driver-ranking-775c6d58bb-2kbhq" Sep 11 09:04:15.104488 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:04:15.104126 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-driver-ranking-775c6d58bb-2kbhq" Sep 11 09:04:16.104042 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:04:16.103997 2572 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-775c6d58bb-2kbhq" podUID="76e60db5-7a8e-46ba-87a0-becb0dcb32a8" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.37:6571: connect: connection refused" Sep 11 09:04:16.110401 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:04:16.110368 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-driver-ranking-775c6d58bb-2kbhq" Sep 11 09:04:16.114964 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:04:16.114940 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-driver-ranking-775c6d58bb-2kbhq" Sep 11 09:04:19.103838 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:04:19.103793 2572 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-775c6d58bb-2kbhq" podUID="76e60db5-7a8e-46ba-87a0-becb0dcb32a8" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.37:6571: connect: connection refused" Sep 11 09:04:22.104321 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:04:22.104288 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-driver-ranking-775c6d58bb-2kbhq" Sep 11 09:04:22.105111 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:04:22.105093 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-driver-ranking-775c6d58bb-2kbhq" Sep 11 09:05:42.290938 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:05:42.290858 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-58764ff88f-954hx"] Sep 11 09:05:42.291439 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:05:42.291053 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="feast-operator-system/feast-operator-controller-manager-58764ff88f-954hx" podUID="5dc2a815-4d2a-4c3b-aa18-735eb0a414cf" containerName="manager" containerID="cri-o://012012b2abdf56e1a76506234c309b9d9a8521b194263d9800215c7cb161a759" gracePeriod=10 Sep 11 09:05:42.558211 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:05:42.558182 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-58764ff88f-954hx" Sep 11 09:05:42.715576 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:05:42.715549 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-frkwn\" (UniqueName: \"kubernetes.io/projected/5dc2a815-4d2a-4c3b-aa18-735eb0a414cf-kube-api-access-frkwn\") pod \"5dc2a815-4d2a-4c3b-aa18-735eb0a414cf\" (UID: \"5dc2a815-4d2a-4c3b-aa18-735eb0a414cf\") " Sep 11 09:05:42.717744 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:05:42.717716 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dc2a815-4d2a-4c3b-aa18-735eb0a414cf-kube-api-access-frkwn" (OuterVolumeSpecName: "kube-api-access-frkwn") pod "5dc2a815-4d2a-4c3b-aa18-735eb0a414cf" (UID: "5dc2a815-4d2a-4c3b-aa18-735eb0a414cf"). InnerVolumeSpecName "kube-api-access-frkwn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 09:05:42.735654 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:05:42.735624 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-6fbf9ffc6f-v4jhk"] Sep 11 09:05:42.735910 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:05:42.735883 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast-rest/feast-credit-scoring-6fbf9ffc6f-v4jhk" podUID="aea9836a-fd24-40de-83f7-6a60dcef0c7e" containerName="registry" containerID="cri-o://1092dd438cfc1a67a42c272bde1211321486ed76c0c225e925de5a1f9eb81690" gracePeriod=30 Sep 11 09:05:42.735987 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:05:42.735932 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast-rest/feast-credit-scoring-6fbf9ffc6f-v4jhk" podUID="aea9836a-fd24-40de-83f7-6a60dcef0c7e" containerName="online" containerID="cri-o://708210725af9835ff31aca4364897cc38755bc36ec0f0a111990a0c2d1f3ebaf" gracePeriod=30 Sep 11 09:05:42.737388 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:05:42.737364 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-775c6d58bb-2kbhq"] Sep 11 09:05:42.737578 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:05:42.737554 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast-rest/feast-driver-ranking-775c6d58bb-2kbhq" podUID="76e60db5-7a8e-46ba-87a0-becb0dcb32a8" containerName="registry" containerID="cri-o://0cdb8c0a2a9b834dff89429f7c85a43ae0c292de0c64800dd1e34defb20a2b1b" gracePeriod=30 Sep 11 09:05:42.737694 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:05:42.737593 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast-rest/feast-driver-ranking-775c6d58bb-2kbhq" podUID="76e60db5-7a8e-46ba-87a0-becb0dcb32a8" containerName="online" containerID="cri-o://eb09066bd01d3c543918a263f9f84a71c5f1c98036c9ce69edbb6021ae25c88e" gracePeriod=30 Sep 11 09:05:42.816753 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:05:42.816682 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frkwn\" (UniqueName: \"kubernetes.io/projected/5dc2a815-4d2a-4c3b-aa18-735eb0a414cf-kube-api-access-frkwn\") on node \"ip-10-0-128-166.ec2.internal\" DevicePath \"\"" Sep 11 09:05:43.323154 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:05:43.323117 2572 generic.go:356] "Generic (PLEG): container finished" podID="5dc2a815-4d2a-4c3b-aa18-735eb0a414cf" containerID="012012b2abdf56e1a76506234c309b9d9a8521b194263d9800215c7cb161a759" exitCode=0 Sep 11 09:05:43.323670 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:05:43.323178 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-58764ff88f-954hx" event={"ID":"5dc2a815-4d2a-4c3b-aa18-735eb0a414cf","Type":"ContainerDied","Data":"012012b2abdf56e1a76506234c309b9d9a8521b194263d9800215c7cb161a759"} Sep 11 09:05:43.323670 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:05:43.323193 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-58764ff88f-954hx" Sep 11 09:05:43.323670 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:05:43.323241 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-58764ff88f-954hx" event={"ID":"5dc2a815-4d2a-4c3b-aa18-735eb0a414cf","Type":"ContainerDied","Data":"d62810db6259609f4eee8e644020b4c101a686bc51988e18fb1423ec51c0c084"} Sep 11 09:05:43.323670 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:05:43.323269 2572 scope.go:117] "RemoveContainer" containerID="012012b2abdf56e1a76506234c309b9d9a8521b194263d9800215c7cb161a759" Sep 11 09:05:43.333352 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:05:43.332943 2572 scope.go:117] "RemoveContainer" containerID="012012b2abdf56e1a76506234c309b9d9a8521b194263d9800215c7cb161a759" Sep 11 09:05:43.333352 ip-10-0-128-166 kubenswrapper[2572]: E0911 09:05:43.333249 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"012012b2abdf56e1a76506234c309b9d9a8521b194263d9800215c7cb161a759\": container with ID starting with 012012b2abdf56e1a76506234c309b9d9a8521b194263d9800215c7cb161a759 not found: ID does not exist" containerID="012012b2abdf56e1a76506234c309b9d9a8521b194263d9800215c7cb161a759" Sep 11 09:05:43.333352 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:05:43.333283 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"012012b2abdf56e1a76506234c309b9d9a8521b194263d9800215c7cb161a759"} err="failed to get container status \"012012b2abdf56e1a76506234c309b9d9a8521b194263d9800215c7cb161a759\": rpc error: code = NotFound desc = could not find container \"012012b2abdf56e1a76506234c309b9d9a8521b194263d9800215c7cb161a759\": container with ID starting with 012012b2abdf56e1a76506234c309b9d9a8521b194263d9800215c7cb161a759 not found: ID does not exist" Sep 11 09:05:43.346284 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:05:43.346258 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-58764ff88f-954hx"] Sep 11 09:05:43.353030 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:05:43.353004 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-58764ff88f-954hx"] Sep 11 09:05:44.252183 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:05:44.252152 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5dc2a815-4d2a-4c3b-aa18-735eb0a414cf" path="/var/lib/kubelet/pods/5dc2a815-4d2a-4c3b-aa18-735eb0a414cf/volumes" Sep 11 09:05:44.329984 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:05:44.329958 2572 generic.go:356] "Generic (PLEG): container finished" podID="aea9836a-fd24-40de-83f7-6a60dcef0c7e" containerID="708210725af9835ff31aca4364897cc38755bc36ec0f0a111990a0c2d1f3ebaf" exitCode=0 Sep 11 09:05:44.330405 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:05:44.330030 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-6fbf9ffc6f-v4jhk" event={"ID":"aea9836a-fd24-40de-83f7-6a60dcef0c7e","Type":"ContainerDied","Data":"708210725af9835ff31aca4364897cc38755bc36ec0f0a111990a0c2d1f3ebaf"} Sep 11 09:05:44.332424 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:05:44.332405 2572 generic.go:356] "Generic (PLEG): container finished" podID="76e60db5-7a8e-46ba-87a0-becb0dcb32a8" containerID="eb09066bd01d3c543918a263f9f84a71c5f1c98036c9ce69edbb6021ae25c88e" exitCode=0 Sep 11 09:05:44.332503 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:05:44.332439 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-775c6d58bb-2kbhq" event={"ID":"76e60db5-7a8e-46ba-87a0-becb0dcb32a8","Type":"ContainerDied","Data":"eb09066bd01d3c543918a263f9f84a71c5f1c98036c9ce69edbb6021ae25c88e"} Sep 11 09:05:44.648244 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:05:44.648149 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-fb9h4"] Sep 11 09:05:44.648389 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:05:44.648322 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast-rest/postgres-864846d4f5-fb9h4" podUID="d11d6929-fb34-47a2-8537-dedc99a3d93a" containerName="postgres" containerID="cri-o://a641ae6c5b3f5f964dc589708c0491832614b3c8df2df2d425696fe75f0c387e" gracePeriod=30 Sep 11 09:05:44.661693 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:05:44.661668 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-4xqss"] Sep 11 09:05:44.661842 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:05:44.661824 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast-rest/redis-69bbd798b6-4xqss" podUID="b6c264c9-0874-4e91-a901-9bb0ab9a75c1" containerName="redis" containerID="cri-o://fcd9714cb43bb3f7c325ba3c5758adbe60c3d41656ae6cbc838f6c37e4a581a9" gracePeriod=30 Sep 11 09:05:44.918396 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:05:44.918371 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-864846d4f5-fb9h4" Sep 11 09:05:44.921144 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:05:44.921127 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-69bbd798b6-4xqss" Sep 11 09:05:45.032405 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:05:45.032378 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lbxpv\" (UniqueName: \"kubernetes.io/projected/b6c264c9-0874-4e91-a901-9bb0ab9a75c1-kube-api-access-lbxpv\") pod \"b6c264c9-0874-4e91-a901-9bb0ab9a75c1\" (UID: \"b6c264c9-0874-4e91-a901-9bb0ab9a75c1\") " Sep 11 09:05:45.032591 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:05:45.032490 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r7bbw\" (UniqueName: \"kubernetes.io/projected/d11d6929-fb34-47a2-8537-dedc99a3d93a-kube-api-access-r7bbw\") pod \"d11d6929-fb34-47a2-8537-dedc99a3d93a\" (UID: \"d11d6929-fb34-47a2-8537-dedc99a3d93a\") " Sep 11 09:05:45.032591 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:05:45.032530 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/d11d6929-fb34-47a2-8537-dedc99a3d93a-postgresdata\") pod \"d11d6929-fb34-47a2-8537-dedc99a3d93a\" (UID: \"d11d6929-fb34-47a2-8537-dedc99a3d93a\") " Sep 11 09:05:45.034655 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:05:45.034627 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6c264c9-0874-4e91-a901-9bb0ab9a75c1-kube-api-access-lbxpv" (OuterVolumeSpecName: "kube-api-access-lbxpv") pod "b6c264c9-0874-4e91-a901-9bb0ab9a75c1" (UID: "b6c264c9-0874-4e91-a901-9bb0ab9a75c1"). InnerVolumeSpecName "kube-api-access-lbxpv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 09:05:45.034795 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:05:45.034653 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d11d6929-fb34-47a2-8537-dedc99a3d93a-kube-api-access-r7bbw" (OuterVolumeSpecName: "kube-api-access-r7bbw") pod "d11d6929-fb34-47a2-8537-dedc99a3d93a" (UID: "d11d6929-fb34-47a2-8537-dedc99a3d93a"). InnerVolumeSpecName "kube-api-access-r7bbw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 09:05:45.061642 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:05:45.061611 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d11d6929-fb34-47a2-8537-dedc99a3d93a-postgresdata" (OuterVolumeSpecName: "postgresdata") pod "d11d6929-fb34-47a2-8537-dedc99a3d93a" (UID: "d11d6929-fb34-47a2-8537-dedc99a3d93a"). InnerVolumeSpecName "postgresdata". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 09:05:45.133371 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:05:45.133345 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7bbw\" (UniqueName: \"kubernetes.io/projected/d11d6929-fb34-47a2-8537-dedc99a3d93a-kube-api-access-r7bbw\") on node \"ip-10-0-128-166.ec2.internal\" DevicePath \"\"" Sep 11 09:05:45.133371 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:05:45.133369 2572 reconciler_common.go:299] "Volume detached for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/d11d6929-fb34-47a2-8537-dedc99a3d93a-postgresdata\") on node \"ip-10-0-128-166.ec2.internal\" DevicePath \"\"" Sep 11 09:05:45.133521 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:05:45.133378 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbxpv\" (UniqueName: \"kubernetes.io/projected/b6c264c9-0874-4e91-a901-9bb0ab9a75c1-kube-api-access-lbxpv\") on node \"ip-10-0-128-166.ec2.internal\" DevicePath \"\"" Sep 11 09:05:45.337742 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:05:45.337712 2572 generic.go:356] "Generic (PLEG): container finished" podID="b6c264c9-0874-4e91-a901-9bb0ab9a75c1" containerID="fcd9714cb43bb3f7c325ba3c5758adbe60c3d41656ae6cbc838f6c37e4a581a9" exitCode=0 Sep 11 09:05:45.338178 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:05:45.337753 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-69bbd798b6-4xqss" Sep 11 09:05:45.338178 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:05:45.337793 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-69bbd798b6-4xqss" event={"ID":"b6c264c9-0874-4e91-a901-9bb0ab9a75c1","Type":"ContainerDied","Data":"fcd9714cb43bb3f7c325ba3c5758adbe60c3d41656ae6cbc838f6c37e4a581a9"} Sep 11 09:05:45.338178 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:05:45.337825 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-69bbd798b6-4xqss" event={"ID":"b6c264c9-0874-4e91-a901-9bb0ab9a75c1","Type":"ContainerDied","Data":"030c4bd278501a5ef39375e8b98e317d7c6cc5d18219a078cca1065c7e78a340"} Sep 11 09:05:45.338178 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:05:45.337846 2572 scope.go:117] "RemoveContainer" containerID="fcd9714cb43bb3f7c325ba3c5758adbe60c3d41656ae6cbc838f6c37e4a581a9" Sep 11 09:05:45.339068 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:05:45.339045 2572 generic.go:356] "Generic (PLEG): container finished" podID="d11d6929-fb34-47a2-8537-dedc99a3d93a" containerID="a641ae6c5b3f5f964dc589708c0491832614b3c8df2df2d425696fe75f0c387e" exitCode=0 Sep 11 09:05:45.339183 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:05:45.339076 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-864846d4f5-fb9h4" Sep 11 09:05:45.339183 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:05:45.339095 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-864846d4f5-fb9h4" event={"ID":"d11d6929-fb34-47a2-8537-dedc99a3d93a","Type":"ContainerDied","Data":"a641ae6c5b3f5f964dc589708c0491832614b3c8df2df2d425696fe75f0c387e"} Sep 11 09:05:45.339183 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:05:45.339118 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-864846d4f5-fb9h4" event={"ID":"d11d6929-fb34-47a2-8537-dedc99a3d93a","Type":"ContainerDied","Data":"237d7523e9c7abdadc61d4c5d3c6e141e235f29618357a61abb31c33c5264d4f"} Sep 11 09:05:45.346429 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:05:45.346410 2572 scope.go:117] "RemoveContainer" containerID="fcd9714cb43bb3f7c325ba3c5758adbe60c3d41656ae6cbc838f6c37e4a581a9" Sep 11 09:05:45.346677 ip-10-0-128-166 kubenswrapper[2572]: E0911 09:05:45.346660 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcd9714cb43bb3f7c325ba3c5758adbe60c3d41656ae6cbc838f6c37e4a581a9\": container with ID starting with fcd9714cb43bb3f7c325ba3c5758adbe60c3d41656ae6cbc838f6c37e4a581a9 not found: ID does not exist" containerID="fcd9714cb43bb3f7c325ba3c5758adbe60c3d41656ae6cbc838f6c37e4a581a9" Sep 11 09:05:45.346731 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:05:45.346683 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcd9714cb43bb3f7c325ba3c5758adbe60c3d41656ae6cbc838f6c37e4a581a9"} err="failed to get container status \"fcd9714cb43bb3f7c325ba3c5758adbe60c3d41656ae6cbc838f6c37e4a581a9\": rpc error: code = NotFound desc = could not find container \"fcd9714cb43bb3f7c325ba3c5758adbe60c3d41656ae6cbc838f6c37e4a581a9\": container with ID starting with fcd9714cb43bb3f7c325ba3c5758adbe60c3d41656ae6cbc838f6c37e4a581a9 not found: ID does not exist" Sep 11 09:05:45.346731 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:05:45.346698 2572 scope.go:117] "RemoveContainer" containerID="a641ae6c5b3f5f964dc589708c0491832614b3c8df2df2d425696fe75f0c387e" Sep 11 09:05:45.354246 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:05:45.354216 2572 scope.go:117] "RemoveContainer" containerID="a641ae6c5b3f5f964dc589708c0491832614b3c8df2df2d425696fe75f0c387e" Sep 11 09:05:45.354484 ip-10-0-128-166 kubenswrapper[2572]: E0911 09:05:45.354464 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a641ae6c5b3f5f964dc589708c0491832614b3c8df2df2d425696fe75f0c387e\": container with ID starting with a641ae6c5b3f5f964dc589708c0491832614b3c8df2df2d425696fe75f0c387e not found: ID does not exist" containerID="a641ae6c5b3f5f964dc589708c0491832614b3c8df2df2d425696fe75f0c387e" Sep 11 09:05:45.354545 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:05:45.354489 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a641ae6c5b3f5f964dc589708c0491832614b3c8df2df2d425696fe75f0c387e"} err="failed to get container status \"a641ae6c5b3f5f964dc589708c0491832614b3c8df2df2d425696fe75f0c387e\": rpc error: code = NotFound desc = could not find container \"a641ae6c5b3f5f964dc589708c0491832614b3c8df2df2d425696fe75f0c387e\": container with ID starting with a641ae6c5b3f5f964dc589708c0491832614b3c8df2df2d425696fe75f0c387e not found: ID does not exist" Sep 11 09:05:45.363013 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:05:45.362992 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-4xqss"] Sep 11 09:05:45.366015 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:05:45.365996 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-4xqss"] Sep 11 09:05:45.378588 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:05:45.378565 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-fb9h4"] Sep 11 09:05:45.382892 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:05:45.382874 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-fb9h4"] Sep 11 09:05:46.111617 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:05:46.111577 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-driver-ranking-775c6d58bb-2kbhq" podUID="76e60db5-7a8e-46ba-87a0-becb0dcb32a8" containerName="online" probeResult="failure" output="Get \"https://10.132.0.37:6567/health\": dial tcp 10.132.0.37:6567: connect: connection refused" Sep 11 09:05:46.252089 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:05:46.252056 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b6c264c9-0874-4e91-a901-9bb0ab9a75c1" path="/var/lib/kubelet/pods/b6c264c9-0874-4e91-a901-9bb0ab9a75c1/volumes" Sep 11 09:05:46.252401 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:05:46.252388 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d11d6929-fb34-47a2-8537-dedc99a3d93a" path="/var/lib/kubelet/pods/d11d6929-fb34-47a2-8537-dedc99a3d93a/volumes" Sep 11 09:05:46.659856 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:05:46.659815 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-credit-scoring-6fbf9ffc6f-v4jhk" podUID="aea9836a-fd24-40de-83f7-6a60dcef0c7e" containerName="online" probeResult="failure" output="Get \"https://10.132.0.36:6567/health\": dial tcp 10.132.0.36:6567: connect: connection refused" Sep 11 09:05:56.110967 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:05:56.110926 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-driver-ranking-775c6d58bb-2kbhq" podUID="76e60db5-7a8e-46ba-87a0-becb0dcb32a8" containerName="online" probeResult="failure" output="Get \"https://10.132.0.37:6567/health\": dial tcp 10.132.0.37:6567: connect: connection refused" Sep 11 09:05:56.659924 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:05:56.659883 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-credit-scoring-6fbf9ffc6f-v4jhk" podUID="aea9836a-fd24-40de-83f7-6a60dcef0c7e" containerName="online" probeResult="failure" output="Get \"https://10.132.0.36:6567/health\": dial tcp 10.132.0.36:6567: connect: connection refused" Sep 11 09:06:06.111596 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:06:06.111542 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-driver-ranking-775c6d58bb-2kbhq" podUID="76e60db5-7a8e-46ba-87a0-becb0dcb32a8" containerName="online" probeResult="failure" output="Get \"https://10.132.0.37:6567/health\": dial tcp 10.132.0.37:6567: connect: connection refused" Sep 11 09:06:06.111969 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:06:06.111714 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-driver-ranking-775c6d58bb-2kbhq" Sep 11 09:06:06.660546 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:06:06.659816 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-credit-scoring-6fbf9ffc6f-v4jhk" podUID="aea9836a-fd24-40de-83f7-6a60dcef0c7e" containerName="online" probeResult="failure" output="Get \"https://10.132.0.36:6567/health\": dial tcp 10.132.0.36:6567: connect: connection refused" Sep 11 09:06:06.660546 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:06:06.659975 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-credit-scoring-6fbf9ffc6f-v4jhk" Sep 11 09:06:13.412632 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:06:13.412589 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-6fbf9ffc6f-v4jhk" Sep 11 09:06:13.414856 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:06:13.414836 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-775c6d58bb-2kbhq" Sep 11 09:06:13.449729 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:06:13.449708 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cddbn\" (UniqueName: \"kubernetes.io/projected/aea9836a-fd24-40de-83f7-6a60dcef0c7e-kube-api-access-cddbn\") pod \"aea9836a-fd24-40de-83f7-6a60dcef0c7e\" (UID: \"aea9836a-fd24-40de-83f7-6a60dcef0c7e\") " Sep 11 09:06:13.449831 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:06:13.449738 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zztmf\" (UniqueName: \"kubernetes.io/projected/76e60db5-7a8e-46ba-87a0-becb0dcb32a8-kube-api-access-zztmf\") pod \"76e60db5-7a8e-46ba-87a0-becb0dcb32a8\" (UID: \"76e60db5-7a8e-46ba-87a0-becb0dcb32a8\") " Sep 11 09:06:13.449831 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:06:13.449758 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/aea9836a-fd24-40de-83f7-6a60dcef0c7e-registry-tls\") pod \"aea9836a-fd24-40de-83f7-6a60dcef0c7e\" (UID: \"aea9836a-fd24-40de-83f7-6a60dcef0c7e\") " Sep 11 09:06:13.449831 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:06:13.449794 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/aea9836a-fd24-40de-83f7-6a60dcef0c7e-online-tls\") pod \"aea9836a-fd24-40de-83f7-6a60dcef0c7e\" (UID: \"aea9836a-fd24-40de-83f7-6a60dcef0c7e\") " Sep 11 09:06:13.449831 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:06:13.449820 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/76e60db5-7a8e-46ba-87a0-becb0dcb32a8-online-tls\") pod \"76e60db5-7a8e-46ba-87a0-becb0dcb32a8\" (UID: \"76e60db5-7a8e-46ba-87a0-becb0dcb32a8\") " Sep 11 09:06:13.450096 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:06:13.449845 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/aea9836a-fd24-40de-83f7-6a60dcef0c7e-feast-data\") pod \"aea9836a-fd24-40de-83f7-6a60dcef0c7e\" (UID: \"aea9836a-fd24-40de-83f7-6a60dcef0c7e\") " Sep 11 09:06:13.450096 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:06:13.449902 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/76e60db5-7a8e-46ba-87a0-becb0dcb32a8-registry-tls\") pod \"76e60db5-7a8e-46ba-87a0-becb0dcb32a8\" (UID: \"76e60db5-7a8e-46ba-87a0-becb0dcb32a8\") " Sep 11 09:06:13.450096 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:06:13.449934 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/76e60db5-7a8e-46ba-87a0-becb0dcb32a8-feast-data\") pod \"76e60db5-7a8e-46ba-87a0-becb0dcb32a8\" (UID: \"76e60db5-7a8e-46ba-87a0-becb0dcb32a8\") " Sep 11 09:06:13.451025 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:06:13.450984 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76e60db5-7a8e-46ba-87a0-becb0dcb32a8-feast-data" (OuterVolumeSpecName: "feast-data") pod "76e60db5-7a8e-46ba-87a0-becb0dcb32a8" (UID: "76e60db5-7a8e-46ba-87a0-becb0dcb32a8"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 09:06:13.452659 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:06:13.452625 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aea9836a-fd24-40de-83f7-6a60dcef0c7e-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "aea9836a-fd24-40de-83f7-6a60dcef0c7e" (UID: "aea9836a-fd24-40de-83f7-6a60dcef0c7e"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 09:06:13.452751 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:06:13.452636 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76e60db5-7a8e-46ba-87a0-becb0dcb32a8-kube-api-access-zztmf" (OuterVolumeSpecName: "kube-api-access-zztmf") pod "76e60db5-7a8e-46ba-87a0-becb0dcb32a8" (UID: "76e60db5-7a8e-46ba-87a0-becb0dcb32a8"). InnerVolumeSpecName "kube-api-access-zztmf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 09:06:13.452751 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:06:13.452720 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76e60db5-7a8e-46ba-87a0-becb0dcb32a8-online-tls" (OuterVolumeSpecName: "online-tls") pod "76e60db5-7a8e-46ba-87a0-becb0dcb32a8" (UID: "76e60db5-7a8e-46ba-87a0-becb0dcb32a8"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 09:06:13.452751 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:06:13.452721 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aea9836a-fd24-40de-83f7-6a60dcef0c7e-online-tls" (OuterVolumeSpecName: "online-tls") pod "aea9836a-fd24-40de-83f7-6a60dcef0c7e" (UID: "aea9836a-fd24-40de-83f7-6a60dcef0c7e"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 09:06:13.453012 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:06:13.452987 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aea9836a-fd24-40de-83f7-6a60dcef0c7e-kube-api-access-cddbn" (OuterVolumeSpecName: "kube-api-access-cddbn") pod "aea9836a-fd24-40de-83f7-6a60dcef0c7e" (UID: "aea9836a-fd24-40de-83f7-6a60dcef0c7e"). InnerVolumeSpecName "kube-api-access-cddbn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 09:06:13.453376 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:06:13.453351 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76e60db5-7a8e-46ba-87a0-becb0dcb32a8-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "76e60db5-7a8e-46ba-87a0-becb0dcb32a8" (UID: "76e60db5-7a8e-46ba-87a0-becb0dcb32a8"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 09:06:13.460400 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:06:13.460373 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aea9836a-fd24-40de-83f7-6a60dcef0c7e-feast-data" (OuterVolumeSpecName: "feast-data") pod "aea9836a-fd24-40de-83f7-6a60dcef0c7e" (UID: "aea9836a-fd24-40de-83f7-6a60dcef0c7e"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 09:06:13.467845 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:06:13.467813 2572 generic.go:356] "Generic (PLEG): container finished" podID="76e60db5-7a8e-46ba-87a0-becb0dcb32a8" containerID="0cdb8c0a2a9b834dff89429f7c85a43ae0c292de0c64800dd1e34defb20a2b1b" exitCode=137 Sep 11 09:06:13.468095 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:06:13.468068 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-775c6d58bb-2kbhq" event={"ID":"76e60db5-7a8e-46ba-87a0-becb0dcb32a8","Type":"ContainerDied","Data":"0cdb8c0a2a9b834dff89429f7c85a43ae0c292de0c64800dd1e34defb20a2b1b"} Sep 11 09:06:13.468217 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:06:13.468128 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-775c6d58bb-2kbhq" event={"ID":"76e60db5-7a8e-46ba-87a0-becb0dcb32a8","Type":"ContainerDied","Data":"15dd4fecba6b7018389c677868e2b15ed622ac2af02325c7bf5dd7b1d31bd664"} Sep 11 09:06:13.468217 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:06:13.468155 2572 scope.go:117] "RemoveContainer" containerID="eb09066bd01d3c543918a263f9f84a71c5f1c98036c9ce69edbb6021ae25c88e" Sep 11 09:06:13.468352 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:06:13.468082 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-775c6d58bb-2kbhq" Sep 11 09:06:13.471705 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:06:13.471682 2572 generic.go:356] "Generic (PLEG): container finished" podID="aea9836a-fd24-40de-83f7-6a60dcef0c7e" containerID="1092dd438cfc1a67a42c272bde1211321486ed76c0c225e925de5a1f9eb81690" exitCode=137 Sep 11 09:06:13.471818 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:06:13.471714 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-6fbf9ffc6f-v4jhk" event={"ID":"aea9836a-fd24-40de-83f7-6a60dcef0c7e","Type":"ContainerDied","Data":"1092dd438cfc1a67a42c272bde1211321486ed76c0c225e925de5a1f9eb81690"} Sep 11 09:06:13.471818 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:06:13.471745 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-6fbf9ffc6f-v4jhk" event={"ID":"aea9836a-fd24-40de-83f7-6a60dcef0c7e","Type":"ContainerDied","Data":"3c2dc50c2e54bfdf298b08ce912ac03c1b9d40fa1fcb2201c150198864bac4a2"} Sep 11 09:06:13.471818 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:06:13.471805 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-6fbf9ffc6f-v4jhk" Sep 11 09:06:13.479300 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:06:13.479257 2572 scope.go:117] "RemoveContainer" containerID="0cdb8c0a2a9b834dff89429f7c85a43ae0c292de0c64800dd1e34defb20a2b1b" Sep 11 09:06:13.487394 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:06:13.487375 2572 scope.go:117] "RemoveContainer" containerID="260aa15790613111e946036d9be4504661a487b5a6f5aa8805818eb1b5355335" Sep 11 09:06:13.496045 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:06:13.496027 2572 scope.go:117] "RemoveContainer" containerID="a2258d21fa12bf64161b9354b1a712465d5335b4d68c38aa29766020e7de4cac" Sep 11 09:06:13.498067 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:06:13.498046 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-6fbf9ffc6f-v4jhk"] Sep 11 09:06:13.503762 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:06:13.503744 2572 scope.go:117] "RemoveContainer" containerID="eb09066bd01d3c543918a263f9f84a71c5f1c98036c9ce69edbb6021ae25c88e" Sep 11 09:06:13.504059 ip-10-0-128-166 kubenswrapper[2572]: E0911 09:06:13.504039 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb09066bd01d3c543918a263f9f84a71c5f1c98036c9ce69edbb6021ae25c88e\": container with ID starting with eb09066bd01d3c543918a263f9f84a71c5f1c98036c9ce69edbb6021ae25c88e not found: ID does not exist" containerID="eb09066bd01d3c543918a263f9f84a71c5f1c98036c9ce69edbb6021ae25c88e" Sep 11 09:06:13.504127 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:06:13.504098 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb09066bd01d3c543918a263f9f84a71c5f1c98036c9ce69edbb6021ae25c88e"} err="failed to get container status \"eb09066bd01d3c543918a263f9f84a71c5f1c98036c9ce69edbb6021ae25c88e\": rpc error: code = NotFound desc = could not find container \"eb09066bd01d3c543918a263f9f84a71c5f1c98036c9ce69edbb6021ae25c88e\": container with ID starting with eb09066bd01d3c543918a263f9f84a71c5f1c98036c9ce69edbb6021ae25c88e not found: ID does not exist" Sep 11 09:06:13.504127 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:06:13.504117 2572 scope.go:117] "RemoveContainer" containerID="0cdb8c0a2a9b834dff89429f7c85a43ae0c292de0c64800dd1e34defb20a2b1b" Sep 11 09:06:13.504389 ip-10-0-128-166 kubenswrapper[2572]: E0911 09:06:13.504370 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0cdb8c0a2a9b834dff89429f7c85a43ae0c292de0c64800dd1e34defb20a2b1b\": container with ID starting with 0cdb8c0a2a9b834dff89429f7c85a43ae0c292de0c64800dd1e34defb20a2b1b not found: ID does not exist" containerID="0cdb8c0a2a9b834dff89429f7c85a43ae0c292de0c64800dd1e34defb20a2b1b" Sep 11 09:06:13.504441 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:06:13.504405 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cdb8c0a2a9b834dff89429f7c85a43ae0c292de0c64800dd1e34defb20a2b1b"} err="failed to get container status \"0cdb8c0a2a9b834dff89429f7c85a43ae0c292de0c64800dd1e34defb20a2b1b\": rpc error: code = NotFound desc = could not find container \"0cdb8c0a2a9b834dff89429f7c85a43ae0c292de0c64800dd1e34defb20a2b1b\": container with ID starting with 0cdb8c0a2a9b834dff89429f7c85a43ae0c292de0c64800dd1e34defb20a2b1b not found: ID does not exist" Sep 11 09:06:13.504441 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:06:13.504419 2572 scope.go:117] "RemoveContainer" containerID="260aa15790613111e946036d9be4504661a487b5a6f5aa8805818eb1b5355335" Sep 11 09:06:13.504656 ip-10-0-128-166 kubenswrapper[2572]: E0911 09:06:13.504639 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"260aa15790613111e946036d9be4504661a487b5a6f5aa8805818eb1b5355335\": container with ID starting with 260aa15790613111e946036d9be4504661a487b5a6f5aa8805818eb1b5355335 not found: ID does not exist" containerID="260aa15790613111e946036d9be4504661a487b5a6f5aa8805818eb1b5355335" Sep 11 09:06:13.504704 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:06:13.504659 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"260aa15790613111e946036d9be4504661a487b5a6f5aa8805818eb1b5355335"} err="failed to get container status \"260aa15790613111e946036d9be4504661a487b5a6f5aa8805818eb1b5355335\": rpc error: code = NotFound desc = could not find container \"260aa15790613111e946036d9be4504661a487b5a6f5aa8805818eb1b5355335\": container with ID starting with 260aa15790613111e946036d9be4504661a487b5a6f5aa8805818eb1b5355335 not found: ID does not exist" Sep 11 09:06:13.504704 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:06:13.504672 2572 scope.go:117] "RemoveContainer" containerID="a2258d21fa12bf64161b9354b1a712465d5335b4d68c38aa29766020e7de4cac" Sep 11 09:06:13.504787 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:06:13.504731 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-6fbf9ffc6f-v4jhk"] Sep 11 09:06:13.504944 ip-10-0-128-166 kubenswrapper[2572]: E0911 09:06:13.504926 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2258d21fa12bf64161b9354b1a712465d5335b4d68c38aa29766020e7de4cac\": container with ID starting with a2258d21fa12bf64161b9354b1a712465d5335b4d68c38aa29766020e7de4cac not found: ID does not exist" containerID="a2258d21fa12bf64161b9354b1a712465d5335b4d68c38aa29766020e7de4cac" Sep 11 09:06:13.504984 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:06:13.504949 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2258d21fa12bf64161b9354b1a712465d5335b4d68c38aa29766020e7de4cac"} err="failed to get container status \"a2258d21fa12bf64161b9354b1a712465d5335b4d68c38aa29766020e7de4cac\": rpc error: code = NotFound desc = could not find container \"a2258d21fa12bf64161b9354b1a712465d5335b4d68c38aa29766020e7de4cac\": container with ID starting with a2258d21fa12bf64161b9354b1a712465d5335b4d68c38aa29766020e7de4cac not found: ID does not exist" Sep 11 09:06:13.504984 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:06:13.504965 2572 scope.go:117] "RemoveContainer" containerID="708210725af9835ff31aca4364897cc38755bc36ec0f0a111990a0c2d1f3ebaf" Sep 11 09:06:13.511998 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:06:13.511981 2572 scope.go:117] "RemoveContainer" containerID="1092dd438cfc1a67a42c272bde1211321486ed76c0c225e925de5a1f9eb81690" Sep 11 09:06:13.517329 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:06:13.517310 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-775c6d58bb-2kbhq"] Sep 11 09:06:13.519433 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:06:13.519419 2572 scope.go:117] "RemoveContainer" containerID="67b58f57c1a49e7fe70a2d9e59e2c8e6426e3fc7ebe2b8bea35fb3ebbdd5d664" Sep 11 09:06:13.524626 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:06:13.524609 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-775c6d58bb-2kbhq"] Sep 11 09:06:13.526676 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:06:13.526664 2572 scope.go:117] "RemoveContainer" containerID="ac092b1cc84a132e55f42f1c06252ad549e5542979ec2bbe73e22d5f46db1d86" Sep 11 09:06:13.533283 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:06:13.533267 2572 scope.go:117] "RemoveContainer" containerID="708210725af9835ff31aca4364897cc38755bc36ec0f0a111990a0c2d1f3ebaf" Sep 11 09:06:13.533550 ip-10-0-128-166 kubenswrapper[2572]: E0911 09:06:13.533524 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"708210725af9835ff31aca4364897cc38755bc36ec0f0a111990a0c2d1f3ebaf\": container with ID starting with 708210725af9835ff31aca4364897cc38755bc36ec0f0a111990a0c2d1f3ebaf not found: ID does not exist" containerID="708210725af9835ff31aca4364897cc38755bc36ec0f0a111990a0c2d1f3ebaf" Sep 11 09:06:13.533611 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:06:13.533547 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"708210725af9835ff31aca4364897cc38755bc36ec0f0a111990a0c2d1f3ebaf"} err="failed to get container status \"708210725af9835ff31aca4364897cc38755bc36ec0f0a111990a0c2d1f3ebaf\": rpc error: code = NotFound desc = could not find container \"708210725af9835ff31aca4364897cc38755bc36ec0f0a111990a0c2d1f3ebaf\": container with ID starting with 708210725af9835ff31aca4364897cc38755bc36ec0f0a111990a0c2d1f3ebaf not found: ID does not exist" Sep 11 09:06:13.533611 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:06:13.533569 2572 scope.go:117] "RemoveContainer" containerID="1092dd438cfc1a67a42c272bde1211321486ed76c0c225e925de5a1f9eb81690" Sep 11 09:06:13.533829 ip-10-0-128-166 kubenswrapper[2572]: E0911 09:06:13.533814 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1092dd438cfc1a67a42c272bde1211321486ed76c0c225e925de5a1f9eb81690\": container with ID starting with 1092dd438cfc1a67a42c272bde1211321486ed76c0c225e925de5a1f9eb81690 not found: ID does not exist" containerID="1092dd438cfc1a67a42c272bde1211321486ed76c0c225e925de5a1f9eb81690" Sep 11 09:06:13.533874 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:06:13.533831 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1092dd438cfc1a67a42c272bde1211321486ed76c0c225e925de5a1f9eb81690"} err="failed to get container status \"1092dd438cfc1a67a42c272bde1211321486ed76c0c225e925de5a1f9eb81690\": rpc error: code = NotFound desc = could not find container \"1092dd438cfc1a67a42c272bde1211321486ed76c0c225e925de5a1f9eb81690\": container with ID starting with 1092dd438cfc1a67a42c272bde1211321486ed76c0c225e925de5a1f9eb81690 not found: ID does not exist" Sep 11 09:06:13.533874 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:06:13.533843 2572 scope.go:117] "RemoveContainer" containerID="67b58f57c1a49e7fe70a2d9e59e2c8e6426e3fc7ebe2b8bea35fb3ebbdd5d664" Sep 11 09:06:13.534069 ip-10-0-128-166 kubenswrapper[2572]: E0911 09:06:13.534056 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67b58f57c1a49e7fe70a2d9e59e2c8e6426e3fc7ebe2b8bea35fb3ebbdd5d664\": container with ID starting with 67b58f57c1a49e7fe70a2d9e59e2c8e6426e3fc7ebe2b8bea35fb3ebbdd5d664 not found: ID does not exist" containerID="67b58f57c1a49e7fe70a2d9e59e2c8e6426e3fc7ebe2b8bea35fb3ebbdd5d664" Sep 11 09:06:13.534109 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:06:13.534072 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67b58f57c1a49e7fe70a2d9e59e2c8e6426e3fc7ebe2b8bea35fb3ebbdd5d664"} err="failed to get container status \"67b58f57c1a49e7fe70a2d9e59e2c8e6426e3fc7ebe2b8bea35fb3ebbdd5d664\": rpc error: code = NotFound desc = could not find container \"67b58f57c1a49e7fe70a2d9e59e2c8e6426e3fc7ebe2b8bea35fb3ebbdd5d664\": container with ID starting with 67b58f57c1a49e7fe70a2d9e59e2c8e6426e3fc7ebe2b8bea35fb3ebbdd5d664 not found: ID does not exist" Sep 11 09:06:13.534109 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:06:13.534083 2572 scope.go:117] "RemoveContainer" containerID="ac092b1cc84a132e55f42f1c06252ad549e5542979ec2bbe73e22d5f46db1d86" Sep 11 09:06:13.534344 ip-10-0-128-166 kubenswrapper[2572]: E0911 09:06:13.534329 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac092b1cc84a132e55f42f1c06252ad549e5542979ec2bbe73e22d5f46db1d86\": container with ID starting with ac092b1cc84a132e55f42f1c06252ad549e5542979ec2bbe73e22d5f46db1d86 not found: ID does not exist" containerID="ac092b1cc84a132e55f42f1c06252ad549e5542979ec2bbe73e22d5f46db1d86" Sep 11 09:06:13.534402 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:06:13.534346 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac092b1cc84a132e55f42f1c06252ad549e5542979ec2bbe73e22d5f46db1d86"} err="failed to get container status \"ac092b1cc84a132e55f42f1c06252ad549e5542979ec2bbe73e22d5f46db1d86\": rpc error: code = NotFound desc = could not find container \"ac092b1cc84a132e55f42f1c06252ad549e5542979ec2bbe73e22d5f46db1d86\": container with ID starting with ac092b1cc84a132e55f42f1c06252ad549e5542979ec2bbe73e22d5f46db1d86 not found: ID does not exist" Sep 11 09:06:13.550747 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:06:13.550726 2572 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/76e60db5-7a8e-46ba-87a0-becb0dcb32a8-registry-tls\") on node \"ip-10-0-128-166.ec2.internal\" DevicePath \"\"" Sep 11 09:06:13.550747 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:06:13.550746 2572 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/76e60db5-7a8e-46ba-87a0-becb0dcb32a8-feast-data\") on node \"ip-10-0-128-166.ec2.internal\" DevicePath \"\"" Sep 11 09:06:13.550836 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:06:13.550755 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cddbn\" (UniqueName: \"kubernetes.io/projected/aea9836a-fd24-40de-83f7-6a60dcef0c7e-kube-api-access-cddbn\") on node \"ip-10-0-128-166.ec2.internal\" DevicePath \"\"" Sep 11 09:06:13.550836 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:06:13.550764 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zztmf\" (UniqueName: \"kubernetes.io/projected/76e60db5-7a8e-46ba-87a0-becb0dcb32a8-kube-api-access-zztmf\") on node \"ip-10-0-128-166.ec2.internal\" DevicePath \"\"" Sep 11 09:06:13.550836 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:06:13.550772 2572 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/aea9836a-fd24-40de-83f7-6a60dcef0c7e-registry-tls\") on node \"ip-10-0-128-166.ec2.internal\" DevicePath \"\"" Sep 11 09:06:13.550836 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:06:13.550780 2572 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/aea9836a-fd24-40de-83f7-6a60dcef0c7e-online-tls\") on node \"ip-10-0-128-166.ec2.internal\" DevicePath \"\"" Sep 11 09:06:13.550836 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:06:13.550786 2572 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/76e60db5-7a8e-46ba-87a0-becb0dcb32a8-online-tls\") on node \"ip-10-0-128-166.ec2.internal\" DevicePath \"\"" Sep 11 09:06:13.550836 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:06:13.550794 2572 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/aea9836a-fd24-40de-83f7-6a60dcef0c7e-feast-data\") on node \"ip-10-0-128-166.ec2.internal\" DevicePath \"\"" Sep 11 09:06:14.251992 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:06:14.251959 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="76e60db5-7a8e-46ba-87a0-becb0dcb32a8" path="/var/lib/kubelet/pods/76e60db5-7a8e-46ba-87a0-becb0dcb32a8/volumes" Sep 11 09:06:14.252459 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:06:14.252446 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aea9836a-fd24-40de-83f7-6a60dcef0c7e" path="/var/lib/kubelet/pods/aea9836a-fd24-40de-83f7-6a60dcef0c7e/volumes" Sep 11 09:06:20.237394 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:06:20.237366 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-xjs9g_546705a6-1aaf-49fe-9f7f-3fc9e566437b/console-operator/2.log" Sep 11 09:06:20.240056 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:06:20.240032 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-xjs9g_546705a6-1aaf-49fe-9f7f-3fc9e566437b/console-operator/2.log" Sep 11 09:11:20.261334 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:11:20.261306 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-xjs9g_546705a6-1aaf-49fe-9f7f-3fc9e566437b/console-operator/2.log" Sep 11 09:11:20.268176 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:11:20.268151 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-xjs9g_546705a6-1aaf-49fe-9f7f-3fc9e566437b/console-operator/2.log" Sep 11 09:16:20.287650 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:16:20.287618 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-xjs9g_546705a6-1aaf-49fe-9f7f-3fc9e566437b/console-operator/2.log" Sep 11 09:16:20.293187 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:16:20.293165 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-xjs9g_546705a6-1aaf-49fe-9f7f-3fc9e566437b/console-operator/2.log" Sep 11 09:17:33.106207 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:17:33.106175 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-fbvp7/must-gather-zm92t"] Sep 11 09:17:33.106692 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:17:33.106482 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5dc2a815-4d2a-4c3b-aa18-735eb0a414cf" containerName="manager" Sep 11 09:17:33.106692 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:17:33.106493 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="5dc2a815-4d2a-4c3b-aa18-735eb0a414cf" containerName="manager" Sep 11 09:17:33.106692 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:17:33.106501 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aea9836a-fd24-40de-83f7-6a60dcef0c7e" containerName="online" Sep 11 09:17:33.106692 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:17:33.106507 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="aea9836a-fd24-40de-83f7-6a60dcef0c7e" containerName="online" Sep 11 09:17:33.106692 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:17:33.106515 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76e60db5-7a8e-46ba-87a0-becb0dcb32a8" containerName="feast-apply" Sep 11 09:17:33.106692 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:17:33.106521 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="76e60db5-7a8e-46ba-87a0-becb0dcb32a8" containerName="feast-apply" Sep 11 09:17:33.106692 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:17:33.106531 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76e60db5-7a8e-46ba-87a0-becb0dcb32a8" containerName="feast-init" Sep 11 09:17:33.106692 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:17:33.106536 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="76e60db5-7a8e-46ba-87a0-becb0dcb32a8" containerName="feast-init" Sep 11 09:17:33.106692 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:17:33.106544 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aea9836a-fd24-40de-83f7-6a60dcef0c7e" containerName="registry" Sep 11 09:17:33.106692 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:17:33.106549 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="aea9836a-fd24-40de-83f7-6a60dcef0c7e" containerName="registry" Sep 11 09:17:33.106692 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:17:33.106555 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76e60db5-7a8e-46ba-87a0-becb0dcb32a8" containerName="online" Sep 11 09:17:33.106692 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:17:33.106560 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="76e60db5-7a8e-46ba-87a0-becb0dcb32a8" containerName="online" Sep 11 09:17:33.106692 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:17:33.106568 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d11d6929-fb34-47a2-8537-dedc99a3d93a" containerName="postgres" Sep 11 09:17:33.106692 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:17:33.106573 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="d11d6929-fb34-47a2-8537-dedc99a3d93a" containerName="postgres" Sep 11 09:17:33.106692 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:17:33.106579 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aea9836a-fd24-40de-83f7-6a60dcef0c7e" containerName="feast-init" Sep 11 09:17:33.106692 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:17:33.106584 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="aea9836a-fd24-40de-83f7-6a60dcef0c7e" containerName="feast-init" Sep 11 09:17:33.106692 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:17:33.106592 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aea9836a-fd24-40de-83f7-6a60dcef0c7e" containerName="feast-apply" Sep 11 09:17:33.106692 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:17:33.106597 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="aea9836a-fd24-40de-83f7-6a60dcef0c7e" containerName="feast-apply" Sep 11 09:17:33.106692 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:17:33.106604 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6c264c9-0874-4e91-a901-9bb0ab9a75c1" containerName="redis" Sep 11 09:17:33.106692 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:17:33.106608 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6c264c9-0874-4e91-a901-9bb0ab9a75c1" containerName="redis" Sep 11 09:17:33.106692 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:17:33.106613 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76e60db5-7a8e-46ba-87a0-becb0dcb32a8" containerName="registry" Sep 11 09:17:33.106692 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:17:33.106618 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="76e60db5-7a8e-46ba-87a0-becb0dcb32a8" containerName="registry" Sep 11 09:17:33.106692 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:17:33.106660 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="d11d6929-fb34-47a2-8537-dedc99a3d93a" containerName="postgres" Sep 11 09:17:33.106692 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:17:33.106668 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="76e60db5-7a8e-46ba-87a0-becb0dcb32a8" containerName="registry" Sep 11 09:17:33.106692 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:17:33.106675 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="aea9836a-fd24-40de-83f7-6a60dcef0c7e" containerName="online" Sep 11 09:17:33.106692 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:17:33.106681 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="76e60db5-7a8e-46ba-87a0-becb0dcb32a8" containerName="online" Sep 11 09:17:33.106692 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:17:33.106687 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="5dc2a815-4d2a-4c3b-aa18-735eb0a414cf" containerName="manager" Sep 11 09:17:33.106692 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:17:33.106692 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6c264c9-0874-4e91-a901-9bb0ab9a75c1" containerName="redis" Sep 11 09:17:33.106692 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:17:33.106697 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="aea9836a-fd24-40de-83f7-6a60dcef0c7e" containerName="registry" Sep 11 09:17:33.109713 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:17:33.109690 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fbvp7/must-gather-zm92t" Sep 11 09:17:33.112421 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:17:33.112402 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-fbvp7\"/\"kube-root-ca.crt\"" Sep 11 09:17:33.112558 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:17:33.112403 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-fbvp7\"/\"default-dockercfg-dqd6j\"" Sep 11 09:17:33.112775 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:17:33.112760 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-fbvp7\"/\"openshift-service-ca.crt\"" Sep 11 09:17:33.118986 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:17:33.118964 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fbvp7/must-gather-zm92t"] Sep 11 09:17:33.271550 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:17:33.271518 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/44e726b7-4ce6-4f3c-aced-540b57b5cab4-must-gather-output\") pod \"must-gather-zm92t\" (UID: \"44e726b7-4ce6-4f3c-aced-540b57b5cab4\") " pod="openshift-must-gather-fbvp7/must-gather-zm92t" Sep 11 09:17:33.271673 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:17:33.271563 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krw65\" (UniqueName: \"kubernetes.io/projected/44e726b7-4ce6-4f3c-aced-540b57b5cab4-kube-api-access-krw65\") pod \"must-gather-zm92t\" (UID: \"44e726b7-4ce6-4f3c-aced-540b57b5cab4\") " pod="openshift-must-gather-fbvp7/must-gather-zm92t" Sep 11 09:17:33.372453 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:17:33.372387 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krw65\" (UniqueName: \"kubernetes.io/projected/44e726b7-4ce6-4f3c-aced-540b57b5cab4-kube-api-access-krw65\") pod \"must-gather-zm92t\" (UID: \"44e726b7-4ce6-4f3c-aced-540b57b5cab4\") " pod="openshift-must-gather-fbvp7/must-gather-zm92t" Sep 11 09:17:33.372634 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:17:33.372586 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/44e726b7-4ce6-4f3c-aced-540b57b5cab4-must-gather-output\") pod \"must-gather-zm92t\" (UID: \"44e726b7-4ce6-4f3c-aced-540b57b5cab4\") " pod="openshift-must-gather-fbvp7/must-gather-zm92t" Sep 11 09:17:33.373012 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:17:33.372992 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/44e726b7-4ce6-4f3c-aced-540b57b5cab4-must-gather-output\") pod \"must-gather-zm92t\" (UID: \"44e726b7-4ce6-4f3c-aced-540b57b5cab4\") " pod="openshift-must-gather-fbvp7/must-gather-zm92t" Sep 11 09:17:33.381946 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:17:33.381918 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-krw65\" (UniqueName: \"kubernetes.io/projected/44e726b7-4ce6-4f3c-aced-540b57b5cab4-kube-api-access-krw65\") pod \"must-gather-zm92t\" (UID: \"44e726b7-4ce6-4f3c-aced-540b57b5cab4\") " pod="openshift-must-gather-fbvp7/must-gather-zm92t" Sep 11 09:17:33.430984 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:17:33.430952 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fbvp7/must-gather-zm92t" Sep 11 09:17:33.702898 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:17:33.702875 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fbvp7/must-gather-zm92t"] Sep 11 09:17:33.704907 ip-10-0-128-166 kubenswrapper[2572]: W0911 09:17:33.704874 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod44e726b7_4ce6_4f3c_aced_540b57b5cab4.slice/crio-e6e7efe18dd3ff6143b497e5703cc9ba7bd050a8b16894bbecce2ed5db25b110 WatchSource:0}: Error finding container e6e7efe18dd3ff6143b497e5703cc9ba7bd050a8b16894bbecce2ed5db25b110: Status 404 returned error can't find the container with id e6e7efe18dd3ff6143b497e5703cc9ba7bd050a8b16894bbecce2ed5db25b110 Sep 11 09:17:33.706892 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:17:33.706875 2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 09:17:34.073283 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:17:34.073187 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fbvp7/must-gather-zm92t" event={"ID":"44e726b7-4ce6-4f3c-aced-540b57b5cab4","Type":"ContainerStarted","Data":"e6e7efe18dd3ff6143b497e5703cc9ba7bd050a8b16894bbecce2ed5db25b110"} Sep 11 09:17:40.101856 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:17:40.101816 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fbvp7/must-gather-zm92t" event={"ID":"44e726b7-4ce6-4f3c-aced-540b57b5cab4","Type":"ContainerStarted","Data":"01f03a022ba4547d7573edb26c42f1523172d037c1fc8411d55e3a89143d52e8"} Sep 11 09:17:41.108663 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:17:41.108545 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fbvp7/must-gather-zm92t" event={"ID":"44e726b7-4ce6-4f3c-aced-540b57b5cab4","Type":"ContainerStarted","Data":"6e374f7b7036fe12494ae945ae696a0dcaae96c0f165d6e2dfaa39228627d9ac"} Sep 11 09:17:41.126075 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:17:41.126022 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-fbvp7/must-gather-zm92t" podStartSLOduration=1.8659930949999999 podStartE2EDuration="8.12600757s" totalImagesPullingTime="6.260014475s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 09:17:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 09:17:41.124844184 +0000 UTC m=+1881.399419883" watchObservedRunningTime="2026-09-11 09:17:41.12600757 +0000 UTC m=+1881.400583266" Sep 11 09:17:50.149332 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:17:50.149296 2572 generic.go:356] "Generic (PLEG): container finished" podID="44e726b7-4ce6-4f3c-aced-540b57b5cab4" containerID="01f03a022ba4547d7573edb26c42f1523172d037c1fc8411d55e3a89143d52e8" exitCode=0 Sep 11 09:17:50.149864 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:17:50.149372 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fbvp7/must-gather-zm92t" event={"ID":"44e726b7-4ce6-4f3c-aced-540b57b5cab4","Type":"ContainerDied","Data":"01f03a022ba4547d7573edb26c42f1523172d037c1fc8411d55e3a89143d52e8"} Sep 11 09:17:50.149864 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:17:50.149712 2572 scope.go:117] "RemoveContainer" containerID="01f03a022ba4547d7573edb26c42f1523172d037c1fc8411d55e3a89143d52e8" Sep 11 09:17:50.526958 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:17:50.526931 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-fbvp7_must-gather-zm92t_44e726b7-4ce6-4f3c-aced-540b57b5cab4/gather/0.log" Sep 11 09:17:53.714165 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:17:53.714128 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-v7fx2_a370ef3a-98cb-4251-86b9-cb3155c587a5/global-pull-secret-syncer/0.log" Sep 11 09:17:53.827470 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:17:53.827439 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-p5sg8_2c0c8738-0dd8-441e-8c6e-3ff6009337ca/konnectivity-agent/0.log" Sep 11 09:17:53.846819 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:17:53.846790 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-166.ec2.internal_1f3dec62f93374a659aa01a42bf55935/haproxy/0.log" Sep 11 09:17:55.857641 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:17:55.857609 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-fbvp7/must-gather-zm92t"] Sep 11 09:17:55.858030 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:17:55.857802 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-fbvp7/must-gather-zm92t" podUID="44e726b7-4ce6-4f3c-aced-540b57b5cab4" containerName="copy" containerID="cri-o://6e374f7b7036fe12494ae945ae696a0dcaae96c0f165d6e2dfaa39228627d9ac" gracePeriod=2 Sep 11 09:17:55.862725 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:17:55.862688 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-fbvp7/must-gather-zm92t"] Sep 11 09:17:56.090583 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:17:56.090556 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-fbvp7_must-gather-zm92t_44e726b7-4ce6-4f3c-aced-540b57b5cab4/copy/0.log" Sep 11 09:17:56.093100 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:17:56.093082 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fbvp7/must-gather-zm92t" Sep 11 09:17:56.094910 ip-10-0-128-166 kubenswrapper[2572]: E0911 09:17:56.094886 2572 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-zm92t\" is forbidden: User \"system:node:ip-10-0-128-166.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-fbvp7\": no relationship found between node 'ip-10-0-128-166.ec2.internal' and this object" podUID="44e726b7-4ce6-4f3c-aced-540b57b5cab4" pod="openshift-must-gather-fbvp7/must-gather-zm92t" Sep 11 09:17:56.155129 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:17:56.155078 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-krw65\" (UniqueName: \"kubernetes.io/projected/44e726b7-4ce6-4f3c-aced-540b57b5cab4-kube-api-access-krw65\") pod \"44e726b7-4ce6-4f3c-aced-540b57b5cab4\" (UID: \"44e726b7-4ce6-4f3c-aced-540b57b5cab4\") " Sep 11 09:17:56.155129 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:17:56.155110 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/44e726b7-4ce6-4f3c-aced-540b57b5cab4-must-gather-output\") pod \"44e726b7-4ce6-4f3c-aced-540b57b5cab4\" (UID: \"44e726b7-4ce6-4f3c-aced-540b57b5cab4\") " Sep 11 09:17:56.155504 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:17:56.155480 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44e726b7-4ce6-4f3c-aced-540b57b5cab4-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "44e726b7-4ce6-4f3c-aced-540b57b5cab4" (UID: "44e726b7-4ce6-4f3c-aced-540b57b5cab4"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 09:17:56.157396 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:17:56.157370 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44e726b7-4ce6-4f3c-aced-540b57b5cab4-kube-api-access-krw65" (OuterVolumeSpecName: "kube-api-access-krw65") pod "44e726b7-4ce6-4f3c-aced-540b57b5cab4" (UID: "44e726b7-4ce6-4f3c-aced-540b57b5cab4"). InnerVolumeSpecName "kube-api-access-krw65". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 09:17:56.174679 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:17:56.174659 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-fbvp7_must-gather-zm92t_44e726b7-4ce6-4f3c-aced-540b57b5cab4/copy/0.log" Sep 11 09:17:56.175018 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:17:56.174999 2572 generic.go:356] "Generic (PLEG): container finished" podID="44e726b7-4ce6-4f3c-aced-540b57b5cab4" containerID="6e374f7b7036fe12494ae945ae696a0dcaae96c0f165d6e2dfaa39228627d9ac" exitCode=143 Sep 11 09:17:56.175082 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:17:56.175042 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fbvp7/must-gather-zm92t" Sep 11 09:17:56.175123 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:17:56.175098 2572 scope.go:117] "RemoveContainer" containerID="6e374f7b7036fe12494ae945ae696a0dcaae96c0f165d6e2dfaa39228627d9ac" Sep 11 09:17:56.176850 ip-10-0-128-166 kubenswrapper[2572]: E0911 09:17:56.176820 2572 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-zm92t\" is forbidden: User \"system:node:ip-10-0-128-166.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-fbvp7\": no relationship found between node 'ip-10-0-128-166.ec2.internal' and this object" podUID="44e726b7-4ce6-4f3c-aced-540b57b5cab4" pod="openshift-must-gather-fbvp7/must-gather-zm92t" Sep 11 09:17:56.182749 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:17:56.182733 2572 scope.go:117] "RemoveContainer" containerID="01f03a022ba4547d7573edb26c42f1523172d037c1fc8411d55e3a89143d52e8" Sep 11 09:17:56.184255 ip-10-0-128-166 kubenswrapper[2572]: E0911 09:17:56.184197 2572 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-zm92t\" is forbidden: User \"system:node:ip-10-0-128-166.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-fbvp7\": no relationship found between node 'ip-10-0-128-166.ec2.internal' and this object" podUID="44e726b7-4ce6-4f3c-aced-540b57b5cab4" pod="openshift-must-gather-fbvp7/must-gather-zm92t" Sep 11 09:17:56.192996 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:17:56.192982 2572 scope.go:117] "RemoveContainer" containerID="6e374f7b7036fe12494ae945ae696a0dcaae96c0f165d6e2dfaa39228627d9ac" Sep 11 09:17:56.193310 ip-10-0-128-166 kubenswrapper[2572]: E0911 09:17:56.193290 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e374f7b7036fe12494ae945ae696a0dcaae96c0f165d6e2dfaa39228627d9ac\": container with ID starting with 6e374f7b7036fe12494ae945ae696a0dcaae96c0f165d6e2dfaa39228627d9ac not found: ID does not exist" containerID="6e374f7b7036fe12494ae945ae696a0dcaae96c0f165d6e2dfaa39228627d9ac" Sep 11 09:17:56.193398 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:17:56.193325 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e374f7b7036fe12494ae945ae696a0dcaae96c0f165d6e2dfaa39228627d9ac"} err="failed to get container status \"6e374f7b7036fe12494ae945ae696a0dcaae96c0f165d6e2dfaa39228627d9ac\": rpc error: code = NotFound desc = could not find container \"6e374f7b7036fe12494ae945ae696a0dcaae96c0f165d6e2dfaa39228627d9ac\": container with ID starting with 6e374f7b7036fe12494ae945ae696a0dcaae96c0f165d6e2dfaa39228627d9ac not found: ID does not exist" Sep 11 09:17:56.193398 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:17:56.193341 2572 scope.go:117] "RemoveContainer" containerID="01f03a022ba4547d7573edb26c42f1523172d037c1fc8411d55e3a89143d52e8" Sep 11 09:17:56.193593 ip-10-0-128-166 kubenswrapper[2572]: E0911 09:17:56.193575 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01f03a022ba4547d7573edb26c42f1523172d037c1fc8411d55e3a89143d52e8\": container with ID starting with 01f03a022ba4547d7573edb26c42f1523172d037c1fc8411d55e3a89143d52e8 not found: ID does not exist" containerID="01f03a022ba4547d7573edb26c42f1523172d037c1fc8411d55e3a89143d52e8" Sep 11 09:17:56.193640 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:17:56.193602 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01f03a022ba4547d7573edb26c42f1523172d037c1fc8411d55e3a89143d52e8"} err="failed to get container status \"01f03a022ba4547d7573edb26c42f1523172d037c1fc8411d55e3a89143d52e8\": rpc error: code = NotFound desc = could not find container \"01f03a022ba4547d7573edb26c42f1523172d037c1fc8411d55e3a89143d52e8\": container with ID starting with 01f03a022ba4547d7573edb26c42f1523172d037c1fc8411d55e3a89143d52e8 not found: ID does not exist" Sep 11 09:17:56.252265 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:17:56.252208 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44e726b7-4ce6-4f3c-aced-540b57b5cab4" path="/var/lib/kubelet/pods/44e726b7-4ce6-4f3c-aced-540b57b5cab4/volumes" Sep 11 09:17:56.256492 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:17:56.256470 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krw65\" (UniqueName: \"kubernetes.io/projected/44e726b7-4ce6-4f3c-aced-540b57b5cab4-kube-api-access-krw65\") on node \"ip-10-0-128-166.ec2.internal\" DevicePath \"\"" Sep 11 09:17:56.256594 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:17:56.256496 2572 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/44e726b7-4ce6-4f3c-aced-540b57b5cab4-must-gather-output\") on node \"ip-10-0-128-166.ec2.internal\" DevicePath \"\"" Sep 11 09:17:57.774666 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:17:57.774595 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cd954b897-blrfj_aa036e63-1ef3-481a-9786-90b2f397ab9a/cluster-monitoring-operator/0.log" Sep 11 09:17:57.895097 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:17:57.895069 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-fdbb5cd6-nqdhs_f3e3367f-aee2-495a-aec0-16c4363d5403/metrics-server/0.log" Sep 11 09:17:58.046984 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:17:58.046911 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lpvfd_9fb78fb0-b471-4ef3-a08c-3ac9b7db0cc8/node-exporter/0.log" Sep 11 09:17:58.069977 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:17:58.069954 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lpvfd_9fb78fb0-b471-4ef3-a08c-3ac9b7db0cc8/kube-rbac-proxy/0.log" Sep 11 09:17:58.097572 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:17:58.097552 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lpvfd_9fb78fb0-b471-4ef3-a08c-3ac9b7db0cc8/init-textfile/0.log" Sep 11 09:17:58.221530 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:17:58.221501 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-pfjml_201f5aa0-d681-4869-90c6-d9b1b804f99c/kube-rbac-proxy-main/0.log" Sep 11 09:17:58.255882 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:17:58.255860 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-pfjml_201f5aa0-d681-4869-90c6-d9b1b804f99c/kube-rbac-proxy-self/0.log" Sep 11 09:17:58.283920 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:17:58.283890 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-pfjml_201f5aa0-d681-4869-90c6-d9b1b804f99c/openshift-state-metrics/0.log" Sep 11 09:17:58.634735 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:17:58.634708 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7c67c9b8f-zfw99_1b0a05c4-b687-4c17-94e6-11d6d473c332/telemeter-client/0.log" Sep 11 09:17:58.661087 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:17:58.661063 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7c67c9b8f-zfw99_1b0a05c4-b687-4c17-94e6-11d6d473c332/reload/0.log" Sep 11 09:17:58.684046 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:17:58.684024 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7c67c9b8f-zfw99_1b0a05c4-b687-4c17-94e6-11d6d473c332/kube-rbac-proxy/0.log" Sep 11 09:18:00.622313 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:18:00.622252 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-xjs9g_546705a6-1aaf-49fe-9f7f-3fc9e566437b/console-operator/2.log" Sep 11 09:18:00.625896 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:18:00.625879 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-xjs9g_546705a6-1aaf-49fe-9f7f-3fc9e566437b/console-operator/3.log" Sep 11 09:18:00.894927 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:18:00.894859 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-qk7h7/perf-node-gather-daemonset-w75mh"] Sep 11 09:18:00.895146 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:18:00.895127 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44e726b7-4ce6-4f3c-aced-540b57b5cab4" containerName="gather" Sep 11 09:18:00.895146 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:18:00.895141 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="44e726b7-4ce6-4f3c-aced-540b57b5cab4" containerName="gather" Sep 11 09:18:00.895314 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:18:00.895157 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44e726b7-4ce6-4f3c-aced-540b57b5cab4" containerName="copy" Sep 11 09:18:00.895314 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:18:00.895162 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="44e726b7-4ce6-4f3c-aced-540b57b5cab4" containerName="copy" Sep 11 09:18:00.895314 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:18:00.895219 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="44e726b7-4ce6-4f3c-aced-540b57b5cab4" containerName="gather" Sep 11 09:18:00.895314 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:18:00.895244 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="44e726b7-4ce6-4f3c-aced-540b57b5cab4" containerName="copy" Sep 11 09:18:00.899923 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:18:00.899906 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qk7h7/perf-node-gather-daemonset-w75mh" Sep 11 09:18:00.901965 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:18:00.901928 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-qk7h7\"/\"kube-root-ca.crt\"" Sep 11 09:18:00.901965 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:18:00.901965 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-qk7h7\"/\"openshift-service-ca.crt\"" Sep 11 09:18:00.902341 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:18:00.902328 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-qk7h7\"/\"default-dockercfg-2zmbc\"" Sep 11 09:18:00.909002 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:18:00.908978 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qk7h7/perf-node-gather-daemonset-w75mh"] Sep 11 09:18:00.990570 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:18:00.990544 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/92387e99-7337-4166-9201-c3b737a2238a-podres\") pod \"perf-node-gather-daemonset-w75mh\" (UID: \"92387e99-7337-4166-9201-c3b737a2238a\") " pod="openshift-must-gather-qk7h7/perf-node-gather-daemonset-w75mh" Sep 11 09:18:00.990663 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:18:00.990587 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9nww\" (UniqueName: \"kubernetes.io/projected/92387e99-7337-4166-9201-c3b737a2238a-kube-api-access-k9nww\") pod \"perf-node-gather-daemonset-w75mh\" (UID: \"92387e99-7337-4166-9201-c3b737a2238a\") " pod="openshift-must-gather-qk7h7/perf-node-gather-daemonset-w75mh" Sep 11 09:18:00.990663 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:18:00.990615 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/92387e99-7337-4166-9201-c3b737a2238a-sys\") pod \"perf-node-gather-daemonset-w75mh\" (UID: \"92387e99-7337-4166-9201-c3b737a2238a\") " pod="openshift-must-gather-qk7h7/perf-node-gather-daemonset-w75mh" Sep 11 09:18:00.990663 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:18:00.990659 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/92387e99-7337-4166-9201-c3b737a2238a-proc\") pod \"perf-node-gather-daemonset-w75mh\" (UID: \"92387e99-7337-4166-9201-c3b737a2238a\") " pod="openshift-must-gather-qk7h7/perf-node-gather-daemonset-w75mh" Sep 11 09:18:00.990776 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:18:00.990681 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/92387e99-7337-4166-9201-c3b737a2238a-lib-modules\") pod \"perf-node-gather-daemonset-w75mh\" (UID: \"92387e99-7337-4166-9201-c3b737a2238a\") " pod="openshift-must-gather-qk7h7/perf-node-gather-daemonset-w75mh" Sep 11 09:18:01.091641 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:18:01.091611 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9nww\" (UniqueName: \"kubernetes.io/projected/92387e99-7337-4166-9201-c3b737a2238a-kube-api-access-k9nww\") pod \"perf-node-gather-daemonset-w75mh\" (UID: \"92387e99-7337-4166-9201-c3b737a2238a\") " pod="openshift-must-gather-qk7h7/perf-node-gather-daemonset-w75mh" Sep 11 09:18:01.091774 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:18:01.091647 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/92387e99-7337-4166-9201-c3b737a2238a-sys\") pod \"perf-node-gather-daemonset-w75mh\" (UID: \"92387e99-7337-4166-9201-c3b737a2238a\") " pod="openshift-must-gather-qk7h7/perf-node-gather-daemonset-w75mh" Sep 11 09:18:01.091774 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:18:01.091720 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/92387e99-7337-4166-9201-c3b737a2238a-sys\") pod \"perf-node-gather-daemonset-w75mh\" (UID: \"92387e99-7337-4166-9201-c3b737a2238a\") " pod="openshift-must-gather-qk7h7/perf-node-gather-daemonset-w75mh" Sep 11 09:18:01.091774 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:18:01.091746 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/92387e99-7337-4166-9201-c3b737a2238a-proc\") pod \"perf-node-gather-daemonset-w75mh\" (UID: \"92387e99-7337-4166-9201-c3b737a2238a\") " pod="openshift-must-gather-qk7h7/perf-node-gather-daemonset-w75mh" Sep 11 09:18:01.091882 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:18:01.091778 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/92387e99-7337-4166-9201-c3b737a2238a-lib-modules\") pod \"perf-node-gather-daemonset-w75mh\" (UID: \"92387e99-7337-4166-9201-c3b737a2238a\") " pod="openshift-must-gather-qk7h7/perf-node-gather-daemonset-w75mh" Sep 11 09:18:01.091882 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:18:01.091808 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/92387e99-7337-4166-9201-c3b737a2238a-podres\") pod \"perf-node-gather-daemonset-w75mh\" (UID: \"92387e99-7337-4166-9201-c3b737a2238a\") " pod="openshift-must-gather-qk7h7/perf-node-gather-daemonset-w75mh" Sep 11 09:18:01.091882 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:18:01.091861 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/92387e99-7337-4166-9201-c3b737a2238a-proc\") pod \"perf-node-gather-daemonset-w75mh\" (UID: \"92387e99-7337-4166-9201-c3b737a2238a\") " pod="openshift-must-gather-qk7h7/perf-node-gather-daemonset-w75mh" Sep 11 09:18:01.091988 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:18:01.091935 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/92387e99-7337-4166-9201-c3b737a2238a-podres\") pod \"perf-node-gather-daemonset-w75mh\" (UID: \"92387e99-7337-4166-9201-c3b737a2238a\") " pod="openshift-must-gather-qk7h7/perf-node-gather-daemonset-w75mh" Sep 11 09:18:01.091988 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:18:01.091941 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/92387e99-7337-4166-9201-c3b737a2238a-lib-modules\") pod \"perf-node-gather-daemonset-w75mh\" (UID: \"92387e99-7337-4166-9201-c3b737a2238a\") " pod="openshift-must-gather-qk7h7/perf-node-gather-daemonset-w75mh" Sep 11 09:18:01.100002 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:18:01.099976 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9nww\" (UniqueName: \"kubernetes.io/projected/92387e99-7337-4166-9201-c3b737a2238a-kube-api-access-k9nww\") pod \"perf-node-gather-daemonset-w75mh\" (UID: \"92387e99-7337-4166-9201-c3b737a2238a\") " pod="openshift-must-gather-qk7h7/perf-node-gather-daemonset-w75mh" Sep 11 09:18:01.135557 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:18:01.135539 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-65cff9fbff-n9cpf_2980cbe4-c160-4a9f-98ba-df1066e695be/console/0.log" Sep 11 09:18:01.170058 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:18:01.169980 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-76f99584f4-8vnhx_47bbded3-73b6-4258-b620-c9c8f6e918cf/download-server/0.log" Sep 11 09:18:01.210599 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:18:01.210578 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qk7h7/perf-node-gather-daemonset-w75mh" Sep 11 09:18:01.488198 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:18:01.488074 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qk7h7/perf-node-gather-daemonset-w75mh"] Sep 11 09:18:01.490711 ip-10-0-128-166 kubenswrapper[2572]: W0911 09:18:01.490683 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod92387e99_7337_4166_9201_c3b737a2238a.slice/crio-e3839058253aac29a246a4212dd5ce6af028cbf6963aad579310d3477113d910 WatchSource:0}: Error finding container e3839058253aac29a246a4212dd5ce6af028cbf6963aad579310d3477113d910: Status 404 returned error can't find the container with id e3839058253aac29a246a4212dd5ce6af028cbf6963aad579310d3477113d910 Sep 11 09:18:01.694777 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:18:01.694755 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-86695b7cc7-xp8k6_8382b5d8-4142-449a-93e9-02c5666f9848/volume-data-source-validator/0.log" Sep 11 09:18:02.206134 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:18:02.206104 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qk7h7/perf-node-gather-daemonset-w75mh" event={"ID":"92387e99-7337-4166-9201-c3b737a2238a","Type":"ContainerStarted","Data":"c2bc2bd4074ef4a8e07270b5b1391655e120cf5bcdc15b45d36466c9aafd200d"} Sep 11 09:18:02.206348 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:18:02.206159 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qk7h7/perf-node-gather-daemonset-w75mh" event={"ID":"92387e99-7337-4166-9201-c3b737a2238a","Type":"ContainerStarted","Data":"e3839058253aac29a246a4212dd5ce6af028cbf6963aad579310d3477113d910"} Sep 11 09:18:02.206348 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:18:02.206198 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-qk7h7/perf-node-gather-daemonset-w75mh" Sep 11 09:18:02.226995 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:18:02.226951 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-qk7h7/perf-node-gather-daemonset-w75mh" podStartSLOduration=2.226938255 podStartE2EDuration="2.226938255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 09:18:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 09:18:02.225656095 +0000 UTC m=+1902.500231791" watchObservedRunningTime="2026-09-11 09:18:02.226938255 +0000 UTC m=+1902.501513950" Sep 11 09:18:02.974270 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:18:02.974245 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-d67t9_8ea2d42c-f484-483d-8714-603b8f05cc54/dns/0.log" Sep 11 09:18:03.022201 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:18:03.022174 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-d67t9_8ea2d42c-f484-483d-8714-603b8f05cc54/kube-rbac-proxy/0.log" Sep 11 09:18:03.090320 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:18:03.090278 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-4vdzw_82687de7-ba90-4516-99cc-43f545df75bd/dns-node-resolver/0.log" Sep 11 09:18:03.774093 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:18:03.774064 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-67p7z_eb6eba26-53e8-4946-9491-cf154a8286f5/node-ca/0.log" Sep 11 09:18:04.700924 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:18:04.700900 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-57979bfb5b-9fr79_a882da04-e40f-472b-b823-dd7650334c17/router/0.log" Sep 11 09:18:05.050271 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:18:05.050187 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-2tlm8_a9917159-f37b-4842-ae12-10c24123dff1/serve-healthcheck-canary/0.log" Sep 11 09:18:05.474446 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:18:05.474422 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-xjhvj_8a3bbadc-80e8-4436-9c8e-077a0dd03df3/insights-operator/0.log" Sep 11 09:18:05.474722 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:18:05.474704 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-xjhvj_8a3bbadc-80e8-4436-9c8e-077a0dd03df3/insights-operator/1.log" Sep 11 09:18:05.690299 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:18:05.690273 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-srrd7_b67622ee-ebc8-4db2-8211-df0e6e15cefd/kube-rbac-proxy/0.log" Sep 11 09:18:05.720787 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:18:05.720766 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-srrd7_b67622ee-ebc8-4db2-8211-df0e6e15cefd/exporter/0.log" Sep 11 09:18:05.744357 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:18:05.744304 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-srrd7_b67622ee-ebc8-4db2-8211-df0e6e15cefd/extractor/0.log" Sep 11 09:18:08.219200 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:18:08.219172 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-qk7h7/perf-node-gather-daemonset-w75mh" Sep 11 09:18:10.710901 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:18:10.710868 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-wdwcs_9222c5e5-755c-460e-8568-db40848b8123/kube-storage-version-migrator-operator/1.log" Sep 11 09:18:10.711644 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:18:10.711625 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-wdwcs_9222c5e5-755c-460e-8568-db40848b8123/kube-storage-version-migrator-operator/0.log" Sep 11 09:18:11.701333 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:18:11.701303 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-4w5tl_bb4b69fd-d8d3-4134-94c5-16faabe2ec30/kube-multus/0.log" Sep 11 09:18:11.943751 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:18:11.943722 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-k599z_0c0c64bd-73f4-4cb4-b660-c7232ff8e573/kube-multus-additional-cni-plugins/0.log" Sep 11 09:18:11.970443 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:18:11.970422 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-k599z_0c0c64bd-73f4-4cb4-b660-c7232ff8e573/egress-router-binary-copy/0.log" Sep 11 09:18:11.994800 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:18:11.994778 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-k599z_0c0c64bd-73f4-4cb4-b660-c7232ff8e573/cni-plugins/0.log" Sep 11 09:18:12.029416 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:18:12.029370 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-k599z_0c0c64bd-73f4-4cb4-b660-c7232ff8e573/bond-cni-plugin/0.log" Sep 11 09:18:12.055974 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:18:12.055955 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-k599z_0c0c64bd-73f4-4cb4-b660-c7232ff8e573/routeoverride-cni/0.log" Sep 11 09:18:12.078135 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:18:12.078114 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-k599z_0c0c64bd-73f4-4cb4-b660-c7232ff8e573/whereabouts-cni-bincopy/0.log" Sep 11 09:18:12.100778 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:18:12.100758 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-k599z_0c0c64bd-73f4-4cb4-b660-c7232ff8e573/whereabouts-cni/0.log" Sep 11 09:18:12.628635 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:18:12.628608 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-ntkhj_e78a69db-9d98-4923-b082-ccb30b77cc9e/network-metrics-daemon/0.log" Sep 11 09:18:12.666900 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:18:12.666873 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-ntkhj_e78a69db-9d98-4923-b082-ccb30b77cc9e/kube-rbac-proxy/0.log" Sep 11 09:18:13.769066 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:18:13.769039 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lhkkf_83409be8-1bb2-43dc-af76-d3d74e569101/ovn-controller/0.log" Sep 11 09:18:13.789403 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:18:13.789378 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lhkkf_83409be8-1bb2-43dc-af76-d3d74e569101/ovn-acl-logging/0.log" Sep 11 09:18:13.830320 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:18:13.830293 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lhkkf_83409be8-1bb2-43dc-af76-d3d74e569101/kube-rbac-proxy-node/0.log" Sep 11 09:18:13.863955 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:18:13.863929 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lhkkf_83409be8-1bb2-43dc-af76-d3d74e569101/kube-rbac-proxy-ovn-metrics/0.log" Sep 11 09:18:13.890188 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:18:13.890167 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lhkkf_83409be8-1bb2-43dc-af76-d3d74e569101/northd/0.log" Sep 11 09:18:13.916193 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:18:13.916167 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lhkkf_83409be8-1bb2-43dc-af76-d3d74e569101/nbdb/0.log" Sep 11 09:18:13.949509 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:18:13.949489 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lhkkf_83409be8-1bb2-43dc-af76-d3d74e569101/sbdb/0.log" Sep 11 09:18:14.004939 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:18:14.004921 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lhkkf_83409be8-1bb2-43dc-af76-d3d74e569101/ovnkube-controller/0.log" Sep 11 09:18:15.338440 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:18:15.338416 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-source-6f8b768455-f6twl_356c3904-2228-43a3-9045-b34e49f0c527/check-endpoints/0.log" Sep 11 09:18:15.363392 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:18:15.363352 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-82xp9_12ac768c-2c6a-43e7-9bd3-44679781ebd7/network-check-target-container/0.log" Sep 11 09:18:16.405219 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:18:16.405191 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-v5rw5_510eab66-1dcf-4ffc-983d-d570fb9e9fda/iptables-alerter/0.log" Sep 11 09:18:17.146058 ip-10-0-128-166 kubenswrapper[2572]: I0911 09:18:17.146021 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-n56xz_46676d4c-0012-4617-9fe5-73d33015eb8c/tuned/0.log"