Sep 16 07:27:46.882509 ip-10-0-141-221 systemd[1]: Starting Kubernetes Kubelet... Sep 16 07:27:47.287256 ip-10-0-141-221 kubenswrapper[2557]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 16 07:27:47.287256 ip-10-0-141-221 kubenswrapper[2557]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 16 07:27:47.287256 ip-10-0-141-221 kubenswrapper[2557]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 16 07:27:47.287256 ip-10-0-141-221 kubenswrapper[2557]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 16 07:27:47.287256 ip-10-0-141-221 kubenswrapper[2557]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 16 07:27:47.288312 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.288213 2557 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 16 07:27:47.293113 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.293090 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 07:27:47.293113 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.293108 2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 07:27:47.293113 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.293113 2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 07:27:47.293113 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.293117 2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 07:27:47.293113 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.293121 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 07:27:47.293427 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.293125 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 07:27:47.293427 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.293129 2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 07:27:47.293427 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.293133 2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 07:27:47.293427 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.293139 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 07:27:47.293427 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.293142 2557 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 07:27:47.293427 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.293146 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 07:27:47.293427 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.293150 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 07:27:47.293427 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.293154 2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 07:27:47.293427 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.293157 2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 07:27:47.293427 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.293161 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 07:27:47.293427 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.293169 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 07:27:47.293427 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.293173 2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 07:27:47.293427 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.293176 2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 07:27:47.293427 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.293180 2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 07:27:47.293427 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.293184 2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 07:27:47.293427 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.293191 2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 07:27:47.293427 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.293197 2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 07:27:47.293427 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.293201 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 07:27:47.293427 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.293205 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 07:27:47.294202 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.293210 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 07:27:47.294202 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.293214 2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 07:27:47.294202 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.293218 2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 07:27:47.294202 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.293222 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 07:27:47.294202 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.293226 2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 07:27:47.294202 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.293230 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 07:27:47.294202 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.293254 2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 07:27:47.294202 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.293258 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 07:27:47.294202 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.293263 2557 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 07:27:47.294202 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.293267 2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 07:27:47.294202 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.293271 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 07:27:47.294202 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.293275 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 07:27:47.294202 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.293279 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 07:27:47.294202 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.293284 2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 07:27:47.294202 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.293289 2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 07:27:47.294202 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.293293 2557 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 07:27:47.294202 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.293297 2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 07:27:47.294202 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.293301 2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 07:27:47.294202 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.293309 2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 07:27:47.295079 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.293314 2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 07:27:47.295079 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.293319 2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 07:27:47.295079 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.293324 2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 07:27:47.295079 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.293329 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 07:27:47.295079 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.293335 2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 07:27:47.295079 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.293339 2557 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 07:27:47.295079 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.293343 2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 07:27:47.295079 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.293347 2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 07:27:47.295079 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.293351 2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 07:27:47.295079 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.293355 2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 07:27:47.295079 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.293359 2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 07:27:47.295079 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.293363 2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 07:27:47.295079 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.293368 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 07:27:47.295079 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.293372 2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 07:27:47.295079 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.293377 2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 07:27:47.295079 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.293381 2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 07:27:47.295079 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.293385 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 07:27:47.295079 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.293389 2557 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 07:27:47.295079 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.293393 2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 07:27:47.295079 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.293398 2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 07:27:47.295946 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.293403 2557 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 07:27:47.295946 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.293407 2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 07:27:47.295946 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.293411 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 07:27:47.295946 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.293415 2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 07:27:47.295946 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.293420 2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 07:27:47.295946 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.293424 2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 07:27:47.295946 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.293428 2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 07:27:47.295946 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.293433 2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 07:27:47.295946 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.293439 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 07:27:47.295946 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.293443 2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 07:27:47.295946 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.293447 2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 07:27:47.295946 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.293451 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 07:27:47.295946 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.293456 2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 07:27:47.295946 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.293460 2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 07:27:47.295946 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.293464 2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 07:27:47.295946 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.293468 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 07:27:47.295946 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.293473 2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 07:27:47.295946 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.293477 2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 07:27:47.295946 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.293482 2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 07:27:47.295946 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.293486 2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 07:27:47.296827 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.293490 2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 07:27:47.296827 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.293495 2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 07:27:47.296827 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.293499 2557 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 07:27:47.296827 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.293504 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 07:27:47.296827 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.293508 2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 07:27:47.296827 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.293512 2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 07:27:47.296827 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.293516 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 07:27:47.296827 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.293520 2557 feature_gate.go:328] unrecognized feature gate: Example Sep 16 07:27:47.296827 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.293524 2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 07:27:47.296827 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.293529 2557 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 07:27:47.296827 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.293535 2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 07:27:47.296827 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.293541 2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 07:27:47.296827 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.293546 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 07:27:47.296827 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.293551 2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 07:27:47.296827 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.293555 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 07:27:47.296827 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.293559 2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 07:27:47.296827 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.294716 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 07:27:47.296827 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.294727 2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 07:27:47.296827 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.294732 2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 07:27:47.296827 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.294736 2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 07:27:47.297703 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.294741 2557 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 07:27:47.297703 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.294746 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 07:27:47.297703 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.294751 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 07:27:47.297703 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.294756 2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 07:27:47.297703 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.294761 2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 07:27:47.297703 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.294766 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 07:27:47.297703 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.294770 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 07:27:47.297703 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.294775 2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 07:27:47.297703 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.294779 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 07:27:47.297703 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.294784 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 07:27:47.297703 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.294790 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 07:27:47.297703 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.294794 2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 07:27:47.297703 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.294799 2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 07:27:47.297703 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.294803 2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 07:27:47.297703 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.294807 2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 07:27:47.297703 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.294811 2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 07:27:47.297703 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.294815 2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 07:27:47.297703 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.294820 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 07:27:47.297703 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.294824 2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 07:27:47.298219 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.294830 2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 07:27:47.298219 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.294836 2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 07:27:47.298219 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.294841 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 07:27:47.298219 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.294846 2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 07:27:47.298219 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.294851 2557 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 07:27:47.298219 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.294856 2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 07:27:47.298219 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.294860 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 07:27:47.298219 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.294864 2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 07:27:47.298219 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.294869 2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 07:27:47.298219 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.294875 2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 07:27:47.298219 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.294881 2557 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 07:27:47.298219 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.294885 2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 07:27:47.298219 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.294889 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 07:27:47.298219 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.294893 2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 07:27:47.298219 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.294898 2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 07:27:47.298219 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.294901 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 07:27:47.298219 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.294905 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 07:27:47.298219 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.294910 2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 07:27:47.298219 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.294913 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 07:27:47.298778 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.294918 2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 07:27:47.298778 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.294923 2557 feature_gate.go:328] unrecognized feature gate: Example Sep 16 07:27:47.298778 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.294927 2557 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 07:27:47.298778 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.294931 2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 07:27:47.298778 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.294935 2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 07:27:47.298778 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.294940 2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 07:27:47.298778 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.294945 2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 07:27:47.298778 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.294949 2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 07:27:47.298778 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.294952 2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 07:27:47.298778 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.294956 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 07:27:47.298778 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.294961 2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 07:27:47.298778 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.294966 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 07:27:47.298778 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.294971 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 07:27:47.298778 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.294975 2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 07:27:47.298778 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.294979 2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 07:27:47.298778 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.294984 2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 07:27:47.298778 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.294988 2557 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 07:27:47.298778 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.294993 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 07:27:47.298778 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.294997 2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 07:27:47.298778 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.295001 2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 07:27:47.299330 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.295005 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 07:27:47.299330 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.295010 2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 07:27:47.299330 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.295014 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 07:27:47.299330 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.295018 2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 07:27:47.299330 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.295022 2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 07:27:47.299330 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.295026 2557 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 07:27:47.299330 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.295030 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 07:27:47.299330 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.295034 2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 07:27:47.299330 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.295038 2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 07:27:47.299330 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.295042 2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 07:27:47.299330 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.295046 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 07:27:47.299330 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.295051 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 07:27:47.299330 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.295055 2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 07:27:47.299330 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.295059 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 07:27:47.299330 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.295063 2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 07:27:47.299330 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.295067 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 07:27:47.299330 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.295073 2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 07:27:47.299330 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.295079 2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 07:27:47.299330 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.295085 2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 07:27:47.299830 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.295089 2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 07:27:47.299830 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.295093 2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 07:27:47.299830 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.295097 2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 07:27:47.299830 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.295102 2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 07:27:47.299830 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.295106 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 07:27:47.299830 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.295110 2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 07:27:47.299830 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.295114 2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 07:27:47.299830 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.295118 2557 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 07:27:47.299830 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.295123 2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 07:27:47.299830 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.295127 2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 07:27:47.299830 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.295131 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 07:27:47.299830 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.295136 2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 07:27:47.299830 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.295140 2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 07:27:47.299830 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.295145 2557 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 07:27:47.299830 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.295149 2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 07:27:47.299830 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.295153 2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 07:27:47.299830 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.295157 2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 07:27:47.299830 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.295162 2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 07:27:47.299830 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.295760 2557 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 07:27:47.299830 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.295768 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 07:27:47.300366 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.295773 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 07:27:47.300366 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.295780 2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 07:27:47.300366 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.295786 2557 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 07:27:47.300366 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.295792 2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 07:27:47.300366 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.295797 2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 07:27:47.300366 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.295802 2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 07:27:47.300366 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.295806 2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 07:27:47.300366 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.295810 2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 07:27:47.300366 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.295815 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 07:27:47.300366 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.295819 2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 07:27:47.300366 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.295822 2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 07:27:47.300366 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.295832 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 07:27:47.300366 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.295836 2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 07:27:47.300366 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.295841 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 07:27:47.300366 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.295845 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 07:27:47.300366 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.295849 2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 07:27:47.300366 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.295853 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 07:27:47.300366 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.295857 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 07:27:47.300366 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.295861 2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 07:27:47.300833 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.295865 2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 07:27:47.300833 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.295870 2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 07:27:47.300833 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.295874 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 07:27:47.300833 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.295878 2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 07:27:47.300833 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.295882 2557 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 07:27:47.300833 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.295887 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 07:27:47.300833 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.295891 2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 07:27:47.300833 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.295895 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 07:27:47.300833 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.295899 2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 07:27:47.300833 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.295903 2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 07:27:47.300833 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.295907 2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 07:27:47.300833 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.295911 2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 07:27:47.300833 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.295915 2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 07:27:47.300833 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.295919 2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 07:27:47.300833 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.295923 2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 07:27:47.300833 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.295928 2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 07:27:47.300833 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.295932 2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 07:27:47.300833 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.295936 2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 07:27:47.300833 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.295941 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 07:27:47.300833 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.295944 2557 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 07:27:47.301362 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.295948 2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 07:27:47.301362 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.295952 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 07:27:47.301362 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.295956 2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 07:27:47.301362 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.295960 2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 07:27:47.301362 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.295963 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 07:27:47.301362 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.295968 2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 07:27:47.301362 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.295972 2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 07:27:47.301362 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.295978 2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 07:27:47.301362 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.295982 2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 07:27:47.301362 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.295986 2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 07:27:47.301362 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.295990 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 07:27:47.301362 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.295994 2557 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 07:27:47.301362 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.295999 2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 07:27:47.301362 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.296004 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 07:27:47.301362 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.296010 2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 07:27:47.301362 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.296015 2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 07:27:47.301362 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.296020 2557 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 07:27:47.301362 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.296025 2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 07:27:47.301362 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.296029 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 07:27:47.301362 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.296034 2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 07:27:47.301855 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.296038 2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 07:27:47.301855 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.296043 2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 07:27:47.301855 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.296047 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 07:27:47.301855 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.296051 2557 feature_gate.go:328] unrecognized feature gate: Example Sep 16 07:27:47.301855 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.296055 2557 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 07:27:47.301855 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.296059 2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 07:27:47.301855 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.296062 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 07:27:47.301855 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.296067 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 07:27:47.301855 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.296071 2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 07:27:47.301855 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.296076 2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 07:27:47.301855 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.296080 2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 07:27:47.301855 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.296084 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 07:27:47.301855 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.296088 2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 07:27:47.301855 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.296092 2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 07:27:47.301855 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.296096 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 07:27:47.301855 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.296103 2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 07:27:47.301855 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.296108 2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 07:27:47.301855 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.296112 2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 07:27:47.301855 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.296117 2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 07:27:47.302397 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.296121 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 07:27:47.302397 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.296126 2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 07:27:47.302397 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.296130 2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 07:27:47.302397 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.296134 2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 07:27:47.302397 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.296138 2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 07:27:47.302397 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.296142 2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 07:27:47.302397 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.296146 2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 07:27:47.302397 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.296150 2557 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 07:27:47.302397 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.296155 2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 07:27:47.302397 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.296159 2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 07:27:47.302397 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.296163 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 07:27:47.302397 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.296167 2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 07:27:47.302397 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.296171 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 07:27:47.302397 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.296175 2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 07:27:47.302397 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.296179 2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 07:27:47.302397 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.296184 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 07:27:47.302397 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.296188 2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 07:27:47.302397 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.296193 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 07:27:47.302397 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.296197 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 07:27:47.302397 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.296309 2557 flags.go:64] FLAG: --address="0.0.0.0" Sep 16 07:27:47.302397 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.296359 2557 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 16 07:27:47.303055 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.296368 2557 flags.go:64] FLAG: --anonymous-auth="true" Sep 16 07:27:47.303055 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.296375 2557 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 16 07:27:47.303055 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.296381 2557 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 16 07:27:47.303055 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.296387 2557 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 16 07:27:47.303055 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.296393 2557 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 16 07:27:47.303055 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.296399 2557 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 16 07:27:47.303055 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.296404 2557 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 16 07:27:47.303055 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.296409 2557 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 16 07:27:47.303055 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.296414 2557 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 16 07:27:47.303055 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.296419 2557 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 16 07:27:47.303055 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.296424 2557 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 16 07:27:47.303055 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.296431 2557 flags.go:64] FLAG: --cgroup-root="" Sep 16 07:27:47.303055 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.296435 2557 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 16 07:27:47.303055 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.296440 2557 flags.go:64] FLAG: --client-ca-file="" Sep 16 07:27:47.303055 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.296445 2557 flags.go:64] FLAG: --cloud-provider="external" Sep 16 07:27:47.303055 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.296450 2557 flags.go:64] FLAG: --cluster-dns="[]" Sep 16 07:27:47.303055 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.296456 2557 flags.go:64] FLAG: --cluster-domain="" Sep 16 07:27:47.303055 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.296461 2557 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 16 07:27:47.303055 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.296468 2557 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 16 07:27:47.303055 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.296474 2557 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 16 07:27:47.303055 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.296479 2557 flags.go:64] FLAG: --container-log-max-files="5" Sep 16 07:27:47.303055 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.296485 2557 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 16 07:27:47.303055 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.296490 2557 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 16 07:27:47.303628 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.296494 2557 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 16 07:27:47.303628 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.296500 2557 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 16 07:27:47.303628 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.296505 2557 flags.go:64] FLAG: --contention-profiling="false" Sep 16 07:27:47.303628 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.296510 2557 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 16 07:27:47.303628 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.296514 2557 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 16 07:27:47.303628 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.296519 2557 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 16 07:27:47.303628 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.296524 2557 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 16 07:27:47.303628 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.296530 2557 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 16 07:27:47.303628 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.296535 2557 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 16 07:27:47.303628 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.296540 2557 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 16 07:27:47.303628 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.296545 2557 flags.go:64] FLAG: --enable-load-reader="false" Sep 16 07:27:47.303628 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.296550 2557 flags.go:64] FLAG: --enable-server="true" Sep 16 07:27:47.303628 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.296555 2557 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 16 07:27:47.303628 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.296561 2557 flags.go:64] FLAG: --event-burst="100" Sep 16 07:27:47.303628 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.296566 2557 flags.go:64] FLAG: --event-qps="50" Sep 16 07:27:47.303628 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.296571 2557 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 16 07:27:47.303628 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.296576 2557 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 16 07:27:47.303628 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.296580 2557 flags.go:64] FLAG: --eviction-hard="" Sep 16 07:27:47.303628 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.296586 2557 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 16 07:27:47.303628 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.296591 2557 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 16 07:27:47.303628 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.296596 2557 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 16 07:27:47.303628 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.296601 2557 flags.go:64] FLAG: --eviction-soft="" Sep 16 07:27:47.303628 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.296606 2557 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 16 07:27:47.303628 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.296611 2557 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 16 07:27:47.303628 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.296615 2557 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 16 07:27:47.304405 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.296620 2557 flags.go:64] FLAG: --experimental-mounter-path="" Sep 16 07:27:47.304405 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.296625 2557 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 16 07:27:47.304405 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.296631 2557 flags.go:64] FLAG: --fail-swap-on="true" Sep 16 07:27:47.304405 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.296636 2557 flags.go:64] FLAG: --feature-gates="" Sep 16 07:27:47.304405 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.296642 2557 flags.go:64] FLAG: --file-check-frequency="20s" Sep 16 07:27:47.304405 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.296647 2557 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 16 07:27:47.304405 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.296652 2557 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 16 07:27:47.304405 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.296656 2557 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 16 07:27:47.304405 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.296661 2557 flags.go:64] FLAG: --healthz-port="10248" Sep 16 07:27:47.304405 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.296666 2557 flags.go:64] FLAG: --help="false" Sep 16 07:27:47.304405 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.296670 2557 flags.go:64] FLAG: --hostname-override="ip-10-0-141-221.ec2.internal" Sep 16 07:27:47.304405 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.296676 2557 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 16 07:27:47.304405 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.296681 2557 flags.go:64] FLAG: --http-check-frequency="20s" Sep 16 07:27:47.304405 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.296686 2557 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 16 07:27:47.304405 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.296691 2557 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 16 07:27:47.304405 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.296697 2557 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 16 07:27:47.304405 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.296702 2557 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 16 07:27:47.304405 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.296706 2557 flags.go:64] FLAG: --image-service-endpoint="" Sep 16 07:27:47.304405 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.296711 2557 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 16 07:27:47.304405 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.296716 2557 flags.go:64] FLAG: --kube-api-burst="100" Sep 16 07:27:47.304405 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.296721 2557 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 16 07:27:47.304405 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.296726 2557 flags.go:64] FLAG: --kube-api-qps="50" Sep 16 07:27:47.304405 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.296731 2557 flags.go:64] FLAG: --kube-reserved="" Sep 16 07:27:47.304405 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.296735 2557 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 16 07:27:47.305019 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.296740 2557 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 16 07:27:47.305019 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.296745 2557 flags.go:64] FLAG: --kubelet-cgroups="" Sep 16 07:27:47.305019 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.296750 2557 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 16 07:27:47.305019 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.296754 2557 flags.go:64] FLAG: --lock-file="" Sep 16 07:27:47.305019 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.296759 2557 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 16 07:27:47.305019 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.296766 2557 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 16 07:27:47.305019 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.296771 2557 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 16 07:27:47.305019 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.296779 2557 flags.go:64] FLAG: --log-json-split-stream="false" Sep 16 07:27:47.305019 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.296784 2557 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 16 07:27:47.305019 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.296788 2557 flags.go:64] FLAG: --log-text-split-stream="false" Sep 16 07:27:47.305019 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.296794 2557 flags.go:64] FLAG: --logging-format="text" Sep 16 07:27:47.305019 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.296799 2557 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 16 07:27:47.305019 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.296805 2557 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 16 07:27:47.305019 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.296809 2557 flags.go:64] FLAG: --manifest-url="" Sep 16 07:27:47.305019 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.296814 2557 flags.go:64] FLAG: --manifest-url-header="" Sep 16 07:27:47.305019 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.296821 2557 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 16 07:27:47.305019 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.296825 2557 flags.go:64] FLAG: --max-open-files="1000000" Sep 16 07:27:47.305019 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.296831 2557 flags.go:64] FLAG: --max-pods="110" Sep 16 07:27:47.305019 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.296835 2557 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 16 07:27:47.305019 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.296840 2557 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 16 07:27:47.305019 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.296844 2557 flags.go:64] FLAG: --memory-manager-policy="None" Sep 16 07:27:47.305019 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.296848 2557 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 16 07:27:47.305019 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.296853 2557 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 16 07:27:47.305019 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.296861 2557 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 16 07:27:47.305019 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.296866 2557 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 16 07:27:47.305724 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.296877 2557 flags.go:64] FLAG: --node-status-max-images="50" Sep 16 07:27:47.305724 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.296882 2557 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 16 07:27:47.305724 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.296887 2557 flags.go:64] FLAG: --oom-score-adj="-999" Sep 16 07:27:47.305724 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.296891 2557 flags.go:64] FLAG: --pod-cidr="" Sep 16 07:27:47.305724 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.296896 2557 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10" Sep 16 07:27:47.305724 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.296904 2557 flags.go:64] FLAG: --pod-manifest-path="" Sep 16 07:27:47.305724 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.296909 2557 flags.go:64] FLAG: --pod-max-pids="-1" Sep 16 07:27:47.305724 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.296914 2557 flags.go:64] FLAG: --pods-per-core="0" Sep 16 07:27:47.305724 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.296918 2557 flags.go:64] FLAG: --port="10250" Sep 16 07:27:47.305724 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.296923 2557 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 16 07:27:47.305724 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.296927 2557 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-030d0a5f40acae8ee" Sep 16 07:27:47.305724 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.296932 2557 flags.go:64] FLAG: --qos-reserved="" Sep 16 07:27:47.305724 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.296937 2557 flags.go:64] FLAG: --read-only-port="10255" Sep 16 07:27:47.305724 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.296943 2557 flags.go:64] FLAG: --register-node="true" Sep 16 07:27:47.305724 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.296948 2557 flags.go:64] FLAG: --register-with-taints="" Sep 16 07:27:47.305724 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.296954 2557 flags.go:64] FLAG: --registry-burst="10" Sep 16 07:27:47.305724 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.296958 2557 flags.go:64] FLAG: --registry-qps="5" Sep 16 07:27:47.305724 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.296963 2557 flags.go:64] FLAG: --reserved-cpus="" Sep 16 07:27:47.305724 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.296969 2557 flags.go:64] FLAG: --reserved-memory="" Sep 16 07:27:47.305724 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.296975 2557 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 16 07:27:47.305724 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.296980 2557 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 16 07:27:47.305724 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.296985 2557 flags.go:64] FLAG: --rotate-certificates="false" Sep 16 07:27:47.305724 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.296990 2557 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 16 07:27:47.305724 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.296994 2557 flags.go:64] FLAG: --runonce="false" Sep 16 07:27:47.305724 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.296999 2557 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 16 07:27:47.306366 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.297004 2557 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 16 07:27:47.306366 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.297009 2557 flags.go:64] FLAG: --seccomp-default="false" Sep 16 07:27:47.306366 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.297013 2557 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 16 07:27:47.306366 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.297018 2557 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 16 07:27:47.306366 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.297022 2557 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 16 07:27:47.306366 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.297029 2557 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 16 07:27:47.306366 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.297034 2557 flags.go:64] FLAG: --storage-driver-password="root" Sep 16 07:27:47.306366 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.297039 2557 flags.go:64] FLAG: --storage-driver-secure="false" Sep 16 07:27:47.306366 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.297043 2557 flags.go:64] FLAG: --storage-driver-table="stats" Sep 16 07:27:47.306366 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.297048 2557 flags.go:64] FLAG: --storage-driver-user="root" Sep 16 07:27:47.306366 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.297052 2557 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 16 07:27:47.306366 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.297057 2557 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 16 07:27:47.306366 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.297062 2557 flags.go:64] FLAG: --system-cgroups="" Sep 16 07:27:47.306366 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.297067 2557 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 16 07:27:47.306366 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.297075 2557 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 16 07:27:47.306366 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.297080 2557 flags.go:64] FLAG: --tls-cert-file="" Sep 16 07:27:47.306366 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.297085 2557 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 16 07:27:47.306366 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.297092 2557 flags.go:64] FLAG: --tls-min-version="" Sep 16 07:27:47.306366 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.297096 2557 flags.go:64] FLAG: --tls-private-key-file="" Sep 16 07:27:47.306366 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.297101 2557 flags.go:64] FLAG: --topology-manager-policy="none" Sep 16 07:27:47.306366 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.297105 2557 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 16 07:27:47.306366 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.297110 2557 flags.go:64] FLAG: --topology-manager-scope="container" Sep 16 07:27:47.306366 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.297115 2557 flags.go:64] FLAG: --v="2" Sep 16 07:27:47.306366 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.297121 2557 flags.go:64] FLAG: --version="false" Sep 16 07:27:47.306366 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.297128 2557 flags.go:64] FLAG: --vmodule="" Sep 16 07:27:47.307035 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.297134 2557 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 16 07:27:47.307035 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.297140 2557 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 16 07:27:47.307035 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.297328 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 07:27:47.307035 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.297345 2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 07:27:47.307035 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.297351 2557 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 07:27:47.307035 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.297355 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 07:27:47.307035 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.297362 2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 07:27:47.307035 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.297368 2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 07:27:47.307035 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.297372 2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 07:27:47.307035 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.297377 2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 07:27:47.307035 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.297381 2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 07:27:47.307035 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.297386 2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 07:27:47.307035 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.297393 2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 07:27:47.307035 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.297398 2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 07:27:47.307035 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.297402 2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 07:27:47.307035 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.297406 2557 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 07:27:47.307035 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.297411 2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 07:27:47.307035 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.297414 2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 07:27:47.307035 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.297419 2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 07:27:47.307035 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.297423 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 07:27:47.307559 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.297428 2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 07:27:47.307559 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.297432 2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 07:27:47.307559 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.297436 2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 07:27:47.307559 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.297441 2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 07:27:47.307559 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.297445 2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 07:27:47.307559 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.297449 2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 07:27:47.307559 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.297453 2557 feature_gate.go:328] unrecognized feature gate: Example Sep 16 07:27:47.307559 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.297457 2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 07:27:47.307559 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.297461 2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 07:27:47.307559 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.297467 2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 07:27:47.307559 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.297472 2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 07:27:47.307559 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.297476 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 07:27:47.307559 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.297482 2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 07:27:47.307559 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.297486 2557 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 07:27:47.307559 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.297490 2557 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 07:27:47.307559 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.297494 2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 07:27:47.307559 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.297499 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 07:27:47.307559 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.297503 2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 07:27:47.307559 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.297507 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 07:27:47.307559 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.297511 2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 07:27:47.308062 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.297516 2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 07:27:47.308062 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.297520 2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 07:27:47.308062 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.297524 2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 07:27:47.308062 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.297528 2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 07:27:47.308062 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.297534 2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 07:27:47.308062 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.297538 2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 07:27:47.308062 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.297543 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 07:27:47.308062 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.297547 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 07:27:47.308062 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.297551 2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 07:27:47.308062 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.297555 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 07:27:47.308062 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.297560 2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 07:27:47.308062 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.297564 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 07:27:47.308062 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.297569 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 07:27:47.308062 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.297573 2557 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 07:27:47.308062 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.297577 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 07:27:47.308062 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.297581 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 07:27:47.308062 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.297586 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 07:27:47.308062 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.297591 2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 07:27:47.308062 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.297595 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 07:27:47.308062 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.297599 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 07:27:47.308606 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.297603 2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 07:27:47.308606 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.297607 2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 07:27:47.308606 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.297612 2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 07:27:47.308606 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.297616 2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 07:27:47.308606 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.297622 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 07:27:47.308606 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.297626 2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 07:27:47.308606 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.297630 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 07:27:47.308606 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.297635 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 07:27:47.308606 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.297639 2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 07:27:47.308606 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.297643 2557 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 07:27:47.308606 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.297647 2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 07:27:47.308606 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.297651 2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 07:27:47.308606 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.297655 2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 07:27:47.308606 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.297660 2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 07:27:47.308606 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.297664 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 07:27:47.308606 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.297668 2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 07:27:47.308606 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.297678 2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 07:27:47.308606 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.297682 2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 07:27:47.308606 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.297686 2557 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 07:27:47.308606 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.297690 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 07:27:47.309110 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.297694 2557 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 07:27:47.309110 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.297699 2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 07:27:47.309110 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.297702 2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 07:27:47.309110 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.297706 2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 07:27:47.309110 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.297710 2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 07:27:47.309110 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.297716 2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 07:27:47.309110 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.297722 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 07:27:47.309110 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.297727 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 07:27:47.309110 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.297733 2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 07:27:47.309110 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.297739 2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 07:27:47.309110 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.297744 2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 07:27:47.309110 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.297748 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 07:27:47.309110 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.297753 2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 07:27:47.309110 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.297758 2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 07:27:47.309110 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.297762 2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 07:27:47.309110 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.297767 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 07:27:47.309110 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.297774 2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 07:27:47.309110 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.297778 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 07:27:47.309110 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.297782 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 07:27:47.309630 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.297786 2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 07:27:47.309630 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.297791 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 07:27:47.309630 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.298694 2557 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 16 07:27:47.309630 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.308489 2557 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 16 07:27:47.309630 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.308503 2557 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 16 07:27:47.309630 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.309386 2557 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 16 07:27:47.309630 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.309394 2557 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 16 07:27:47.309630 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309449 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 07:27:47.309630 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309454 2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 07:27:47.309630 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309459 2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 07:27:47.309630 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309463 2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 07:27:47.309630 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309467 2557 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 07:27:47.309630 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309470 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 07:27:47.309630 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309473 2557 feature_gate.go:328] unrecognized feature gate: Example Sep 16 07:27:47.309991 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309477 2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 07:27:47.309991 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309481 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 07:27:47.309991 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309484 2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 07:27:47.309991 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309486 2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 07:27:47.309991 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309489 2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 07:27:47.309991 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309491 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 07:27:47.309991 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309494 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 07:27:47.309991 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309496 2557 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 07:27:47.309991 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309499 2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 07:27:47.309991 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309502 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 07:27:47.309991 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309504 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 07:27:47.309991 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309507 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 07:27:47.309991 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309509 2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 07:27:47.309991 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309512 2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 07:27:47.309991 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309515 2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 07:27:47.309991 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309517 2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 07:27:47.309991 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309520 2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 07:27:47.309991 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309523 2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 07:27:47.309991 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309526 2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 07:27:47.309991 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309528 2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 07:27:47.310492 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309531 2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 07:27:47.310492 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309533 2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 07:27:47.310492 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309537 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 07:27:47.310492 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309540 2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 07:27:47.310492 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309543 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 07:27:47.310492 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309545 2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 07:27:47.310492 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309548 2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 07:27:47.310492 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309550 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 07:27:47.310492 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309553 2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 07:27:47.310492 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309556 2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 07:27:47.310492 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309558 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 07:27:47.310492 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309561 2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 07:27:47.310492 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309563 2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 07:27:47.310492 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309566 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 07:27:47.310492 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309569 2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 07:27:47.310492 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309571 2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 07:27:47.310492 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309574 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 07:27:47.310492 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309576 2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 07:27:47.310492 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309579 2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 07:27:47.310492 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309582 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 07:27:47.310960 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309584 2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 07:27:47.310960 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309587 2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 07:27:47.310960 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309589 2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 07:27:47.310960 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309593 2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 07:27:47.310960 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309597 2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 07:27:47.310960 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309600 2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 07:27:47.310960 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309603 2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 07:27:47.310960 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309606 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 07:27:47.310960 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309608 2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 07:27:47.310960 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309611 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 07:27:47.310960 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309614 2557 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 07:27:47.310960 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309617 2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 07:27:47.310960 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309619 2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 07:27:47.310960 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309622 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 07:27:47.310960 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309626 2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 07:27:47.310960 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309629 2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 07:27:47.310960 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309632 2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 07:27:47.310960 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309635 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 07:27:47.310960 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309638 2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 07:27:47.311440 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309640 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 07:27:47.311440 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309643 2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 07:27:47.311440 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309646 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 07:27:47.311440 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309648 2557 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 07:27:47.311440 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309651 2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 07:27:47.311440 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309653 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 07:27:47.311440 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309656 2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 07:27:47.311440 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309659 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 07:27:47.311440 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309661 2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 07:27:47.311440 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309664 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 07:27:47.311440 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309666 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 07:27:47.311440 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309668 2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 07:27:47.311440 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309671 2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 07:27:47.311440 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309674 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 07:27:47.311440 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309676 2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 07:27:47.311440 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309679 2557 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 07:27:47.311440 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309681 2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 07:27:47.311440 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309684 2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 07:27:47.311440 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309686 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 07:27:47.311440 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309689 2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 07:27:47.311956 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309691 2557 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 07:27:47.311956 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309694 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 07:27:47.311956 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309697 2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 07:27:47.311956 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309699 2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 07:27:47.311956 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309701 2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 07:27:47.311956 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309704 2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 07:27:47.311956 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309706 2557 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 07:27:47.311956 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309709 2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 07:27:47.311956 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309712 2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 07:27:47.311956 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309715 2557 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 07:27:47.311956 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309718 2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 07:27:47.311956 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309720 2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 07:27:47.311956 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309723 2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 07:27:47.311956 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.309728 2557 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 16 07:27:47.311956 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309831 2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 07:27:47.312340 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309836 2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 07:27:47.312340 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309839 2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 07:27:47.312340 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309842 2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 07:27:47.312340 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309845 2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 07:27:47.312340 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309848 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 07:27:47.312340 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309851 2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 07:27:47.312340 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309853 2557 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 07:27:47.312340 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309856 2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 07:27:47.312340 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309859 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 07:27:47.312340 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309861 2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 07:27:47.312340 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309864 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 07:27:47.312340 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309867 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 07:27:47.312340 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309869 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 07:27:47.312340 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309872 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 07:27:47.312340 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309874 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 07:27:47.312340 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309877 2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 07:27:47.312340 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309879 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 07:27:47.312340 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309882 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 07:27:47.312340 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309884 2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 07:27:47.312340 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309887 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 07:27:47.312826 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309889 2557 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 07:27:47.312826 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309892 2557 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 07:27:47.312826 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309894 2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 07:27:47.312826 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309897 2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 07:27:47.312826 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309899 2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 07:27:47.312826 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309902 2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 07:27:47.312826 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309905 2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 07:27:47.312826 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309908 2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 07:27:47.312826 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309910 2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 07:27:47.312826 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309913 2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 07:27:47.312826 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309915 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 07:27:47.312826 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309918 2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 07:27:47.312826 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309920 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 07:27:47.312826 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309923 2557 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 07:27:47.312826 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309925 2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 07:27:47.312826 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309928 2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 07:27:47.312826 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309930 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 07:27:47.312826 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309933 2557 feature_gate.go:328] unrecognized feature gate: Example Sep 16 07:27:47.312826 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309935 2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 07:27:47.312826 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309938 2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 07:27:47.313322 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309940 2557 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 07:27:47.313322 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309943 2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 07:27:47.313322 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309945 2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 07:27:47.313322 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309948 2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 07:27:47.313322 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309950 2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 07:27:47.313322 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309953 2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 07:27:47.313322 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309955 2557 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 07:27:47.313322 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309958 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 07:27:47.313322 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309961 2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 07:27:47.313322 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309964 2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 07:27:47.313322 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309966 2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 07:27:47.313322 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309969 2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 07:27:47.313322 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309972 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 07:27:47.313322 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309975 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 07:27:47.313322 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309977 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 07:27:47.313322 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309980 2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 07:27:47.313322 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309982 2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 07:27:47.313322 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309985 2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 07:27:47.313322 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309987 2557 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 07:27:47.313322 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309990 2557 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 07:27:47.313801 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309993 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 07:27:47.313801 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309996 2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 07:27:47.313801 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.309998 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 07:27:47.313801 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.310001 2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 07:27:47.313801 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.310004 2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 07:27:47.313801 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.310006 2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 07:27:47.313801 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.310010 2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 07:27:47.313801 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.310013 2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 07:27:47.313801 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.310016 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 07:27:47.313801 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.310020 2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 07:27:47.313801 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.310023 2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 07:27:47.313801 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.310026 2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 07:27:47.313801 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.310028 2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 07:27:47.313801 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.310032 2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 07:27:47.313801 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.310035 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 07:27:47.313801 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.310038 2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 07:27:47.313801 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.310041 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 07:27:47.313801 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.310045 2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 07:27:47.313801 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.310048 2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 07:27:47.314262 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.310051 2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 07:27:47.314262 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.310054 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 07:27:47.314262 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.310057 2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 07:27:47.314262 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.310060 2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 07:27:47.314262 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.310063 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 07:27:47.314262 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.310066 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 07:27:47.314262 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.310069 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 07:27:47.314262 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.310072 2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 07:27:47.314262 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.310074 2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 07:27:47.314262 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.310077 2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 07:27:47.314262 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.310079 2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 07:27:47.314262 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.310082 2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 07:27:47.314262 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.310084 2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 07:27:47.314262 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.310086 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 07:27:47.314262 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.310089 2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 07:27:47.314262 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.310092 2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 07:27:47.314262 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.310094 2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 07:27:47.314262 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.310097 2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 07:27:47.314262 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:47.310099 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 07:27:47.314727 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.310104 2557 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 16 07:27:47.314727 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.310192 2557 server.go:962] "Client rotation is on, will bootstrap in background" Sep 16 07:27:47.314727 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.312436 2557 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 16 07:27:47.314727 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.313562 2557 server.go:1019] "Starting client certificate rotation" Sep 16 07:27:47.314727 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.313659 2557 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 16 07:27:47.314727 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.313695 2557 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 16 07:27:47.316404 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.316391 2557 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 16 07:27:47.319458 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.319439 2557 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 16 07:27:47.330136 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.330117 2557 log.go:25] "Validated CRI v1 runtime API" Sep 16 07:27:47.334771 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.334756 2557 log.go:25] "Validated CRI v1 image API" Sep 16 07:27:47.335577 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.335552 2557 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 16 07:27:47.339635 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.339606 2557 fs.go:135] Filesystem UUIDs: map[2ff432e4-57f2-49a7-b0fa-c819727559f6:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 c6cfe10e-b957-4702-aa36-fdcb62ddbd20:/dev/nvme0n1p4] Sep 16 07:27:47.339717 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.339633 2557 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 16 07:27:47.343522 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.343498 2557 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 16 07:27:47.345047 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.344947 2557 manager.go:216] Machine: {Timestamp:2026-09-16 07:27:47.343452617 +0000 UTC m=+0.370353623 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101352 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec202e724d8c452ac1c0c761922322c9 SystemUUID:ec202e72-4d8c-452a-c1c0-c761922322c9 BootID:78569f5c-468d-4c9c-a750-12ef8e09108f Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:c5:ca:1a:91 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:c5:ca:1a:91 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:6e:27:ae:12:91:ea Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 16 07:27:47.345047 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.345042 2557 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 16 07:27:47.345160 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.345111 2557 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 16 07:27:47.347941 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.347921 2557 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 16 07:27:47.348071 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.347944 2557 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-221.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 16 07:27:47.348764 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.348753 2557 topology_manager.go:138] "Creating topology manager with none policy" Sep 16 07:27:47.348799 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.348765 2557 container_manager_linux.go:309] "Creating device plugin manager" Sep 16 07:27:47.348799 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.348772 2557 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 16 07:27:47.349603 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.349593 2557 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 16 07:27:47.350752 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.350743 2557 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 16 07:27:47.355267 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.355256 2557 state_mem.go:36] "Initialized new in-memory state store" Sep 16 07:27:47.355372 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.355363 2557 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 16 07:27:47.355401 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.355392 2557 kubelet.go:486] "Attempting to sync node with API server" Sep 16 07:27:47.355428 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.355403 2557 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 16 07:27:47.355428 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.355413 2557 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 16 07:27:47.355428 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.355423 2557 kubelet.go:398] "Adding apiserver pod source" Sep 16 07:27:47.355523 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.355431 2557 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 16 07:27:47.355749 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.355738 2557 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 16 07:27:47.355795 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.355756 2557 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 16 07:27:47.358208 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.358191 2557 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 16 07:27:47.361124 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.361111 2557 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 16 07:27:47.361170 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.361141 2557 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 16 07:27:47.363524 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.363504 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 16 07:27:47.363600 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.363531 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 16 07:27:47.363600 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.363541 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 16 07:27:47.363600 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.363550 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 16 07:27:47.363600 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.363563 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 16 07:27:47.363600 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.363578 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 16 07:27:47.363600 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.363598 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 16 07:27:47.363780 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.363607 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 16 07:27:47.363780 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.363619 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 16 07:27:47.363780 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.363628 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 16 07:27:47.363780 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.363688 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 16 07:27:47.363780 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.363705 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 16 07:27:47.365276 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.365254 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 16 07:27:47.365348 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.365289 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 16 07:27:47.367340 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:27:47.367318 2557 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-221.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 16 07:27:47.367478 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:27:47.367357 2557 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 16 07:27:47.370043 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.370027 2557 server.go:1268] "Started kubelet" Sep 16 07:27:47.370128 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.370081 2557 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 16 07:27:47.370178 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.370103 2557 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 16 07:27:47.370178 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.370165 2557 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 16 07:27:47.370798 ip-10-0-141-221 systemd[1]: Started Kubernetes Kubelet. Sep 16 07:27:47.371635 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.371615 2557 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 16 07:27:47.372057 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.372041 2557 server.go:310] "Adding debug handlers to kubelet server" Sep 16 07:27:47.373476 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.372560 2557 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-ttrlv" Sep 16 07:27:47.374666 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:27:47.374643 2557 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 16 07:27:47.377993 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.377884 2557 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-221.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 16 07:27:47.379184 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:27:47.378138 2557 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-221.ec2.internal.18d5bc7beba37bc5 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-221.ec2.internal,UID:ip-10-0-141-221.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-221.ec2.internal,},FirstTimestamp:2026-09-16 07:27:47.369999301 +0000 UTC m=+0.396900303,LastTimestamp:2026-09-16 07:27:47.369999301 +0000 UTC m=+0.396900303,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-221.ec2.internal,}" Sep 16 07:27:47.380349 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.380330 2557 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 16 07:27:47.380349 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.380343 2557 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 16 07:27:47.380499 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:27:47.380423 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-221.ec2.internal\" not found" Sep 16 07:27:47.381142 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.381125 2557 factory.go:55] Registering systemd factory Sep 16 07:27:47.381254 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.381158 2557 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 16 07:27:47.381254 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.381168 2557 factory.go:223] Registration of the systemd container factory successfully Sep 16 07:27:47.381254 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.381180 2557 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 16 07:27:47.381254 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.381204 2557 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 16 07:27:47.381449 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.381288 2557 reconstruct.go:97] "Volume reconstruction finished" Sep 16 07:27:47.381449 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.381295 2557 reconciler.go:29] "Reconciler: start to sync state" Sep 16 07:27:47.381449 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.381423 2557 factory.go:153] Registering CRI-O factory Sep 16 07:27:47.381449 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.381438 2557 factory.go:223] Registration of the crio container factory successfully Sep 16 07:27:47.381616 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.381473 2557 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-ttrlv" Sep 16 07:27:47.381616 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.381499 2557 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 16 07:27:47.381616 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.381517 2557 factory.go:103] Registering Raw factory Sep 16 07:27:47.381616 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.381533 2557 manager.go:1196] Started watching for new ooms in manager Sep 16 07:27:47.381873 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.381859 2557 manager.go:349] Starting recovery of all containers Sep 16 07:27:47.382874 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:27:47.382850 2557 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-141-221.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 16 07:27:47.383002 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:27:47.382968 2557 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 16 07:27:47.393538 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.393517 2557 manager.go:354] Recovery completed Sep 16 07:27:47.396431 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:27:47.395958 2557 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Sep 16 07:27:47.399794 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.399782 2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 07:27:47.401491 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.401476 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-221.ec2.internal" event="NodeHasSufficientMemory" Sep 16 07:27:47.401580 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.401507 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-221.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 07:27:47.401580 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.401521 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-221.ec2.internal" event="NodeHasSufficientPID" Sep 16 07:27:47.402095 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.402081 2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 07:27:47.402095 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.402086 2557 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 16 07:27:47.402252 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.402098 2557 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 16 07:27:47.402252 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.402115 2557 state_mem.go:36] "Initialized new in-memory state store" Sep 16 07:27:47.402823 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.402806 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-221.ec2.internal" event="NodeHasSufficientMemory" Sep 16 07:27:47.402900 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.402838 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-221.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 07:27:47.402900 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.402852 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-221.ec2.internal" event="NodeHasSufficientPID" Sep 16 07:27:47.404092 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.404079 2557 policy_none.go:49] "None policy: Start" Sep 16 07:27:47.404136 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.404095 2557 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 16 07:27:47.404136 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.404104 2557 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 16 07:27:47.406037 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.406026 2557 policy_none.go:47] "Start" Sep 16 07:27:47.466553 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.442803 2557 manager.go:337] "Starting Device Plugin manager" Sep 16 07:27:47.466553 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:27:47.442846 2557 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 16 07:27:47.466553 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.442858 2557 server.go:87] "Starting device plugin registration server" Sep 16 07:27:47.466553 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.443141 2557 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 16 07:27:47.466553 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.443153 2557 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 16 07:27:47.466553 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.443262 2557 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 16 07:27:47.466553 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.443333 2557 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 16 07:27:47.466553 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.443342 2557 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 16 07:27:47.466553 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:27:47.443938 2557 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 16 07:27:47.466553 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:27:47.443987 2557 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-221.ec2.internal\" not found" Sep 16 07:27:47.517074 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.517047 2557 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 16 07:27:47.518150 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.518130 2557 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 16 07:27:47.518256 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.518179 2557 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 16 07:27:47.518256 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.518203 2557 kubelet.go:2448] "Starting kubelet main sync loop" Sep 16 07:27:47.518367 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:27:47.518264 2557 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 16 07:27:47.520614 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.520597 2557 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 16 07:27:47.543768 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.543717 2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 07:27:47.544494 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.544474 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-221.ec2.internal" event="NodeHasSufficientMemory" Sep 16 07:27:47.544567 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.544505 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-221.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 07:27:47.544567 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.544516 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-221.ec2.internal" event="NodeHasSufficientPID" Sep 16 07:27:47.544567 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.544551 2557 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-141-221.ec2.internal" Sep 16 07:27:47.550799 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.550785 2557 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-141-221.ec2.internal" Sep 16 07:27:47.550853 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:27:47.550804 2557 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-221.ec2.internal\": node \"ip-10-0-141-221.ec2.internal\" not found" Sep 16 07:27:47.579387 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:27:47.579367 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-221.ec2.internal\" not found" Sep 16 07:27:47.619221 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.619187 2557 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-221.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-221.ec2.internal"] Sep 16 07:27:47.619315 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.619280 2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 07:27:47.620487 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.620474 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-221.ec2.internal" event="NodeHasSufficientMemory" Sep 16 07:27:47.620553 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.620502 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-221.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 07:27:47.620553 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.620514 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-221.ec2.internal" event="NodeHasSufficientPID" Sep 16 07:27:47.622623 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.622612 2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 07:27:47.622718 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.622706 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-221.ec2.internal" Sep 16 07:27:47.622760 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.622736 2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 07:27:47.623252 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.623224 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-221.ec2.internal" event="NodeHasSufficientMemory" Sep 16 07:27:47.623331 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.623259 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-221.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 07:27:47.623331 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.623271 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-221.ec2.internal" event="NodeHasSufficientPID" Sep 16 07:27:47.623331 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.623299 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-221.ec2.internal" event="NodeHasSufficientMemory" Sep 16 07:27:47.623331 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.623318 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-221.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 07:27:47.623331 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.623329 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-221.ec2.internal" event="NodeHasSufficientPID" Sep 16 07:27:47.625372 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.625356 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-221.ec2.internal" Sep 16 07:27:47.625447 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.625384 2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 07:27:47.626032 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.626018 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-221.ec2.internal" event="NodeHasSufficientMemory" Sep 16 07:27:47.626105 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.626038 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-221.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 07:27:47.626105 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.626047 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-221.ec2.internal" event="NodeHasSufficientPID" Sep 16 07:27:47.647911 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:27:47.647886 2557 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-221.ec2.internal\" not found" node="ip-10-0-141-221.ec2.internal" Sep 16 07:27:47.651171 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:27:47.651153 2557 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-221.ec2.internal\" not found" node="ip-10-0-141-221.ec2.internal" Sep 16 07:27:47.680127 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:27:47.680107 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-221.ec2.internal\" not found" Sep 16 07:27:47.682921 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.682904 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8ac46b15b636b4c144bce2b4a1b3535a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-221.ec2.internal\" (UID: \"8ac46b15b636b4c144bce2b4a1b3535a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-221.ec2.internal" Sep 16 07:27:47.683007 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.682934 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8ac46b15b636b4c144bce2b4a1b3535a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-221.ec2.internal\" (UID: \"8ac46b15b636b4c144bce2b4a1b3535a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-221.ec2.internal" Sep 16 07:27:47.683007 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.682959 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3825bcd696c373ea98c5e3521a26b861-config\") pod \"kube-apiserver-proxy-ip-10-0-141-221.ec2.internal\" (UID: \"3825bcd696c373ea98c5e3521a26b861\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-221.ec2.internal" Sep 16 07:27:47.780951 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:27:47.780928 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-221.ec2.internal\" not found" Sep 16 07:27:47.783159 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.783143 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8ac46b15b636b4c144bce2b4a1b3535a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-221.ec2.internal\" (UID: \"8ac46b15b636b4c144bce2b4a1b3535a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-221.ec2.internal" Sep 16 07:27:47.783216 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.783171 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8ac46b15b636b4c144bce2b4a1b3535a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-221.ec2.internal\" (UID: \"8ac46b15b636b4c144bce2b4a1b3535a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-221.ec2.internal" Sep 16 07:27:47.783216 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.783188 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3825bcd696c373ea98c5e3521a26b861-config\") pod \"kube-apiserver-proxy-ip-10-0-141-221.ec2.internal\" (UID: \"3825bcd696c373ea98c5e3521a26b861\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-221.ec2.internal" Sep 16 07:27:47.783216 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.783211 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3825bcd696c373ea98c5e3521a26b861-config\") pod \"kube-apiserver-proxy-ip-10-0-141-221.ec2.internal\" (UID: \"3825bcd696c373ea98c5e3521a26b861\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-221.ec2.internal" Sep 16 07:27:47.783324 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.783226 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8ac46b15b636b4c144bce2b4a1b3535a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-221.ec2.internal\" (UID: \"8ac46b15b636b4c144bce2b4a1b3535a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-221.ec2.internal" Sep 16 07:27:47.783324 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.783250 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8ac46b15b636b4c144bce2b4a1b3535a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-221.ec2.internal\" (UID: \"8ac46b15b636b4c144bce2b4a1b3535a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-221.ec2.internal" Sep 16 07:27:47.881325 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:27:47.881233 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-221.ec2.internal\" not found" Sep 16 07:27:47.950611 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.950587 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-221.ec2.internal" Sep 16 07:27:47.953760 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:47.953738 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-221.ec2.internal" Sep 16 07:27:47.981411 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:27:47.981384 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-221.ec2.internal\" not found" Sep 16 07:27:48.082659 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:27:48.082605 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-221.ec2.internal\" not found" Sep 16 07:27:48.183750 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:27:48.183652 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-221.ec2.internal\" not found" Sep 16 07:27:48.266098 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:48.266063 2557 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 16 07:27:48.284429 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:27:48.284406 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-221.ec2.internal\" not found" Sep 16 07:27:48.314858 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:48.314840 2557 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 16 07:27:48.315229 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:48.314956 2557 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 16 07:27:48.315229 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:48.314993 2557 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 16 07:27:48.385129 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:27:48.385099 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-221.ec2.internal\" not found" Sep 16 07:27:48.387446 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:48.387415 2557 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-15 07:22:48 +0000 UTC" deadline="2028-04-18 22:55:56.874179819 +0000 UTC" Sep 16 07:27:48.387510 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:48.387447 2557 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13935h28m8.486736955s" Sep 16 07:27:48.399969 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:48.399950 2557 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 16 07:27:48.485675 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:27:48.485613 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-221.ec2.internal\" not found" Sep 16 07:27:48.586564 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:27:48.586532 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-221.ec2.internal\" not found" Sep 16 07:27:48.664637 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:48.664611 2557 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 16 07:27:48.681627 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:48.681604 2557 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-221.ec2.internal" Sep 16 07:27:48.693476 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:48.693457 2557 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 16 07:27:48.695947 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:48.695934 2557 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-221.ec2.internal" Sep 16 07:27:48.703482 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:48.703465 2557 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 16 07:27:48.930368 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:48.930333 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3825bcd696c373ea98c5e3521a26b861.slice/crio-1865e5f90c2f37a52aaaddd0b818ccf4751551c828815ba5f35bc31d71473c43 WatchSource:0}: Error finding container 1865e5f90c2f37a52aaaddd0b818ccf4751551c828815ba5f35bc31d71473c43: Status 404 returned error can't find the container with id 1865e5f90c2f37a52aaaddd0b818ccf4751551c828815ba5f35bc31d71473c43 Sep 16 07:27:48.940559 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:48.940540 2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 16 07:27:48.950416 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:48.950391 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ac46b15b636b4c144bce2b4a1b3535a.slice/crio-1357c27ac5cc88b64d9f21268ba261444be1c6f6d33fa4bcf3fef144b401dfed WatchSource:0}: Error finding container 1357c27ac5cc88b64d9f21268ba261444be1c6f6d33fa4bcf3fef144b401dfed: Status 404 returned error can't find the container with id 1357c27ac5cc88b64d9f21268ba261444be1c6f6d33fa4bcf3fef144b401dfed Sep 16 07:27:49.359589 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.359558 2557 apiserver.go:52] "Watching apiserver" Sep 16 07:27:49.365815 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.365784 2557 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 16 07:27:49.366182 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.366160 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-fhnn7","openshift-ovn-kubernetes/ovnkube-node-4ztgb","kube-system/global-pull-secret-syncer-t8thj","kube-system/konnectivity-agent-lq72c","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vj2rd","openshift-cluster-node-tuning-operator/tuned-zjxc4","openshift-dns/node-resolver-8gnk6","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-221.ec2.internal","openshift-multus/multus-additional-cni-plugins-qprhb","openshift-multus/network-metrics-daemon-kqc7r","kube-system/kube-apiserver-proxy-ip-10-0-141-221.ec2.internal","openshift-image-registry/node-ca-vbxvc","openshift-multus/multus-4cb95","openshift-network-diagnostics/network-check-target-jtqz9"] Sep 16 07:27:49.369096 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.369078 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-8gnk6" Sep 16 07:27:49.371285 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.371239 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4cb95" Sep 16 07:27:49.373036 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.373017 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 16 07:27:49.373036 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.373032 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-9kmfp\"" Sep 16 07:27:49.373176 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.373017 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 16 07:27:49.373593 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.373572 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t8thj" Sep 16 07:27:49.373688 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:27:49.373663 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t8thj" podUID="90e5e23a-060b-4612-80f5-f9a600b6d476" Sep 16 07:27:49.375067 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.375041 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 16 07:27:49.375797 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.375711 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 16 07:27:49.375797 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.375718 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 16 07:27:49.376227 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.375962 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-lq72c" Sep 16 07:27:49.376227 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.376083 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 16 07:27:49.376466 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.376443 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-jwkx8\"" Sep 16 07:27:49.378414 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.378392 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vj2rd" Sep 16 07:27:49.379804 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.379787 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-rtrh9\"" Sep 16 07:27:49.380750 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.380734 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zjxc4" Sep 16 07:27:49.380971 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.380908 2557 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 16 07:27:49.381572 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.381540 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 16 07:27:49.381572 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.381558 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 16 07:27:49.381735 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.381613 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 16 07:27:49.381735 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.381618 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 16 07:27:49.381735 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.381559 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 16 07:27:49.381871 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.381862 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-54qvk\"" Sep 16 07:27:49.383140 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.383124 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-qprhb" Sep 16 07:27:49.383445 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.383302 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 16 07:27:49.383445 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.383418 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-knq6g\"" Sep 16 07:27:49.383567 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.383557 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 16 07:27:49.385515 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.385496 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-fhnn7" Sep 16 07:27:49.385898 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.385879 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 16 07:27:49.385968 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.385946 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 16 07:27:49.386423 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.386407 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-8kmcf\"" Sep 16 07:27:49.387757 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.387736 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 16 07:27:49.387757 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.387753 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 16 07:27:49.387948 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.387849 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 16 07:27:49.387948 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.387869 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-2jwjn\"" Sep 16 07:27:49.388175 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.388161 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-4ztgb" Sep 16 07:27:49.392200 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.392175 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1ae92448-74b5-4b28-b337-758f6884d09a-konnectivity-ca\") pod \"konnectivity-agent-lq72c\" (UID: \"1ae92448-74b5-4b28-b337-758f6884d09a\") " pod="kube-system/konnectivity-agent-lq72c" Sep 16 07:27:49.392318 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.392212 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4fdx\" (UniqueName: \"kubernetes.io/projected/17757d39-4eae-4f8d-8c3a-679d3502d985-kube-api-access-t4fdx\") pod \"aws-ebs-csi-driver-node-vj2rd\" (UID: \"17757d39-4eae-4f8d-8c3a-679d3502d985\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vj2rd" Sep 16 07:27:49.392318 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.392236 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/280250dc-6c09-4766-822c-b8bdf97e4a49-host-run-netns\") pod \"multus-4cb95\" (UID: \"280250dc-6c09-4766-822c-b8bdf97e4a49\") " pod="openshift-multus/multus-4cb95" Sep 16 07:27:49.392450 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.392315 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb688a0e-2bd2-4df2-b475-7eba4237e526-etc-kubernetes\") pod \"tuned-zjxc4\" (UID: \"eb688a0e-2bd2-4df2-b475-7eba4237e526\") " pod="openshift-cluster-node-tuning-operator/tuned-zjxc4" Sep 16 07:27:49.392450 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.392383 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/eb688a0e-2bd2-4df2-b475-7eba4237e526-etc-systemd\") pod \"tuned-zjxc4\" (UID: \"eb688a0e-2bd2-4df2-b475-7eba4237e526\") " pod="openshift-cluster-node-tuning-operator/tuned-zjxc4" Sep 16 07:27:49.392450 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.392433 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/eb688a0e-2bd2-4df2-b475-7eba4237e526-run\") pod \"tuned-zjxc4\" (UID: \"eb688a0e-2bd2-4df2-b475-7eba4237e526\") " pod="openshift-cluster-node-tuning-operator/tuned-zjxc4" Sep 16 07:27:49.392568 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.392464 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/280250dc-6c09-4766-822c-b8bdf97e4a49-multus-daemon-config\") pod \"multus-4cb95\" (UID: \"280250dc-6c09-4766-822c-b8bdf97e4a49\") " pod="openshift-multus/multus-4cb95" Sep 16 07:27:49.392568 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.392490 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5f7afb78-2955-49e0-adec-6e8aae7ae51a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qprhb\" (UID: \"5f7afb78-2955-49e0-adec-6e8aae7ae51a\") " pod="openshift-multus/multus-additional-cni-plugins-qprhb" Sep 16 07:27:49.392568 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.392534 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mf5v4\" (UniqueName: \"kubernetes.io/projected/5f7afb78-2955-49e0-adec-6e8aae7ae51a-kube-api-access-mf5v4\") pod \"multus-additional-cni-plugins-qprhb\" (UID: \"5f7afb78-2955-49e0-adec-6e8aae7ae51a\") " pod="openshift-multus/multus-additional-cni-plugins-qprhb" Sep 16 07:27:49.392728 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.392574 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/17757d39-4eae-4f8d-8c3a-679d3502d985-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vj2rd\" (UID: \"17757d39-4eae-4f8d-8c3a-679d3502d985\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vj2rd" Sep 16 07:27:49.392728 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.392620 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/eb688a0e-2bd2-4df2-b475-7eba4237e526-host\") pod \"tuned-zjxc4\" (UID: \"eb688a0e-2bd2-4df2-b475-7eba4237e526\") " pod="openshift-cluster-node-tuning-operator/tuned-zjxc4" Sep 16 07:27:49.392728 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.392654 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/280250dc-6c09-4766-822c-b8bdf97e4a49-host-var-lib-cni-multus\") pod \"multus-4cb95\" (UID: \"280250dc-6c09-4766-822c-b8bdf97e4a49\") " pod="openshift-multus/multus-4cb95" Sep 16 07:27:49.392728 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.392685 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/280250dc-6c09-4766-822c-b8bdf97e4a49-multus-conf-dir\") pod \"multus-4cb95\" (UID: \"280250dc-6c09-4766-822c-b8bdf97e4a49\") " pod="openshift-multus/multus-4cb95" Sep 16 07:27:49.392728 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.392710 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5f7afb78-2955-49e0-adec-6e8aae7ae51a-cnibin\") pod \"multus-additional-cni-plugins-qprhb\" (UID: \"5f7afb78-2955-49e0-adec-6e8aae7ae51a\") " pod="openshift-multus/multus-additional-cni-plugins-qprhb" Sep 16 07:27:49.392986 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.392736 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/17757d39-4eae-4f8d-8c3a-679d3502d985-device-dir\") pod \"aws-ebs-csi-driver-node-vj2rd\" (UID: \"17757d39-4eae-4f8d-8c3a-679d3502d985\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vj2rd" Sep 16 07:27:49.392986 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.392771 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/17757d39-4eae-4f8d-8c3a-679d3502d985-sys-fs\") pod \"aws-ebs-csi-driver-node-vj2rd\" (UID: \"17757d39-4eae-4f8d-8c3a-679d3502d985\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vj2rd" Sep 16 07:27:49.392986 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.392806 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eb688a0e-2bd2-4df2-b475-7eba4237e526-sys\") pod \"tuned-zjxc4\" (UID: \"eb688a0e-2bd2-4df2-b475-7eba4237e526\") " pod="openshift-cluster-node-tuning-operator/tuned-zjxc4" Sep 16 07:27:49.392986 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.392832 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/eb688a0e-2bd2-4df2-b475-7eba4237e526-lib-modules\") pod \"tuned-zjxc4\" (UID: \"eb688a0e-2bd2-4df2-b475-7eba4237e526\") " pod="openshift-cluster-node-tuning-operator/tuned-zjxc4" Sep 16 07:27:49.392986 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.392898 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eb688a0e-2bd2-4df2-b475-7eba4237e526-tmp\") pod \"tuned-zjxc4\" (UID: \"eb688a0e-2bd2-4df2-b475-7eba4237e526\") " pod="openshift-cluster-node-tuning-operator/tuned-zjxc4" Sep 16 07:27:49.392986 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.392945 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5f7afb78-2955-49e0-adec-6e8aae7ae51a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qprhb\" (UID: \"5f7afb78-2955-49e0-adec-6e8aae7ae51a\") " pod="openshift-multus/multus-additional-cni-plugins-qprhb" Sep 16 07:27:49.392986 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.392979 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/90e5e23a-060b-4612-80f5-f9a600b6d476-original-pull-secret\") pod \"global-pull-secret-syncer-t8thj\" (UID: \"90e5e23a-060b-4612-80f5-f9a600b6d476\") " pod="kube-system/global-pull-secret-syncer-t8thj" Sep 16 07:27:49.393351 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.392984 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kqc7r" Sep 16 07:27:49.393351 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.393013 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/280250dc-6c09-4766-822c-b8bdf97e4a49-multus-cni-dir\") pod \"multus-4cb95\" (UID: \"280250dc-6c09-4766-822c-b8bdf97e4a49\") " pod="openshift-multus/multus-4cb95" Sep 16 07:27:49.393351 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.393057 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/280250dc-6c09-4766-822c-b8bdf97e4a49-cni-binary-copy\") pod \"multus-4cb95\" (UID: \"280250dc-6c09-4766-822c-b8bdf97e4a49\") " pod="openshift-multus/multus-4cb95" Sep 16 07:27:49.393351 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:27:49.393058 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kqc7r" podUID="296c5dcf-403c-4a1f-9d6f-05ffb7d75e38" Sep 16 07:27:49.393351 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.393082 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/280250dc-6c09-4766-822c-b8bdf97e4a49-hostroot\") pod \"multus-4cb95\" (UID: \"280250dc-6c09-4766-822c-b8bdf97e4a49\") " pod="openshift-multus/multus-4cb95" Sep 16 07:27:49.393351 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.393122 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5f7afb78-2955-49e0-adec-6e8aae7ae51a-system-cni-dir\") pod \"multus-additional-cni-plugins-qprhb\" (UID: \"5f7afb78-2955-49e0-adec-6e8aae7ae51a\") " pod="openshift-multus/multus-additional-cni-plugins-qprhb" Sep 16 07:27:49.393351 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.393157 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5f7afb78-2955-49e0-adec-6e8aae7ae51a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qprhb\" (UID: \"5f7afb78-2955-49e0-adec-6e8aae7ae51a\") " pod="openshift-multus/multus-additional-cni-plugins-qprhb" Sep 16 07:27:49.393351 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.393184 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/280250dc-6c09-4766-822c-b8bdf97e4a49-multus-socket-dir-parent\") pod \"multus-4cb95\" (UID: \"280250dc-6c09-4766-822c-b8bdf97e4a49\") " pod="openshift-multus/multus-4cb95" Sep 16 07:27:49.393351 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.393222 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/280250dc-6c09-4766-822c-b8bdf97e4a49-host-var-lib-cni-bin\") pod \"multus-4cb95\" (UID: \"280250dc-6c09-4766-822c-b8bdf97e4a49\") " pod="openshift-multus/multus-4cb95" Sep 16 07:27:49.393351 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.393278 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/eb688a0e-2bd2-4df2-b475-7eba4237e526-etc-sysconfig\") pod \"tuned-zjxc4\" (UID: \"eb688a0e-2bd2-4df2-b475-7eba4237e526\") " pod="openshift-cluster-node-tuning-operator/tuned-zjxc4" Sep 16 07:27:49.393351 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.393308 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/eb688a0e-2bd2-4df2-b475-7eba4237e526-etc-sysctl-d\") pod \"tuned-zjxc4\" (UID: \"eb688a0e-2bd2-4df2-b475-7eba4237e526\") " pod="openshift-cluster-node-tuning-operator/tuned-zjxc4" Sep 16 07:27:49.393351 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.393333 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20f6d954-1104-49b9-ab10-32e0de4ef8e0-tmp-dir\") pod \"node-resolver-8gnk6\" (UID: \"20f6d954-1104-49b9-ab10-32e0de4ef8e0\") " pod="openshift-dns/node-resolver-8gnk6" Sep 16 07:27:49.393880 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.393371 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/280250dc-6c09-4766-822c-b8bdf97e4a49-host-var-lib-kubelet\") pod \"multus-4cb95\" (UID: \"280250dc-6c09-4766-822c-b8bdf97e4a49\") " pod="openshift-multus/multus-4cb95" Sep 16 07:27:49.393880 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.393399 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/eb688a0e-2bd2-4df2-b475-7eba4237e526-etc-sysctl-conf\") pod \"tuned-zjxc4\" (UID: \"eb688a0e-2bd2-4df2-b475-7eba4237e526\") " pod="openshift-cluster-node-tuning-operator/tuned-zjxc4" Sep 16 07:27:49.393880 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.393423 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ptk2\" (UniqueName: \"kubernetes.io/projected/20f6d954-1104-49b9-ab10-32e0de4ef8e0-kube-api-access-5ptk2\") pod \"node-resolver-8gnk6\" (UID: \"20f6d954-1104-49b9-ab10-32e0de4ef8e0\") " pod="openshift-dns/node-resolver-8gnk6" Sep 16 07:27:49.393880 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.393442 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/280250dc-6c09-4766-822c-b8bdf97e4a49-system-cni-dir\") pod \"multus-4cb95\" (UID: \"280250dc-6c09-4766-822c-b8bdf97e4a49\") " pod="openshift-multus/multus-4cb95" Sep 16 07:27:49.393880 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.393456 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/280250dc-6c09-4766-822c-b8bdf97e4a49-host-run-k8s-cni-cncf-io\") pod \"multus-4cb95\" (UID: \"280250dc-6c09-4766-822c-b8bdf97e4a49\") " pod="openshift-multus/multus-4cb95" Sep 16 07:27:49.393880 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.393478 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/280250dc-6c09-4766-822c-b8bdf97e4a49-host-run-multus-certs\") pod \"multus-4cb95\" (UID: \"280250dc-6c09-4766-822c-b8bdf97e4a49\") " pod="openshift-multus/multus-4cb95" Sep 16 07:27:49.393880 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.393501 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/90e5e23a-060b-4612-80f5-f9a600b6d476-dbus\") pod \"global-pull-secret-syncer-t8thj\" (UID: \"90e5e23a-060b-4612-80f5-f9a600b6d476\") " pod="kube-system/global-pull-secret-syncer-t8thj" Sep 16 07:27:49.393880 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.393525 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/17757d39-4eae-4f8d-8c3a-679d3502d985-socket-dir\") pod \"aws-ebs-csi-driver-node-vj2rd\" (UID: \"17757d39-4eae-4f8d-8c3a-679d3502d985\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vj2rd" Sep 16 07:27:49.393880 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.393543 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/17757d39-4eae-4f8d-8c3a-679d3502d985-registration-dir\") pod \"aws-ebs-csi-driver-node-vj2rd\" (UID: \"17757d39-4eae-4f8d-8c3a-679d3502d985\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vj2rd" Sep 16 07:27:49.393880 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.393559 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2bcn\" (UniqueName: \"kubernetes.io/projected/eb688a0e-2bd2-4df2-b475-7eba4237e526-kube-api-access-d2bcn\") pod \"tuned-zjxc4\" (UID: \"eb688a0e-2bd2-4df2-b475-7eba4237e526\") " pod="openshift-cluster-node-tuning-operator/tuned-zjxc4" Sep 16 07:27:49.393880 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.393581 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/20f6d954-1104-49b9-ab10-32e0de4ef8e0-hosts-file\") pod \"node-resolver-8gnk6\" (UID: \"20f6d954-1104-49b9-ab10-32e0de4ef8e0\") " pod="openshift-dns/node-resolver-8gnk6" Sep 16 07:27:49.393880 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.393606 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1ae92448-74b5-4b28-b337-758f6884d09a-agent-certs\") pod \"konnectivity-agent-lq72c\" (UID: \"1ae92448-74b5-4b28-b337-758f6884d09a\") " pod="kube-system/konnectivity-agent-lq72c" Sep 16 07:27:49.393880 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.393630 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5f7afb78-2955-49e0-adec-6e8aae7ae51a-cni-binary-copy\") pod \"multus-additional-cni-plugins-qprhb\" (UID: \"5f7afb78-2955-49e0-adec-6e8aae7ae51a\") " pod="openshift-multus/multus-additional-cni-plugins-qprhb" Sep 16 07:27:49.393880 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.393655 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/90e5e23a-060b-4612-80f5-f9a600b6d476-kubelet-config\") pod \"global-pull-secret-syncer-t8thj\" (UID: \"90e5e23a-060b-4612-80f5-f9a600b6d476\") " pod="kube-system/global-pull-secret-syncer-t8thj" Sep 16 07:27:49.393880 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.393684 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/17757d39-4eae-4f8d-8c3a-679d3502d985-etc-selinux\") pod \"aws-ebs-csi-driver-node-vj2rd\" (UID: \"17757d39-4eae-4f8d-8c3a-679d3502d985\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vj2rd" Sep 16 07:27:49.393880 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.393704 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/eb688a0e-2bd2-4df2-b475-7eba4237e526-etc-modprobe-d\") pod \"tuned-zjxc4\" (UID: \"eb688a0e-2bd2-4df2-b475-7eba4237e526\") " pod="openshift-cluster-node-tuning-operator/tuned-zjxc4" Sep 16 07:27:49.394455 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.393750 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eb688a0e-2bd2-4df2-b475-7eba4237e526-var-lib-kubelet\") pod \"tuned-zjxc4\" (UID: \"eb688a0e-2bd2-4df2-b475-7eba4237e526\") " pod="openshift-cluster-node-tuning-operator/tuned-zjxc4" Sep 16 07:27:49.394455 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.393787 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/eb688a0e-2bd2-4df2-b475-7eba4237e526-etc-tuned\") pod \"tuned-zjxc4\" (UID: \"eb688a0e-2bd2-4df2-b475-7eba4237e526\") " pod="openshift-cluster-node-tuning-operator/tuned-zjxc4" Sep 16 07:27:49.394455 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.393810 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/280250dc-6c09-4766-822c-b8bdf97e4a49-cnibin\") pod \"multus-4cb95\" (UID: \"280250dc-6c09-4766-822c-b8bdf97e4a49\") " pod="openshift-multus/multus-4cb95" Sep 16 07:27:49.394455 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.393832 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/280250dc-6c09-4766-822c-b8bdf97e4a49-etc-kubernetes\") pod \"multus-4cb95\" (UID: \"280250dc-6c09-4766-822c-b8bdf97e4a49\") " pod="openshift-multus/multus-4cb95" Sep 16 07:27:49.394455 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.393888 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pcsb\" (UniqueName: \"kubernetes.io/projected/280250dc-6c09-4766-822c-b8bdf97e4a49-kube-api-access-7pcsb\") pod \"multus-4cb95\" (UID: \"280250dc-6c09-4766-822c-b8bdf97e4a49\") " pod="openshift-multus/multus-4cb95" Sep 16 07:27:49.394974 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.394952 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 16 07:27:49.395172 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.395151 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 16 07:27:49.395523 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.395501 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-ghxhs\"" Sep 16 07:27:49.395927 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.395904 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jtqz9" Sep 16 07:27:49.396050 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:27:49.396001 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jtqz9" podUID="bfde2ba8-6a5f-413f-9ce9-d9ce0af2c758" Sep 16 07:27:49.396357 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.396338 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 16 07:27:49.396434 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.396344 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 16 07:27:49.396869 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.396641 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 16 07:27:49.396869 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.396658 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 16 07:27:49.398407 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.398387 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vbxvc" Sep 16 07:27:49.400605 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.400584 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 16 07:27:49.400717 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.400652 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-9qd8s\"" Sep 16 07:27:49.400832 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.400808 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 16 07:27:49.401014 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.400995 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 16 07:27:49.401786 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.401756 2557 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 16 07:27:49.429682 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.429640 2557 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-fccbl" Sep 16 07:27:49.440406 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.440341 2557 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-fccbl" Sep 16 07:27:49.482365 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.482339 2557 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 16 07:27:49.494738 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.494711 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2bcn\" (UniqueName: \"kubernetes.io/projected/eb688a0e-2bd2-4df2-b475-7eba4237e526-kube-api-access-d2bcn\") pod \"tuned-zjxc4\" (UID: \"eb688a0e-2bd2-4df2-b475-7eba4237e526\") " pod="openshift-cluster-node-tuning-operator/tuned-zjxc4" Sep 16 07:27:49.494847 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.494756 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/20f6d954-1104-49b9-ab10-32e0de4ef8e0-hosts-file\") pod \"node-resolver-8gnk6\" (UID: \"20f6d954-1104-49b9-ab10-32e0de4ef8e0\") " pod="openshift-dns/node-resolver-8gnk6" Sep 16 07:27:49.494847 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.494790 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/89f6995b-d178-4fe9-9593-ba8dc8f49351-serviceca\") pod \"node-ca-vbxvc\" (UID: \"89f6995b-d178-4fe9-9593-ba8dc8f49351\") " pod="openshift-image-registry/node-ca-vbxvc" Sep 16 07:27:49.494847 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.494819 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eb688a0e-2bd2-4df2-b475-7eba4237e526-var-lib-kubelet\") pod \"tuned-zjxc4\" (UID: \"eb688a0e-2bd2-4df2-b475-7eba4237e526\") " pod="openshift-cluster-node-tuning-operator/tuned-zjxc4" Sep 16 07:27:49.494847 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.494845 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/eb688a0e-2bd2-4df2-b475-7eba4237e526-etc-tuned\") pod \"tuned-zjxc4\" (UID: \"eb688a0e-2bd2-4df2-b475-7eba4237e526\") " pod="openshift-cluster-node-tuning-operator/tuned-zjxc4" Sep 16 07:27:49.495086 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.494864 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/20f6d954-1104-49b9-ab10-32e0de4ef8e0-hosts-file\") pod \"node-resolver-8gnk6\" (UID: \"20f6d954-1104-49b9-ab10-32e0de4ef8e0\") " pod="openshift-dns/node-resolver-8gnk6" Sep 16 07:27:49.495086 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.494873 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/280250dc-6c09-4766-822c-b8bdf97e4a49-cnibin\") pod \"multus-4cb95\" (UID: \"280250dc-6c09-4766-822c-b8bdf97e4a49\") " pod="openshift-multus/multus-4cb95" Sep 16 07:27:49.495086 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.494898 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/280250dc-6c09-4766-822c-b8bdf97e4a49-etc-kubernetes\") pod \"multus-4cb95\" (UID: \"280250dc-6c09-4766-822c-b8bdf97e4a49\") " pod="openshift-multus/multus-4cb95" Sep 16 07:27:49.495086 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.494926 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8d5c17cc-f191-4354-9c73-ab7f25dd203b-ovn-node-metrics-cert\") pod \"ovnkube-node-4ztgb\" (UID: \"8d5c17cc-f191-4354-9c73-ab7f25dd203b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ztgb" Sep 16 07:27:49.495086 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.494932 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eb688a0e-2bd2-4df2-b475-7eba4237e526-var-lib-kubelet\") pod \"tuned-zjxc4\" (UID: \"eb688a0e-2bd2-4df2-b475-7eba4237e526\") " pod="openshift-cluster-node-tuning-operator/tuned-zjxc4" Sep 16 07:27:49.495086 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.494956 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fznzt\" (UniqueName: \"kubernetes.io/projected/89f6995b-d178-4fe9-9593-ba8dc8f49351-kube-api-access-fznzt\") pod \"node-ca-vbxvc\" (UID: \"89f6995b-d178-4fe9-9593-ba8dc8f49351\") " pod="openshift-image-registry/node-ca-vbxvc" Sep 16 07:27:49.495086 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.494982 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/eb688a0e-2bd2-4df2-b475-7eba4237e526-etc-systemd\") pod \"tuned-zjxc4\" (UID: \"eb688a0e-2bd2-4df2-b475-7eba4237e526\") " pod="openshift-cluster-node-tuning-operator/tuned-zjxc4" Sep 16 07:27:49.495086 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.494987 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/280250dc-6c09-4766-822c-b8bdf97e4a49-etc-kubernetes\") pod \"multus-4cb95\" (UID: \"280250dc-6c09-4766-822c-b8bdf97e4a49\") " pod="openshift-multus/multus-4cb95" Sep 16 07:27:49.495086 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.495009 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/280250dc-6c09-4766-822c-b8bdf97e4a49-multus-daemon-config\") pod \"multus-4cb95\" (UID: \"280250dc-6c09-4766-822c-b8bdf97e4a49\") " pod="openshift-multus/multus-4cb95" Sep 16 07:27:49.495086 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.495045 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5f7afb78-2955-49e0-adec-6e8aae7ae51a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qprhb\" (UID: \"5f7afb78-2955-49e0-adec-6e8aae7ae51a\") " pod="openshift-multus/multus-additional-cni-plugins-qprhb" Sep 16 07:27:49.495723 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.495093 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/280250dc-6c09-4766-822c-b8bdf97e4a49-cnibin\") pod \"multus-4cb95\" (UID: \"280250dc-6c09-4766-822c-b8bdf97e4a49\") " pod="openshift-multus/multus-4cb95" Sep 16 07:27:49.495723 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.495156 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mf5v4\" (UniqueName: \"kubernetes.io/projected/5f7afb78-2955-49e0-adec-6e8aae7ae51a-kube-api-access-mf5v4\") pod \"multus-additional-cni-plugins-qprhb\" (UID: \"5f7afb78-2955-49e0-adec-6e8aae7ae51a\") " pod="openshift-multus/multus-additional-cni-plugins-qprhb" Sep 16 07:27:49.495723 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.495189 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/17757d39-4eae-4f8d-8c3a-679d3502d985-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vj2rd\" (UID: \"17757d39-4eae-4f8d-8c3a-679d3502d985\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vj2rd" Sep 16 07:27:49.495723 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.495198 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/eb688a0e-2bd2-4df2-b475-7eba4237e526-etc-systemd\") pod \"tuned-zjxc4\" (UID: \"eb688a0e-2bd2-4df2-b475-7eba4237e526\") " pod="openshift-cluster-node-tuning-operator/tuned-zjxc4" Sep 16 07:27:49.495723 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.495215 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/17757d39-4eae-4f8d-8c3a-679d3502d985-sys-fs\") pod \"aws-ebs-csi-driver-node-vj2rd\" (UID: \"17757d39-4eae-4f8d-8c3a-679d3502d985\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vj2rd" Sep 16 07:27:49.495723 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.495245 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8d5c17cc-f191-4354-9c73-ab7f25dd203b-host-cni-netd\") pod \"ovnkube-node-4ztgb\" (UID: \"8d5c17cc-f191-4354-9c73-ab7f25dd203b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ztgb" Sep 16 07:27:49.495723 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.495286 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/280250dc-6c09-4766-822c-b8bdf97e4a49-multus-conf-dir\") pod \"multus-4cb95\" (UID: \"280250dc-6c09-4766-822c-b8bdf97e4a49\") " pod="openshift-multus/multus-4cb95" Sep 16 07:27:49.495723 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.495312 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5f7afb78-2955-49e0-adec-6e8aae7ae51a-cnibin\") pod \"multus-additional-cni-plugins-qprhb\" (UID: \"5f7afb78-2955-49e0-adec-6e8aae7ae51a\") " pod="openshift-multus/multus-additional-cni-plugins-qprhb" Sep 16 07:27:49.495723 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.495338 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pqxg\" (UniqueName: \"kubernetes.io/projected/8d5c17cc-f191-4354-9c73-ab7f25dd203b-kube-api-access-2pqxg\") pod \"ovnkube-node-4ztgb\" (UID: \"8d5c17cc-f191-4354-9c73-ab7f25dd203b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ztgb" Sep 16 07:27:49.495723 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.495363 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/eb688a0e-2bd2-4df2-b475-7eba4237e526-lib-modules\") pod \"tuned-zjxc4\" (UID: \"eb688a0e-2bd2-4df2-b475-7eba4237e526\") " pod="openshift-cluster-node-tuning-operator/tuned-zjxc4" Sep 16 07:27:49.495723 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.495392 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/90e5e23a-060b-4612-80f5-f9a600b6d476-original-pull-secret\") pod \"global-pull-secret-syncer-t8thj\" (UID: \"90e5e23a-060b-4612-80f5-f9a600b6d476\") " pod="kube-system/global-pull-secret-syncer-t8thj" Sep 16 07:27:49.495723 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.495416 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8d5c17cc-f191-4354-9c73-ab7f25dd203b-run-ovn\") pod \"ovnkube-node-4ztgb\" (UID: \"8d5c17cc-f191-4354-9c73-ab7f25dd203b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ztgb" Sep 16 07:27:49.495723 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.495441 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/280250dc-6c09-4766-822c-b8bdf97e4a49-multus-cni-dir\") pod \"multus-4cb95\" (UID: \"280250dc-6c09-4766-822c-b8bdf97e4a49\") " pod="openshift-multus/multus-4cb95" Sep 16 07:27:49.495723 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.495473 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/280250dc-6c09-4766-822c-b8bdf97e4a49-cni-binary-copy\") pod \"multus-4cb95\" (UID: \"280250dc-6c09-4766-822c-b8bdf97e4a49\") " pod="openshift-multus/multus-4cb95" Sep 16 07:27:49.495723 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.495516 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8d5c17cc-f191-4354-9c73-ab7f25dd203b-node-log\") pod \"ovnkube-node-4ztgb\" (UID: \"8d5c17cc-f191-4354-9c73-ab7f25dd203b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ztgb" Sep 16 07:27:49.495723 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.495541 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8d5c17cc-f191-4354-9c73-ab7f25dd203b-host-cni-bin\") pod \"ovnkube-node-4ztgb\" (UID: \"8d5c17cc-f191-4354-9c73-ab7f25dd203b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ztgb" Sep 16 07:27:49.495723 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.495567 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/17757d39-4eae-4f8d-8c3a-679d3502d985-socket-dir\") pod \"aws-ebs-csi-driver-node-vj2rd\" (UID: \"17757d39-4eae-4f8d-8c3a-679d3502d985\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vj2rd" Sep 16 07:27:49.496454 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.495594 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4fdx\" (UniqueName: \"kubernetes.io/projected/17757d39-4eae-4f8d-8c3a-679d3502d985-kube-api-access-t4fdx\") pod \"aws-ebs-csi-driver-node-vj2rd\" (UID: \"17757d39-4eae-4f8d-8c3a-679d3502d985\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vj2rd" Sep 16 07:27:49.496454 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.495627 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/280250dc-6c09-4766-822c-b8bdf97e4a49-multus-socket-dir-parent\") pod \"multus-4cb95\" (UID: \"280250dc-6c09-4766-822c-b8bdf97e4a49\") " pod="openshift-multus/multus-4cb95" Sep 16 07:27:49.496454 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.495646 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/eb688a0e-2bd2-4df2-b475-7eba4237e526-etc-sysconfig\") pod \"tuned-zjxc4\" (UID: \"eb688a0e-2bd2-4df2-b475-7eba4237e526\") " pod="openshift-cluster-node-tuning-operator/tuned-zjxc4" Sep 16 07:27:49.496454 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.495663 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/280250dc-6c09-4766-822c-b8bdf97e4a49-multus-daemon-config\") pod \"multus-4cb95\" (UID: \"280250dc-6c09-4766-822c-b8bdf97e4a49\") " pod="openshift-multus/multus-4cb95" Sep 16 07:27:49.496454 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.495676 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/eb688a0e-2bd2-4df2-b475-7eba4237e526-etc-sysctl-d\") pod \"tuned-zjxc4\" (UID: \"eb688a0e-2bd2-4df2-b475-7eba4237e526\") " pod="openshift-cluster-node-tuning-operator/tuned-zjxc4" Sep 16 07:27:49.496454 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.495702 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20f6d954-1104-49b9-ab10-32e0de4ef8e0-tmp-dir\") pod \"node-resolver-8gnk6\" (UID: \"20f6d954-1104-49b9-ab10-32e0de4ef8e0\") " pod="openshift-dns/node-resolver-8gnk6" Sep 16 07:27:49.496454 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.495729 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/280250dc-6c09-4766-822c-b8bdf97e4a49-host-var-lib-kubelet\") pod \"multus-4cb95\" (UID: \"280250dc-6c09-4766-822c-b8bdf97e4a49\") " pod="openshift-multus/multus-4cb95" Sep 16 07:27:49.496454 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:27:49.495755 2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 07:27:49.496454 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.495760 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/eb688a0e-2bd2-4df2-b475-7eba4237e526-etc-sysctl-conf\") pod \"tuned-zjxc4\" (UID: \"eb688a0e-2bd2-4df2-b475-7eba4237e526\") " pod="openshift-cluster-node-tuning-operator/tuned-zjxc4" Sep 16 07:27:49.496454 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.495787 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ptk2\" (UniqueName: \"kubernetes.io/projected/20f6d954-1104-49b9-ab10-32e0de4ef8e0-kube-api-access-5ptk2\") pod \"node-resolver-8gnk6\" (UID: \"20f6d954-1104-49b9-ab10-32e0de4ef8e0\") " pod="openshift-dns/node-resolver-8gnk6" Sep 16 07:27:49.496454 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:27:49.495801 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90e5e23a-060b-4612-80f5-f9a600b6d476-original-pull-secret podName:90e5e23a-060b-4612-80f5-f9a600b6d476 nodeName:}" failed. No retries permitted until 2026-09-16 07:27:49.99578674 +0000 UTC m=+3.022687731 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/90e5e23a-060b-4612-80f5-f9a600b6d476-original-pull-secret") pod "global-pull-secret-syncer-t8thj" (UID: "90e5e23a-060b-4612-80f5-f9a600b6d476") : object "kube-system"/"original-pull-secret" not registered Sep 16 07:27:49.496454 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.495806 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/280250dc-6c09-4766-822c-b8bdf97e4a49-multus-cni-dir\") pod \"multus-4cb95\" (UID: \"280250dc-6c09-4766-822c-b8bdf97e4a49\") " pod="openshift-multus/multus-4cb95" Sep 16 07:27:49.496454 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.495824 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/280250dc-6c09-4766-822c-b8bdf97e4a49-system-cni-dir\") pod \"multus-4cb95\" (UID: \"280250dc-6c09-4766-822c-b8bdf97e4a49\") " pod="openshift-multus/multus-4cb95" Sep 16 07:27:49.496454 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.495843 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/280250dc-6c09-4766-822c-b8bdf97e4a49-host-run-k8s-cni-cncf-io\") pod \"multus-4cb95\" (UID: \"280250dc-6c09-4766-822c-b8bdf97e4a49\") " pod="openshift-multus/multus-4cb95" Sep 16 07:27:49.496454 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.495864 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/296c5dcf-403c-4a1f-9d6f-05ffb7d75e38-metrics-certs\") pod \"network-metrics-daemon-kqc7r\" (UID: \"296c5dcf-403c-4a1f-9d6f-05ffb7d75e38\") " pod="openshift-multus/network-metrics-daemon-kqc7r" Sep 16 07:27:49.496454 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.495881 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1ae92448-74b5-4b28-b337-758f6884d09a-agent-certs\") pod \"konnectivity-agent-lq72c\" (UID: \"1ae92448-74b5-4b28-b337-758f6884d09a\") " pod="kube-system/konnectivity-agent-lq72c" Sep 16 07:27:49.496454 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.495899 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5f7afb78-2955-49e0-adec-6e8aae7ae51a-cni-binary-copy\") pod \"multus-additional-cni-plugins-qprhb\" (UID: \"5f7afb78-2955-49e0-adec-6e8aae7ae51a\") " pod="openshift-multus/multus-additional-cni-plugins-qprhb" Sep 16 07:27:49.497242 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.495916 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/90e5e23a-060b-4612-80f5-f9a600b6d476-kubelet-config\") pod \"global-pull-secret-syncer-t8thj\" (UID: \"90e5e23a-060b-4612-80f5-f9a600b6d476\") " pod="kube-system/global-pull-secret-syncer-t8thj" Sep 16 07:27:49.497242 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.495932 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/eb688a0e-2bd2-4df2-b475-7eba4237e526-etc-modprobe-d\") pod \"tuned-zjxc4\" (UID: \"eb688a0e-2bd2-4df2-b475-7eba4237e526\") " pod="openshift-cluster-node-tuning-operator/tuned-zjxc4" Sep 16 07:27:49.497242 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.495948 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pcsb\" (UniqueName: \"kubernetes.io/projected/280250dc-6c09-4766-822c-b8bdf97e4a49-kube-api-access-7pcsb\") pod \"multus-4cb95\" (UID: \"280250dc-6c09-4766-822c-b8bdf97e4a49\") " pod="openshift-multus/multus-4cb95" Sep 16 07:27:49.497242 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.495965 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1ae92448-74b5-4b28-b337-758f6884d09a-konnectivity-ca\") pod \"konnectivity-agent-lq72c\" (UID: \"1ae92448-74b5-4b28-b337-758f6884d09a\") " pod="kube-system/konnectivity-agent-lq72c" Sep 16 07:27:49.497242 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.495981 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/53ab9fb8-54fd-478d-9b2b-f69c8aac0d02-iptables-alerter-script\") pod \"iptables-alerter-fhnn7\" (UID: \"53ab9fb8-54fd-478d-9b2b-f69c8aac0d02\") " pod="openshift-network-operator/iptables-alerter-fhnn7" Sep 16 07:27:49.497242 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.495997 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8d5c17cc-f191-4354-9c73-ab7f25dd203b-host-kubelet\") pod \"ovnkube-node-4ztgb\" (UID: \"8d5c17cc-f191-4354-9c73-ab7f25dd203b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ztgb" Sep 16 07:27:49.497242 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.496014 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/17757d39-4eae-4f8d-8c3a-679d3502d985-socket-dir\") pod \"aws-ebs-csi-driver-node-vj2rd\" (UID: \"17757d39-4eae-4f8d-8c3a-679d3502d985\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vj2rd" Sep 16 07:27:49.497242 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.496017 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8d5c17cc-f191-4354-9c73-ab7f25dd203b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4ztgb\" (UID: \"8d5c17cc-f191-4354-9c73-ab7f25dd203b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ztgb" Sep 16 07:27:49.497242 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.496061 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/280250dc-6c09-4766-822c-b8bdf97e4a49-multus-conf-dir\") pod \"multus-4cb95\" (UID: \"280250dc-6c09-4766-822c-b8bdf97e4a49\") " pod="openshift-multus/multus-4cb95" Sep 16 07:27:49.497242 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.496062 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/17757d39-4eae-4f8d-8c3a-679d3502d985-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vj2rd\" (UID: \"17757d39-4eae-4f8d-8c3a-679d3502d985\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vj2rd" Sep 16 07:27:49.497242 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.496079 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/280250dc-6c09-4766-822c-b8bdf97e4a49-host-run-netns\") pod \"multus-4cb95\" (UID: \"280250dc-6c09-4766-822c-b8bdf97e4a49\") " pod="openshift-multus/multus-4cb95" Sep 16 07:27:49.497242 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.496099 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/280250dc-6c09-4766-822c-b8bdf97e4a49-host-run-netns\") pod \"multus-4cb95\" (UID: \"280250dc-6c09-4766-822c-b8bdf97e4a49\") " pod="openshift-multus/multus-4cb95" Sep 16 07:27:49.497242 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.496112 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb688a0e-2bd2-4df2-b475-7eba4237e526-etc-kubernetes\") pod \"tuned-zjxc4\" (UID: \"eb688a0e-2bd2-4df2-b475-7eba4237e526\") " pod="openshift-cluster-node-tuning-operator/tuned-zjxc4" Sep 16 07:27:49.497242 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.496126 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5f7afb78-2955-49e0-adec-6e8aae7ae51a-cnibin\") pod \"multus-additional-cni-plugins-qprhb\" (UID: \"5f7afb78-2955-49e0-adec-6e8aae7ae51a\") " pod="openshift-multus/multus-additional-cni-plugins-qprhb" Sep 16 07:27:49.497242 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.496142 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/eb688a0e-2bd2-4df2-b475-7eba4237e526-run\") pod \"tuned-zjxc4\" (UID: \"eb688a0e-2bd2-4df2-b475-7eba4237e526\") " pod="openshift-cluster-node-tuning-operator/tuned-zjxc4" Sep 16 07:27:49.497242 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.496170 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/17757d39-4eae-4f8d-8c3a-679d3502d985-device-dir\") pod \"aws-ebs-csi-driver-node-vj2rd\" (UID: \"17757d39-4eae-4f8d-8c3a-679d3502d985\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vj2rd" Sep 16 07:27:49.497242 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.496195 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8d5c17cc-f191-4354-9c73-ab7f25dd203b-env-overrides\") pod \"ovnkube-node-4ztgb\" (UID: \"8d5c17cc-f191-4354-9c73-ab7f25dd203b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ztgb" Sep 16 07:27:49.498095 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.496221 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8d5c17cc-f191-4354-9c73-ab7f25dd203b-ovnkube-script-lib\") pod \"ovnkube-node-4ztgb\" (UID: \"8d5c17cc-f191-4354-9c73-ab7f25dd203b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ztgb" Sep 16 07:27:49.498095 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.496246 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/eb688a0e-2bd2-4df2-b475-7eba4237e526-host\") pod \"tuned-zjxc4\" (UID: \"eb688a0e-2bd2-4df2-b475-7eba4237e526\") " pod="openshift-cluster-node-tuning-operator/tuned-zjxc4" Sep 16 07:27:49.498095 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.496258 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/eb688a0e-2bd2-4df2-b475-7eba4237e526-lib-modules\") pod \"tuned-zjxc4\" (UID: \"eb688a0e-2bd2-4df2-b475-7eba4237e526\") " pod="openshift-cluster-node-tuning-operator/tuned-zjxc4" Sep 16 07:27:49.498095 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.496259 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/280250dc-6c09-4766-822c-b8bdf97e4a49-multus-socket-dir-parent\") pod \"multus-4cb95\" (UID: \"280250dc-6c09-4766-822c-b8bdf97e4a49\") " pod="openshift-multus/multus-4cb95" Sep 16 07:27:49.498095 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.496300 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/eb688a0e-2bd2-4df2-b475-7eba4237e526-etc-sysconfig\") pod \"tuned-zjxc4\" (UID: \"eb688a0e-2bd2-4df2-b475-7eba4237e526\") " pod="openshift-cluster-node-tuning-operator/tuned-zjxc4" Sep 16 07:27:49.498095 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.496305 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/280250dc-6c09-4766-822c-b8bdf97e4a49-host-var-lib-cni-multus\") pod \"multus-4cb95\" (UID: \"280250dc-6c09-4766-822c-b8bdf97e4a49\") " pod="openshift-multus/multus-4cb95" Sep 16 07:27:49.498095 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.496375 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/17757d39-4eae-4f8d-8c3a-679d3502d985-sys-fs\") pod \"aws-ebs-csi-driver-node-vj2rd\" (UID: \"17757d39-4eae-4f8d-8c3a-679d3502d985\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vj2rd" Sep 16 07:27:49.498095 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.496385 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb688a0e-2bd2-4df2-b475-7eba4237e526-etc-kubernetes\") pod \"tuned-zjxc4\" (UID: \"eb688a0e-2bd2-4df2-b475-7eba4237e526\") " pod="openshift-cluster-node-tuning-operator/tuned-zjxc4" Sep 16 07:27:49.498095 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.496429 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/eb688a0e-2bd2-4df2-b475-7eba4237e526-run\") pod \"tuned-zjxc4\" (UID: \"eb688a0e-2bd2-4df2-b475-7eba4237e526\") " pod="openshift-cluster-node-tuning-operator/tuned-zjxc4" Sep 16 07:27:49.498095 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.496468 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/eb688a0e-2bd2-4df2-b475-7eba4237e526-etc-sysctl-d\") pod \"tuned-zjxc4\" (UID: \"eb688a0e-2bd2-4df2-b475-7eba4237e526\") " pod="openshift-cluster-node-tuning-operator/tuned-zjxc4" Sep 16 07:27:49.498095 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.496470 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/17757d39-4eae-4f8d-8c3a-679d3502d985-device-dir\") pod \"aws-ebs-csi-driver-node-vj2rd\" (UID: \"17757d39-4eae-4f8d-8c3a-679d3502d985\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vj2rd" Sep 16 07:27:49.498095 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.496845 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/280250dc-6c09-4766-822c-b8bdf97e4a49-host-var-lib-kubelet\") pod \"multus-4cb95\" (UID: \"280250dc-6c09-4766-822c-b8bdf97e4a49\") " pod="openshift-multus/multus-4cb95" Sep 16 07:27:49.498095 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.496930 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/eb688a0e-2bd2-4df2-b475-7eba4237e526-host\") pod \"tuned-zjxc4\" (UID: \"eb688a0e-2bd2-4df2-b475-7eba4237e526\") " pod="openshift-cluster-node-tuning-operator/tuned-zjxc4" Sep 16 07:27:49.498095 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.495711 2557 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 16 07:27:49.498095 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.497165 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/280250dc-6c09-4766-822c-b8bdf97e4a49-cni-binary-copy\") pod \"multus-4cb95\" (UID: \"280250dc-6c09-4766-822c-b8bdf97e4a49\") " pod="openshift-multus/multus-4cb95" Sep 16 07:27:49.498095 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.497330 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20f6d954-1104-49b9-ab10-32e0de4ef8e0-tmp-dir\") pod \"node-resolver-8gnk6\" (UID: \"20f6d954-1104-49b9-ab10-32e0de4ef8e0\") " pod="openshift-dns/node-resolver-8gnk6" Sep 16 07:27:49.498095 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.497469 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5f7afb78-2955-49e0-adec-6e8aae7ae51a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qprhb\" (UID: \"5f7afb78-2955-49e0-adec-6e8aae7ae51a\") " pod="openshift-multus/multus-additional-cni-plugins-qprhb" Sep 16 07:27:49.498095 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.497660 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/eb688a0e-2bd2-4df2-b475-7eba4237e526-etc-sysctl-conf\") pod \"tuned-zjxc4\" (UID: \"eb688a0e-2bd2-4df2-b475-7eba4237e526\") " pod="openshift-cluster-node-tuning-operator/tuned-zjxc4" Sep 16 07:27:49.499014 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.497767 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/eb688a0e-2bd2-4df2-b475-7eba4237e526-etc-modprobe-d\") pod \"tuned-zjxc4\" (UID: \"eb688a0e-2bd2-4df2-b475-7eba4237e526\") " pod="openshift-cluster-node-tuning-operator/tuned-zjxc4" Sep 16 07:27:49.499014 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.497829 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/280250dc-6c09-4766-822c-b8bdf97e4a49-system-cni-dir\") pod \"multus-4cb95\" (UID: \"280250dc-6c09-4766-822c-b8bdf97e4a49\") " pod="openshift-multus/multus-4cb95" Sep 16 07:27:49.499014 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.497877 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/280250dc-6c09-4766-822c-b8bdf97e4a49-host-run-k8s-cni-cncf-io\") pod \"multus-4cb95\" (UID: \"280250dc-6c09-4766-822c-b8bdf97e4a49\") " pod="openshift-multus/multus-4cb95" Sep 16 07:27:49.499014 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.498249 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/90e5e23a-060b-4612-80f5-f9a600b6d476-kubelet-config\") pod \"global-pull-secret-syncer-t8thj\" (UID: \"90e5e23a-060b-4612-80f5-f9a600b6d476\") " pod="kube-system/global-pull-secret-syncer-t8thj" Sep 16 07:27:49.499014 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.496268 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/280250dc-6c09-4766-822c-b8bdf97e4a49-host-var-lib-cni-multus\") pod \"multus-4cb95\" (UID: \"280250dc-6c09-4766-822c-b8bdf97e4a49\") " pod="openshift-multus/multus-4cb95" Sep 16 07:27:49.499014 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.498642 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pz6vv\" (UniqueName: \"kubernetes.io/projected/53ab9fb8-54fd-478d-9b2b-f69c8aac0d02-kube-api-access-pz6vv\") pod \"iptables-alerter-fhnn7\" (UID: \"53ab9fb8-54fd-478d-9b2b-f69c8aac0d02\") " pod="openshift-network-operator/iptables-alerter-fhnn7" Sep 16 07:27:49.499014 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.498677 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8d5c17cc-f191-4354-9c73-ab7f25dd203b-etc-openvswitch\") pod \"ovnkube-node-4ztgb\" (UID: \"8d5c17cc-f191-4354-9c73-ab7f25dd203b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ztgb" Sep 16 07:27:49.499014 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.498684 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5f7afb78-2955-49e0-adec-6e8aae7ae51a-cni-binary-copy\") pod \"multus-additional-cni-plugins-qprhb\" (UID: \"5f7afb78-2955-49e0-adec-6e8aae7ae51a\") " pod="openshift-multus/multus-additional-cni-plugins-qprhb" Sep 16 07:27:49.499014 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.498715 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8d5c17cc-f191-4354-9c73-ab7f25dd203b-run-openvswitch\") pod \"ovnkube-node-4ztgb\" (UID: \"8d5c17cc-f191-4354-9c73-ab7f25dd203b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ztgb" Sep 16 07:27:49.499014 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.498773 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8d5c17cc-f191-4354-9c73-ab7f25dd203b-ovnkube-config\") pod \"ovnkube-node-4ztgb\" (UID: \"8d5c17cc-f191-4354-9c73-ab7f25dd203b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ztgb" Sep 16 07:27:49.499014 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.498833 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6r6nv\" (UniqueName: \"kubernetes.io/projected/bfde2ba8-6a5f-413f-9ce9-d9ce0af2c758-kube-api-access-6r6nv\") pod \"network-check-target-jtqz9\" (UID: \"bfde2ba8-6a5f-413f-9ce9-d9ce0af2c758\") " pod="openshift-network-diagnostics/network-check-target-jtqz9" Sep 16 07:27:49.499014 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.498884 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eb688a0e-2bd2-4df2-b475-7eba4237e526-sys\") pod \"tuned-zjxc4\" (UID: \"eb688a0e-2bd2-4df2-b475-7eba4237e526\") " pod="openshift-cluster-node-tuning-operator/tuned-zjxc4" Sep 16 07:27:49.499014 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.498916 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eb688a0e-2bd2-4df2-b475-7eba4237e526-tmp\") pod \"tuned-zjxc4\" (UID: \"eb688a0e-2bd2-4df2-b475-7eba4237e526\") " pod="openshift-cluster-node-tuning-operator/tuned-zjxc4" Sep 16 07:27:49.499014 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.498996 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5f7afb78-2955-49e0-adec-6e8aae7ae51a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qprhb\" (UID: \"5f7afb78-2955-49e0-adec-6e8aae7ae51a\") " pod="openshift-multus/multus-additional-cni-plugins-qprhb" Sep 16 07:27:49.499690 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.499033 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/53ab9fb8-54fd-478d-9b2b-f69c8aac0d02-host-slash\") pod \"iptables-alerter-fhnn7\" (UID: \"53ab9fb8-54fd-478d-9b2b-f69c8aac0d02\") " pod="openshift-network-operator/iptables-alerter-fhnn7" Sep 16 07:27:49.499690 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.499229 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8d5c17cc-f191-4354-9c73-ab7f25dd203b-host-slash\") pod \"ovnkube-node-4ztgb\" (UID: \"8d5c17cc-f191-4354-9c73-ab7f25dd203b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ztgb" Sep 16 07:27:49.499690 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.499264 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8d5c17cc-f191-4354-9c73-ab7f25dd203b-run-systemd\") pod \"ovnkube-node-4ztgb\" (UID: \"8d5c17cc-f191-4354-9c73-ab7f25dd203b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ztgb" Sep 16 07:27:49.499690 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.499274 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1ae92448-74b5-4b28-b337-758f6884d09a-konnectivity-ca\") pod \"konnectivity-agent-lq72c\" (UID: \"1ae92448-74b5-4b28-b337-758f6884d09a\") " pod="kube-system/konnectivity-agent-lq72c" Sep 16 07:27:49.499690 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.499345 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/280250dc-6c09-4766-822c-b8bdf97e4a49-hostroot\") pod \"multus-4cb95\" (UID: \"280250dc-6c09-4766-822c-b8bdf97e4a49\") " pod="openshift-multus/multus-4cb95" Sep 16 07:27:49.499690 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.499358 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eb688a0e-2bd2-4df2-b475-7eba4237e526-sys\") pod \"tuned-zjxc4\" (UID: \"eb688a0e-2bd2-4df2-b475-7eba4237e526\") " pod="openshift-cluster-node-tuning-operator/tuned-zjxc4" Sep 16 07:27:49.499690 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.499383 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5f7afb78-2955-49e0-adec-6e8aae7ae51a-system-cni-dir\") pod \"multus-additional-cni-plugins-qprhb\" (UID: \"5f7afb78-2955-49e0-adec-6e8aae7ae51a\") " pod="openshift-multus/multus-additional-cni-plugins-qprhb" Sep 16 07:27:49.499690 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.499418 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5f7afb78-2955-49e0-adec-6e8aae7ae51a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qprhb\" (UID: \"5f7afb78-2955-49e0-adec-6e8aae7ae51a\") " pod="openshift-multus/multus-additional-cni-plugins-qprhb" Sep 16 07:27:49.499690 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.499541 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5f7afb78-2955-49e0-adec-6e8aae7ae51a-system-cni-dir\") pod \"multus-additional-cni-plugins-qprhb\" (UID: \"5f7afb78-2955-49e0-adec-6e8aae7ae51a\") " pod="openshift-multus/multus-additional-cni-plugins-qprhb" Sep 16 07:27:49.499690 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.499677 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/280250dc-6c09-4766-822c-b8bdf97e4a49-hostroot\") pod \"multus-4cb95\" (UID: \"280250dc-6c09-4766-822c-b8bdf97e4a49\") " pod="openshift-multus/multus-4cb95" Sep 16 07:27:49.500100 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.499703 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5f7afb78-2955-49e0-adec-6e8aae7ae51a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qprhb\" (UID: \"5f7afb78-2955-49e0-adec-6e8aae7ae51a\") " pod="openshift-multus/multus-additional-cni-plugins-qprhb" Sep 16 07:27:49.500100 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.499722 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8d5c17cc-f191-4354-9c73-ab7f25dd203b-host-run-netns\") pod \"ovnkube-node-4ztgb\" (UID: \"8d5c17cc-f191-4354-9c73-ab7f25dd203b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ztgb" Sep 16 07:27:49.500100 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.499767 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8d5c17cc-f191-4354-9c73-ab7f25dd203b-var-lib-openvswitch\") pod \"ovnkube-node-4ztgb\" (UID: \"8d5c17cc-f191-4354-9c73-ab7f25dd203b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ztgb" Sep 16 07:27:49.500100 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.499797 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8d5c17cc-f191-4354-9c73-ab7f25dd203b-host-run-ovn-kubernetes\") pod \"ovnkube-node-4ztgb\" (UID: \"8d5c17cc-f191-4354-9c73-ab7f25dd203b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ztgb" Sep 16 07:27:49.500100 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.499846 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/89f6995b-d178-4fe9-9593-ba8dc8f49351-host\") pod \"node-ca-vbxvc\" (UID: \"89f6995b-d178-4fe9-9593-ba8dc8f49351\") " pod="openshift-image-registry/node-ca-vbxvc" Sep 16 07:27:49.500100 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.499879 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/280250dc-6c09-4766-822c-b8bdf97e4a49-host-run-multus-certs\") pod \"multus-4cb95\" (UID: \"280250dc-6c09-4766-822c-b8bdf97e4a49\") " pod="openshift-multus/multus-4cb95" Sep 16 07:27:49.500100 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.499906 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/280250dc-6c09-4766-822c-b8bdf97e4a49-host-var-lib-cni-bin\") pod \"multus-4cb95\" (UID: \"280250dc-6c09-4766-822c-b8bdf97e4a49\") " pod="openshift-multus/multus-4cb95" Sep 16 07:27:49.500100 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.499920 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5f7afb78-2955-49e0-adec-6e8aae7ae51a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qprhb\" (UID: \"5f7afb78-2955-49e0-adec-6e8aae7ae51a\") " pod="openshift-multus/multus-additional-cni-plugins-qprhb" Sep 16 07:27:49.500100 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.499996 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/280250dc-6c09-4766-822c-b8bdf97e4a49-host-var-lib-cni-bin\") pod \"multus-4cb95\" (UID: \"280250dc-6c09-4766-822c-b8bdf97e4a49\") " pod="openshift-multus/multus-4cb95" Sep 16 07:27:49.500100 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.500044 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/280250dc-6c09-4766-822c-b8bdf97e4a49-host-run-multus-certs\") pod \"multus-4cb95\" (UID: \"280250dc-6c09-4766-822c-b8bdf97e4a49\") " pod="openshift-multus/multus-4cb95" Sep 16 07:27:49.500100 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.500080 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lz2fz\" (UniqueName: \"kubernetes.io/projected/296c5dcf-403c-4a1f-9d6f-05ffb7d75e38-kube-api-access-lz2fz\") pod \"network-metrics-daemon-kqc7r\" (UID: \"296c5dcf-403c-4a1f-9d6f-05ffb7d75e38\") " pod="openshift-multus/network-metrics-daemon-kqc7r" Sep 16 07:27:49.500598 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.500137 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/90e5e23a-060b-4612-80f5-f9a600b6d476-dbus\") pod \"global-pull-secret-syncer-t8thj\" (UID: \"90e5e23a-060b-4612-80f5-f9a600b6d476\") " pod="kube-system/global-pull-secret-syncer-t8thj" Sep 16 07:27:49.500598 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.500180 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/17757d39-4eae-4f8d-8c3a-679d3502d985-registration-dir\") pod \"aws-ebs-csi-driver-node-vj2rd\" (UID: \"17757d39-4eae-4f8d-8c3a-679d3502d985\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vj2rd" Sep 16 07:27:49.500598 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.500201 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/90e5e23a-060b-4612-80f5-f9a600b6d476-dbus\") pod \"global-pull-secret-syncer-t8thj\" (UID: \"90e5e23a-060b-4612-80f5-f9a600b6d476\") " pod="kube-system/global-pull-secret-syncer-t8thj" Sep 16 07:27:49.500598 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.500221 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/17757d39-4eae-4f8d-8c3a-679d3502d985-etc-selinux\") pod \"aws-ebs-csi-driver-node-vj2rd\" (UID: \"17757d39-4eae-4f8d-8c3a-679d3502d985\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vj2rd" Sep 16 07:27:49.500598 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.500288 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8d5c17cc-f191-4354-9c73-ab7f25dd203b-systemd-units\") pod \"ovnkube-node-4ztgb\" (UID: \"8d5c17cc-f191-4354-9c73-ab7f25dd203b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ztgb" Sep 16 07:27:49.500598 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.500309 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/17757d39-4eae-4f8d-8c3a-679d3502d985-registration-dir\") pod \"aws-ebs-csi-driver-node-vj2rd\" (UID: \"17757d39-4eae-4f8d-8c3a-679d3502d985\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vj2rd" Sep 16 07:27:49.500598 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.500313 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8d5c17cc-f191-4354-9c73-ab7f25dd203b-log-socket\") pod \"ovnkube-node-4ztgb\" (UID: \"8d5c17cc-f191-4354-9c73-ab7f25dd203b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ztgb" Sep 16 07:27:49.500598 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.500426 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/17757d39-4eae-4f8d-8c3a-679d3502d985-etc-selinux\") pod \"aws-ebs-csi-driver-node-vj2rd\" (UID: \"17757d39-4eae-4f8d-8c3a-679d3502d985\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vj2rd" Sep 16 07:27:49.504964 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.504767 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/eb688a0e-2bd2-4df2-b475-7eba4237e526-etc-tuned\") pod \"tuned-zjxc4\" (UID: \"eb688a0e-2bd2-4df2-b475-7eba4237e526\") " pod="openshift-cluster-node-tuning-operator/tuned-zjxc4" Sep 16 07:27:49.505075 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.504976 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1ae92448-74b5-4b28-b337-758f6884d09a-agent-certs\") pod \"konnectivity-agent-lq72c\" (UID: \"1ae92448-74b5-4b28-b337-758f6884d09a\") " pod="kube-system/konnectivity-agent-lq72c" Sep 16 07:27:49.506279 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.506238 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eb688a0e-2bd2-4df2-b475-7eba4237e526-tmp\") pod \"tuned-zjxc4\" (UID: \"eb688a0e-2bd2-4df2-b475-7eba4237e526\") " pod="openshift-cluster-node-tuning-operator/tuned-zjxc4" Sep 16 07:27:49.512199 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.512173 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pcsb\" (UniqueName: \"kubernetes.io/projected/280250dc-6c09-4766-822c-b8bdf97e4a49-kube-api-access-7pcsb\") pod \"multus-4cb95\" (UID: \"280250dc-6c09-4766-822c-b8bdf97e4a49\") " pod="openshift-multus/multus-4cb95" Sep 16 07:27:49.512652 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.512619 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mf5v4\" (UniqueName: \"kubernetes.io/projected/5f7afb78-2955-49e0-adec-6e8aae7ae51a-kube-api-access-mf5v4\") pod \"multus-additional-cni-plugins-qprhb\" (UID: \"5f7afb78-2955-49e0-adec-6e8aae7ae51a\") " pod="openshift-multus/multus-additional-cni-plugins-qprhb" Sep 16 07:27:49.513918 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.513887 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4fdx\" (UniqueName: \"kubernetes.io/projected/17757d39-4eae-4f8d-8c3a-679d3502d985-kube-api-access-t4fdx\") pod \"aws-ebs-csi-driver-node-vj2rd\" (UID: \"17757d39-4eae-4f8d-8c3a-679d3502d985\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vj2rd" Sep 16 07:27:49.514358 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.514333 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ptk2\" (UniqueName: \"kubernetes.io/projected/20f6d954-1104-49b9-ab10-32e0de4ef8e0-kube-api-access-5ptk2\") pod \"node-resolver-8gnk6\" (UID: \"20f6d954-1104-49b9-ab10-32e0de4ef8e0\") " pod="openshift-dns/node-resolver-8gnk6" Sep 16 07:27:49.514740 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.514689 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2bcn\" (UniqueName: \"kubernetes.io/projected/eb688a0e-2bd2-4df2-b475-7eba4237e526-kube-api-access-d2bcn\") pod \"tuned-zjxc4\" (UID: \"eb688a0e-2bd2-4df2-b475-7eba4237e526\") " pod="openshift-cluster-node-tuning-operator/tuned-zjxc4" Sep 16 07:27:49.524001 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.523953 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-221.ec2.internal" event={"ID":"8ac46b15b636b4c144bce2b4a1b3535a","Type":"ContainerStarted","Data":"1357c27ac5cc88b64d9f21268ba261444be1c6f6d33fa4bcf3fef144b401dfed"} Sep 16 07:27:49.525725 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.525690 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-221.ec2.internal" event={"ID":"3825bcd696c373ea98c5e3521a26b861","Type":"ContainerStarted","Data":"1865e5f90c2f37a52aaaddd0b818ccf4751551c828815ba5f35bc31d71473c43"} Sep 16 07:27:49.552674 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.552645 2557 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 16 07:27:49.602565 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.600672 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lz2fz\" (UniqueName: \"kubernetes.io/projected/296c5dcf-403c-4a1f-9d6f-05ffb7d75e38-kube-api-access-lz2fz\") pod \"network-metrics-daemon-kqc7r\" (UID: \"296c5dcf-403c-4a1f-9d6f-05ffb7d75e38\") " pod="openshift-multus/network-metrics-daemon-kqc7r" Sep 16 07:27:49.602565 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.600712 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8d5c17cc-f191-4354-9c73-ab7f25dd203b-systemd-units\") pod \"ovnkube-node-4ztgb\" (UID: \"8d5c17cc-f191-4354-9c73-ab7f25dd203b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ztgb" Sep 16 07:27:49.602565 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.600736 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8d5c17cc-f191-4354-9c73-ab7f25dd203b-log-socket\") pod \"ovnkube-node-4ztgb\" (UID: \"8d5c17cc-f191-4354-9c73-ab7f25dd203b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ztgb" Sep 16 07:27:49.602565 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.600762 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/89f6995b-d178-4fe9-9593-ba8dc8f49351-serviceca\") pod \"node-ca-vbxvc\" (UID: \"89f6995b-d178-4fe9-9593-ba8dc8f49351\") " pod="openshift-image-registry/node-ca-vbxvc" Sep 16 07:27:49.602565 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.600787 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8d5c17cc-f191-4354-9c73-ab7f25dd203b-ovn-node-metrics-cert\") pod \"ovnkube-node-4ztgb\" (UID: \"8d5c17cc-f191-4354-9c73-ab7f25dd203b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ztgb" Sep 16 07:27:49.602565 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.600812 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fznzt\" (UniqueName: \"kubernetes.io/projected/89f6995b-d178-4fe9-9593-ba8dc8f49351-kube-api-access-fznzt\") pod \"node-ca-vbxvc\" (UID: \"89f6995b-d178-4fe9-9593-ba8dc8f49351\") " pod="openshift-image-registry/node-ca-vbxvc" Sep 16 07:27:49.602565 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.600841 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8d5c17cc-f191-4354-9c73-ab7f25dd203b-host-cni-netd\") pod \"ovnkube-node-4ztgb\" (UID: \"8d5c17cc-f191-4354-9c73-ab7f25dd203b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ztgb" Sep 16 07:27:49.602565 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.600868 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pqxg\" (UniqueName: \"kubernetes.io/projected/8d5c17cc-f191-4354-9c73-ab7f25dd203b-kube-api-access-2pqxg\") pod \"ovnkube-node-4ztgb\" (UID: \"8d5c17cc-f191-4354-9c73-ab7f25dd203b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ztgb" Sep 16 07:27:49.602565 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.600917 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8d5c17cc-f191-4354-9c73-ab7f25dd203b-run-ovn\") pod \"ovnkube-node-4ztgb\" (UID: \"8d5c17cc-f191-4354-9c73-ab7f25dd203b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ztgb" Sep 16 07:27:49.602565 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.600942 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8d5c17cc-f191-4354-9c73-ab7f25dd203b-node-log\") pod \"ovnkube-node-4ztgb\" (UID: \"8d5c17cc-f191-4354-9c73-ab7f25dd203b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ztgb" Sep 16 07:27:49.602565 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.600965 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8d5c17cc-f191-4354-9c73-ab7f25dd203b-host-cni-bin\") pod \"ovnkube-node-4ztgb\" (UID: \"8d5c17cc-f191-4354-9c73-ab7f25dd203b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ztgb" Sep 16 07:27:49.602565 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.600999 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/296c5dcf-403c-4a1f-9d6f-05ffb7d75e38-metrics-certs\") pod \"network-metrics-daemon-kqc7r\" (UID: \"296c5dcf-403c-4a1f-9d6f-05ffb7d75e38\") " pod="openshift-multus/network-metrics-daemon-kqc7r" Sep 16 07:27:49.602565 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.601028 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/53ab9fb8-54fd-478d-9b2b-f69c8aac0d02-iptables-alerter-script\") pod \"iptables-alerter-fhnn7\" (UID: \"53ab9fb8-54fd-478d-9b2b-f69c8aac0d02\") " pod="openshift-network-operator/iptables-alerter-fhnn7" Sep 16 07:27:49.602565 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.601052 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8d5c17cc-f191-4354-9c73-ab7f25dd203b-host-kubelet\") pod \"ovnkube-node-4ztgb\" (UID: \"8d5c17cc-f191-4354-9c73-ab7f25dd203b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ztgb" Sep 16 07:27:49.602565 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.601079 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8d5c17cc-f191-4354-9c73-ab7f25dd203b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4ztgb\" (UID: \"8d5c17cc-f191-4354-9c73-ab7f25dd203b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ztgb" Sep 16 07:27:49.602565 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.601110 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8d5c17cc-f191-4354-9c73-ab7f25dd203b-env-overrides\") pod \"ovnkube-node-4ztgb\" (UID: \"8d5c17cc-f191-4354-9c73-ab7f25dd203b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ztgb" Sep 16 07:27:49.602565 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.601141 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8d5c17cc-f191-4354-9c73-ab7f25dd203b-ovnkube-script-lib\") pod \"ovnkube-node-4ztgb\" (UID: \"8d5c17cc-f191-4354-9c73-ab7f25dd203b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ztgb" Sep 16 07:27:49.603531 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.601168 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pz6vv\" (UniqueName: \"kubernetes.io/projected/53ab9fb8-54fd-478d-9b2b-f69c8aac0d02-kube-api-access-pz6vv\") pod \"iptables-alerter-fhnn7\" (UID: \"53ab9fb8-54fd-478d-9b2b-f69c8aac0d02\") " pod="openshift-network-operator/iptables-alerter-fhnn7" Sep 16 07:27:49.603531 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.601204 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8d5c17cc-f191-4354-9c73-ab7f25dd203b-etc-openvswitch\") pod \"ovnkube-node-4ztgb\" (UID: \"8d5c17cc-f191-4354-9c73-ab7f25dd203b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ztgb" Sep 16 07:27:49.603531 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.601227 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8d5c17cc-f191-4354-9c73-ab7f25dd203b-run-openvswitch\") pod \"ovnkube-node-4ztgb\" (UID: \"8d5c17cc-f191-4354-9c73-ab7f25dd203b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ztgb" Sep 16 07:27:49.603531 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.601259 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8d5c17cc-f191-4354-9c73-ab7f25dd203b-ovnkube-config\") pod \"ovnkube-node-4ztgb\" (UID: \"8d5c17cc-f191-4354-9c73-ab7f25dd203b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ztgb" Sep 16 07:27:49.603531 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.601284 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6r6nv\" (UniqueName: \"kubernetes.io/projected/bfde2ba8-6a5f-413f-9ce9-d9ce0af2c758-kube-api-access-6r6nv\") pod \"network-check-target-jtqz9\" (UID: \"bfde2ba8-6a5f-413f-9ce9-d9ce0af2c758\") " pod="openshift-network-diagnostics/network-check-target-jtqz9" Sep 16 07:27:49.603531 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.601346 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/53ab9fb8-54fd-478d-9b2b-f69c8aac0d02-host-slash\") pod \"iptables-alerter-fhnn7\" (UID: \"53ab9fb8-54fd-478d-9b2b-f69c8aac0d02\") " pod="openshift-network-operator/iptables-alerter-fhnn7" Sep 16 07:27:49.603531 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.601374 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8d5c17cc-f191-4354-9c73-ab7f25dd203b-host-slash\") pod \"ovnkube-node-4ztgb\" (UID: \"8d5c17cc-f191-4354-9c73-ab7f25dd203b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ztgb" Sep 16 07:27:49.603531 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.601400 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8d5c17cc-f191-4354-9c73-ab7f25dd203b-run-systemd\") pod \"ovnkube-node-4ztgb\" (UID: \"8d5c17cc-f191-4354-9c73-ab7f25dd203b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ztgb" Sep 16 07:27:49.603531 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.601433 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8d5c17cc-f191-4354-9c73-ab7f25dd203b-host-run-netns\") pod \"ovnkube-node-4ztgb\" (UID: \"8d5c17cc-f191-4354-9c73-ab7f25dd203b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ztgb" Sep 16 07:27:49.603531 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.601491 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8d5c17cc-f191-4354-9c73-ab7f25dd203b-var-lib-openvswitch\") pod \"ovnkube-node-4ztgb\" (UID: \"8d5c17cc-f191-4354-9c73-ab7f25dd203b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ztgb" Sep 16 07:27:49.603531 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.601520 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8d5c17cc-f191-4354-9c73-ab7f25dd203b-host-run-ovn-kubernetes\") pod \"ovnkube-node-4ztgb\" (UID: \"8d5c17cc-f191-4354-9c73-ab7f25dd203b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ztgb" Sep 16 07:27:49.603531 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.601546 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/89f6995b-d178-4fe9-9593-ba8dc8f49351-host\") pod \"node-ca-vbxvc\" (UID: \"89f6995b-d178-4fe9-9593-ba8dc8f49351\") " pod="openshift-image-registry/node-ca-vbxvc" Sep 16 07:27:49.603531 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.601636 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/89f6995b-d178-4fe9-9593-ba8dc8f49351-host\") pod \"node-ca-vbxvc\" (UID: \"89f6995b-d178-4fe9-9593-ba8dc8f49351\") " pod="openshift-image-registry/node-ca-vbxvc" Sep 16 07:27:49.603531 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.601769 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8d5c17cc-f191-4354-9c73-ab7f25dd203b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4ztgb\" (UID: \"8d5c17cc-f191-4354-9c73-ab7f25dd203b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ztgb" Sep 16 07:27:49.603531 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.601818 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8d5c17cc-f191-4354-9c73-ab7f25dd203b-systemd-units\") pod \"ovnkube-node-4ztgb\" (UID: \"8d5c17cc-f191-4354-9c73-ab7f25dd203b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ztgb" Sep 16 07:27:49.603531 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.601859 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8d5c17cc-f191-4354-9c73-ab7f25dd203b-log-socket\") pod \"ovnkube-node-4ztgb\" (UID: \"8d5c17cc-f191-4354-9c73-ab7f25dd203b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ztgb" Sep 16 07:27:49.603531 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.602292 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8d5c17cc-f191-4354-9c73-ab7f25dd203b-env-overrides\") pod \"ovnkube-node-4ztgb\" (UID: \"8d5c17cc-f191-4354-9c73-ab7f25dd203b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ztgb" Sep 16 07:27:49.604429 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.602370 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8d5c17cc-f191-4354-9c73-ab7f25dd203b-node-log\") pod \"ovnkube-node-4ztgb\" (UID: \"8d5c17cc-f191-4354-9c73-ab7f25dd203b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ztgb" Sep 16 07:27:49.604429 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.602398 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8d5c17cc-f191-4354-9c73-ab7f25dd203b-var-lib-openvswitch\") pod \"ovnkube-node-4ztgb\" (UID: \"8d5c17cc-f191-4354-9c73-ab7f25dd203b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ztgb" Sep 16 07:27:49.604429 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.602411 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/53ab9fb8-54fd-478d-9b2b-f69c8aac0d02-host-slash\") pod \"iptables-alerter-fhnn7\" (UID: \"53ab9fb8-54fd-478d-9b2b-f69c8aac0d02\") " pod="openshift-network-operator/iptables-alerter-fhnn7" Sep 16 07:27:49.604429 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.602424 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8d5c17cc-f191-4354-9c73-ab7f25dd203b-host-slash\") pod \"ovnkube-node-4ztgb\" (UID: \"8d5c17cc-f191-4354-9c73-ab7f25dd203b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ztgb" Sep 16 07:27:49.604429 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.602431 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8d5c17cc-f191-4354-9c73-ab7f25dd203b-host-run-netns\") pod \"ovnkube-node-4ztgb\" (UID: \"8d5c17cc-f191-4354-9c73-ab7f25dd203b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ztgb" Sep 16 07:27:49.604429 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.602446 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8d5c17cc-f191-4354-9c73-ab7f25dd203b-host-run-ovn-kubernetes\") pod \"ovnkube-node-4ztgb\" (UID: \"8d5c17cc-f191-4354-9c73-ab7f25dd203b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ztgb" Sep 16 07:27:49.604429 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.602292 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/89f6995b-d178-4fe9-9593-ba8dc8f49351-serviceca\") pod \"node-ca-vbxvc\" (UID: \"89f6995b-d178-4fe9-9593-ba8dc8f49351\") " pod="openshift-image-registry/node-ca-vbxvc" Sep 16 07:27:49.604429 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.602470 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8d5c17cc-f191-4354-9c73-ab7f25dd203b-etc-openvswitch\") pod \"ovnkube-node-4ztgb\" (UID: \"8d5c17cc-f191-4354-9c73-ab7f25dd203b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ztgb" Sep 16 07:27:49.604429 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.602492 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8d5c17cc-f191-4354-9c73-ab7f25dd203b-host-cni-netd\") pod \"ovnkube-node-4ztgb\" (UID: \"8d5c17cc-f191-4354-9c73-ab7f25dd203b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ztgb" Sep 16 07:27:49.604429 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.602506 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8d5c17cc-f191-4354-9c73-ab7f25dd203b-run-systemd\") pod \"ovnkube-node-4ztgb\" (UID: \"8d5c17cc-f191-4354-9c73-ab7f25dd203b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ztgb" Sep 16 07:27:49.604429 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.602894 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8d5c17cc-f191-4354-9c73-ab7f25dd203b-run-ovn\") pod \"ovnkube-node-4ztgb\" (UID: \"8d5c17cc-f191-4354-9c73-ab7f25dd203b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ztgb" Sep 16 07:27:49.604429 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.602972 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8d5c17cc-f191-4354-9c73-ab7f25dd203b-run-openvswitch\") pod \"ovnkube-node-4ztgb\" (UID: \"8d5c17cc-f191-4354-9c73-ab7f25dd203b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ztgb" Sep 16 07:27:49.604429 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.603029 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8d5c17cc-f191-4354-9c73-ab7f25dd203b-host-cni-bin\") pod \"ovnkube-node-4ztgb\" (UID: \"8d5c17cc-f191-4354-9c73-ab7f25dd203b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ztgb" Sep 16 07:27:49.604429 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.603061 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8d5c17cc-f191-4354-9c73-ab7f25dd203b-ovnkube-script-lib\") pod \"ovnkube-node-4ztgb\" (UID: \"8d5c17cc-f191-4354-9c73-ab7f25dd203b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ztgb" Sep 16 07:27:49.604429 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:27:49.603086 2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 07:27:49.604429 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.603197 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8d5c17cc-f191-4354-9c73-ab7f25dd203b-host-kubelet\") pod \"ovnkube-node-4ztgb\" (UID: \"8d5c17cc-f191-4354-9c73-ab7f25dd203b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ztgb" Sep 16 07:27:49.604429 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:27:49.603204 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/296c5dcf-403c-4a1f-9d6f-05ffb7d75e38-metrics-certs podName:296c5dcf-403c-4a1f-9d6f-05ffb7d75e38 nodeName:}" failed. No retries permitted until 2026-09-16 07:27:50.103186036 +0000 UTC m=+3.130087034 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/296c5dcf-403c-4a1f-9d6f-05ffb7d75e38-metrics-certs") pod "network-metrics-daemon-kqc7r" (UID: "296c5dcf-403c-4a1f-9d6f-05ffb7d75e38") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 07:27:49.605336 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.603594 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/53ab9fb8-54fd-478d-9b2b-f69c8aac0d02-iptables-alerter-script\") pod \"iptables-alerter-fhnn7\" (UID: \"53ab9fb8-54fd-478d-9b2b-f69c8aac0d02\") " pod="openshift-network-operator/iptables-alerter-fhnn7" Sep 16 07:27:49.605336 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.604310 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8d5c17cc-f191-4354-9c73-ab7f25dd203b-ovnkube-config\") pod \"ovnkube-node-4ztgb\" (UID: \"8d5c17cc-f191-4354-9c73-ab7f25dd203b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ztgb" Sep 16 07:27:49.605853 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.605828 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8d5c17cc-f191-4354-9c73-ab7f25dd203b-ovn-node-metrics-cert\") pod \"ovnkube-node-4ztgb\" (UID: \"8d5c17cc-f191-4354-9c73-ab7f25dd203b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ztgb" Sep 16 07:27:49.611196 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.611076 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pqxg\" (UniqueName: \"kubernetes.io/projected/8d5c17cc-f191-4354-9c73-ab7f25dd203b-kube-api-access-2pqxg\") pod \"ovnkube-node-4ztgb\" (UID: \"8d5c17cc-f191-4354-9c73-ab7f25dd203b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ztgb" Sep 16 07:27:49.611815 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:27:49.611793 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 07:27:49.611815 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:27:49.611816 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 07:27:49.611964 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:27:49.611830 2557 projected.go:196] Error preparing data for projected volume kube-api-access-6r6nv for pod openshift-network-diagnostics/network-check-target-jtqz9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 07:27:49.611964 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:27:49.611884 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bfde2ba8-6a5f-413f-9ce9-d9ce0af2c758-kube-api-access-6r6nv podName:bfde2ba8-6a5f-413f-9ce9-d9ce0af2c758 nodeName:}" failed. No retries permitted until 2026-09-16 07:27:50.111868169 +0000 UTC m=+3.138769182 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-6r6nv" (UniqueName: "kubernetes.io/projected/bfde2ba8-6a5f-413f-9ce9-d9ce0af2c758-kube-api-access-6r6nv") pod "network-check-target-jtqz9" (UID: "bfde2ba8-6a5f-413f-9ce9-d9ce0af2c758") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 07:27:49.612736 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.612690 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lz2fz\" (UniqueName: \"kubernetes.io/projected/296c5dcf-403c-4a1f-9d6f-05ffb7d75e38-kube-api-access-lz2fz\") pod \"network-metrics-daemon-kqc7r\" (UID: \"296c5dcf-403c-4a1f-9d6f-05ffb7d75e38\") " pod="openshift-multus/network-metrics-daemon-kqc7r" Sep 16 07:27:49.614146 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.614126 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pz6vv\" (UniqueName: \"kubernetes.io/projected/53ab9fb8-54fd-478d-9b2b-f69c8aac0d02-kube-api-access-pz6vv\") pod \"iptables-alerter-fhnn7\" (UID: \"53ab9fb8-54fd-478d-9b2b-f69c8aac0d02\") " pod="openshift-network-operator/iptables-alerter-fhnn7" Sep 16 07:27:49.616599 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.616556 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fznzt\" (UniqueName: \"kubernetes.io/projected/89f6995b-d178-4fe9-9593-ba8dc8f49351-kube-api-access-fznzt\") pod \"node-ca-vbxvc\" (UID: \"89f6995b-d178-4fe9-9593-ba8dc8f49351\") " pod="openshift-image-registry/node-ca-vbxvc" Sep 16 07:27:49.678972 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.678849 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-8gnk6" Sep 16 07:27:49.685682 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.685654 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4cb95" Sep 16 07:27:49.694119 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.694086 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-lq72c" Sep 16 07:27:49.694966 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:49.694937 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20f6d954_1104_49b9_ab10_32e0de4ef8e0.slice/crio-6af888f9f06f4e8ec43e3a245251d5e21129d3dbee79c44fcda288a072a79c0e WatchSource:0}: Error finding container 6af888f9f06f4e8ec43e3a245251d5e21129d3dbee79c44fcda288a072a79c0e: Status 404 returned error can't find the container with id 6af888f9f06f4e8ec43e3a245251d5e21129d3dbee79c44fcda288a072a79c0e Sep 16 07:27:49.699110 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:49.699090 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod280250dc_6c09_4766_822c_b8bdf97e4a49.slice/crio-df35861cce6104f0c0588ccdd7c61ebdeb8d984e8947db485272869bceed0f62 WatchSource:0}: Error finding container df35861cce6104f0c0588ccdd7c61ebdeb8d984e8947db485272869bceed0f62: Status 404 returned error can't find the container with id df35861cce6104f0c0588ccdd7c61ebdeb8d984e8947db485272869bceed0f62 Sep 16 07:27:49.701895 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.701871 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vj2rd" Sep 16 07:27:49.708920 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:49.708895 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ae92448_74b5_4b28_b337_758f6884d09a.slice/crio-aad6991c629236a1bbc2f62e86dfd19332424c1b1fd652894fc4fdefa26bf994 WatchSource:0}: Error finding container aad6991c629236a1bbc2f62e86dfd19332424c1b1fd652894fc4fdefa26bf994: Status 404 returned error can't find the container with id aad6991c629236a1bbc2f62e86dfd19332424c1b1fd652894fc4fdefa26bf994 Sep 16 07:27:49.709031 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.709004 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zjxc4" Sep 16 07:27:49.716487 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.716459 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-qprhb" Sep 16 07:27:49.719043 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:49.719019 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17757d39_4eae_4f8d_8c3a_679d3502d985.slice/crio-31213a7a953002c14942ba3b972c088370f7119deaa74476efd89a80f411e314 WatchSource:0}: Error finding container 31213a7a953002c14942ba3b972c088370f7119deaa74476efd89a80f411e314: Status 404 returned error can't find the container with id 31213a7a953002c14942ba3b972c088370f7119deaa74476efd89a80f411e314 Sep 16 07:27:49.721279 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:49.721231 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb688a0e_2bd2_4df2_b475_7eba4237e526.slice/crio-f9c21f7d89a6ad06a1cffd1f007ccb3a22ed9d55117af69279c7b3f644ef0c29 WatchSource:0}: Error finding container f9c21f7d89a6ad06a1cffd1f007ccb3a22ed9d55117af69279c7b3f644ef0c29: Status 404 returned error can't find the container with id f9c21f7d89a6ad06a1cffd1f007ccb3a22ed9d55117af69279c7b3f644ef0c29 Sep 16 07:27:49.725758 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.725657 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-fhnn7" Sep 16 07:27:49.730676 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:49.730650 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f7afb78_2955_49e0_adec_6e8aae7ae51a.slice/crio-90b3d0c1a3a71b8876bb83d09a12aadbd98f3cb3065a17b41c9819cb128857c9 WatchSource:0}: Error finding container 90b3d0c1a3a71b8876bb83d09a12aadbd98f3cb3065a17b41c9819cb128857c9: Status 404 returned error can't find the container with id 90b3d0c1a3a71b8876bb83d09a12aadbd98f3cb3065a17b41c9819cb128857c9 Sep 16 07:27:49.734346 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.734142 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-4ztgb" Sep 16 07:27:49.740462 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:49.740442 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vbxvc" Sep 16 07:27:49.744630 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:49.744611 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d5c17cc_f191_4354_9c73_ab7f25dd203b.slice/crio-3767314a11a9e48dea0cf14e0a3a0b36e3a4f71a55af26dffbd7f93d9712502f WatchSource:0}: Error finding container 3767314a11a9e48dea0cf14e0a3a0b36e3a4f71a55af26dffbd7f93d9712502f: Status 404 returned error can't find the container with id 3767314a11a9e48dea0cf14e0a3a0b36e3a4f71a55af26dffbd7f93d9712502f Sep 16 07:27:49.748745 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:49.748722 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53ab9fb8_54fd_478d_9b2b_f69c8aac0d02.slice/crio-0faeec8e703907e7468ca94693d6c39c33281f8e7e7fda231219852e7a781d01 WatchSource:0}: Error finding container 0faeec8e703907e7468ca94693d6c39c33281f8e7e7fda231219852e7a781d01: Status 404 returned error can't find the container with id 0faeec8e703907e7468ca94693d6c39c33281f8e7e7fda231219852e7a781d01 Sep 16 07:27:49.756094 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:27:49.756068 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89f6995b_d178_4fe9_9593_ba8dc8f49351.slice/crio-5718737659f2fd9c3a0a28671c7535b8c3b08ec97c2fbd166f113102bc365d20 WatchSource:0}: Error finding container 5718737659f2fd9c3a0a28671c7535b8c3b08ec97c2fbd166f113102bc365d20: Status 404 returned error can't find the container with id 5718737659f2fd9c3a0a28671c7535b8c3b08ec97c2fbd166f113102bc365d20 Sep 16 07:27:50.001886 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:50.001790 2557 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 16 07:27:50.004635 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:50.004606 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/90e5e23a-060b-4612-80f5-f9a600b6d476-original-pull-secret\") pod \"global-pull-secret-syncer-t8thj\" (UID: \"90e5e23a-060b-4612-80f5-f9a600b6d476\") " pod="kube-system/global-pull-secret-syncer-t8thj" Sep 16 07:27:50.004896 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:27:50.004818 2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 07:27:50.004896 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:27:50.004889 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90e5e23a-060b-4612-80f5-f9a600b6d476-original-pull-secret podName:90e5e23a-060b-4612-80f5-f9a600b6d476 nodeName:}" failed. No retries permitted until 2026-09-16 07:27:51.004869471 +0000 UTC m=+4.031770464 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/90e5e23a-060b-4612-80f5-f9a600b6d476-original-pull-secret") pod "global-pull-secret-syncer-t8thj" (UID: "90e5e23a-060b-4612-80f5-f9a600b6d476") : object "kube-system"/"original-pull-secret" not registered Sep 16 07:27:50.105416 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:50.105382 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/296c5dcf-403c-4a1f-9d6f-05ffb7d75e38-metrics-certs\") pod \"network-metrics-daemon-kqc7r\" (UID: \"296c5dcf-403c-4a1f-9d6f-05ffb7d75e38\") " pod="openshift-multus/network-metrics-daemon-kqc7r" Sep 16 07:27:50.105620 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:27:50.105596 2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 07:27:50.105689 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:27:50.105678 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/296c5dcf-403c-4a1f-9d6f-05ffb7d75e38-metrics-certs podName:296c5dcf-403c-4a1f-9d6f-05ffb7d75e38 nodeName:}" failed. No retries permitted until 2026-09-16 07:27:51.10566226 +0000 UTC m=+4.132563251 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/296c5dcf-403c-4a1f-9d6f-05ffb7d75e38-metrics-certs") pod "network-metrics-daemon-kqc7r" (UID: "296c5dcf-403c-4a1f-9d6f-05ffb7d75e38") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 07:27:50.206654 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:50.206615 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6r6nv\" (UniqueName: \"kubernetes.io/projected/bfde2ba8-6a5f-413f-9ce9-d9ce0af2c758-kube-api-access-6r6nv\") pod \"network-check-target-jtqz9\" (UID: \"bfde2ba8-6a5f-413f-9ce9-d9ce0af2c758\") " pod="openshift-network-diagnostics/network-check-target-jtqz9" Sep 16 07:27:50.206841 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:27:50.206804 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 07:27:50.206841 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:27:50.206833 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 07:27:50.206971 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:27:50.206847 2557 projected.go:196] Error preparing data for projected volume kube-api-access-6r6nv for pod openshift-network-diagnostics/network-check-target-jtqz9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 07:27:50.206971 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:27:50.206919 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bfde2ba8-6a5f-413f-9ce9-d9ce0af2c758-kube-api-access-6r6nv podName:bfde2ba8-6a5f-413f-9ce9-d9ce0af2c758 nodeName:}" failed. No retries permitted until 2026-09-16 07:27:51.206898131 +0000 UTC m=+4.233799150 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-6r6nv" (UniqueName: "kubernetes.io/projected/bfde2ba8-6a5f-413f-9ce9-d9ce0af2c758-kube-api-access-6r6nv") pod "network-check-target-jtqz9" (UID: "bfde2ba8-6a5f-413f-9ce9-d9ce0af2c758") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 07:27:50.444267 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:50.444086 2557 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 07:22:50 +0000 UTC" deadline="2028-06-08 23:49:32.260740978 +0000 UTC" Sep 16 07:27:50.444267 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:50.444154 2557 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15160h21m41.816591738s" Sep 16 07:27:50.519493 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:50.518730 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jtqz9" Sep 16 07:27:50.519493 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:27:50.518871 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jtqz9" podUID="bfde2ba8-6a5f-413f-9ce9-d9ce0af2c758" Sep 16 07:27:50.519493 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:50.519328 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kqc7r" Sep 16 07:27:50.519493 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:27:50.519444 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kqc7r" podUID="296c5dcf-403c-4a1f-9d6f-05ffb7d75e38" Sep 16 07:27:50.529306 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:50.529230 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vj2rd" event={"ID":"17757d39-4eae-4f8d-8c3a-679d3502d985","Type":"ContainerStarted","Data":"31213a7a953002c14942ba3b972c088370f7119deaa74476efd89a80f411e314"} Sep 16 07:27:50.531101 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:50.531073 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-lq72c" event={"ID":"1ae92448-74b5-4b28-b337-758f6884d09a","Type":"ContainerStarted","Data":"aad6991c629236a1bbc2f62e86dfd19332424c1b1fd652894fc4fdefa26bf994"} Sep 16 07:27:50.532752 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:50.532701 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vbxvc" event={"ID":"89f6995b-d178-4fe9-9593-ba8dc8f49351","Type":"ContainerStarted","Data":"5718737659f2fd9c3a0a28671c7535b8c3b08ec97c2fbd166f113102bc365d20"} Sep 16 07:27:50.534526 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:50.534494 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4ztgb" event={"ID":"8d5c17cc-f191-4354-9c73-ab7f25dd203b","Type":"ContainerStarted","Data":"3767314a11a9e48dea0cf14e0a3a0b36e3a4f71a55af26dffbd7f93d9712502f"} Sep 16 07:27:50.536501 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:50.536475 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qprhb" event={"ID":"5f7afb78-2955-49e0-adec-6e8aae7ae51a","Type":"ContainerStarted","Data":"90b3d0c1a3a71b8876bb83d09a12aadbd98f3cb3065a17b41c9819cb128857c9"} Sep 16 07:27:50.538376 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:50.538349 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4cb95" event={"ID":"280250dc-6c09-4766-822c-b8bdf97e4a49","Type":"ContainerStarted","Data":"df35861cce6104f0c0588ccdd7c61ebdeb8d984e8947db485272869bceed0f62"} Sep 16 07:27:50.540337 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:50.540308 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-8gnk6" event={"ID":"20f6d954-1104-49b9-ab10-32e0de4ef8e0","Type":"ContainerStarted","Data":"6af888f9f06f4e8ec43e3a245251d5e21129d3dbee79c44fcda288a072a79c0e"} Sep 16 07:27:50.542163 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:50.542139 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-fhnn7" event={"ID":"53ab9fb8-54fd-478d-9b2b-f69c8aac0d02","Type":"ContainerStarted","Data":"0faeec8e703907e7468ca94693d6c39c33281f8e7e7fda231219852e7a781d01"} Sep 16 07:27:50.543639 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:50.543614 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zjxc4" event={"ID":"eb688a0e-2bd2-4df2-b475-7eba4237e526","Type":"ContainerStarted","Data":"f9c21f7d89a6ad06a1cffd1f007ccb3a22ed9d55117af69279c7b3f644ef0c29"} Sep 16 07:27:51.015935 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:51.015312 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/90e5e23a-060b-4612-80f5-f9a600b6d476-original-pull-secret\") pod \"global-pull-secret-syncer-t8thj\" (UID: \"90e5e23a-060b-4612-80f5-f9a600b6d476\") " pod="kube-system/global-pull-secret-syncer-t8thj" Sep 16 07:27:51.015935 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:27:51.015513 2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 07:27:51.015935 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:27:51.015575 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90e5e23a-060b-4612-80f5-f9a600b6d476-original-pull-secret podName:90e5e23a-060b-4612-80f5-f9a600b6d476 nodeName:}" failed. No retries permitted until 2026-09-16 07:27:53.015556812 +0000 UTC m=+6.042457810 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/90e5e23a-060b-4612-80f5-f9a600b6d476-original-pull-secret") pod "global-pull-secret-syncer-t8thj" (UID: "90e5e23a-060b-4612-80f5-f9a600b6d476") : object "kube-system"/"original-pull-secret" not registered Sep 16 07:27:51.115923 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:51.115876 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/296c5dcf-403c-4a1f-9d6f-05ffb7d75e38-metrics-certs\") pod \"network-metrics-daemon-kqc7r\" (UID: \"296c5dcf-403c-4a1f-9d6f-05ffb7d75e38\") " pod="openshift-multus/network-metrics-daemon-kqc7r" Sep 16 07:27:51.116111 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:27:51.116059 2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 07:27:51.116185 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:27:51.116128 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/296c5dcf-403c-4a1f-9d6f-05ffb7d75e38-metrics-certs podName:296c5dcf-403c-4a1f-9d6f-05ffb7d75e38 nodeName:}" failed. No retries permitted until 2026-09-16 07:27:53.116108086 +0000 UTC m=+6.143009093 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/296c5dcf-403c-4a1f-9d6f-05ffb7d75e38-metrics-certs") pod "network-metrics-daemon-kqc7r" (UID: "296c5dcf-403c-4a1f-9d6f-05ffb7d75e38") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 07:27:51.216760 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:51.216721 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6r6nv\" (UniqueName: \"kubernetes.io/projected/bfde2ba8-6a5f-413f-9ce9-d9ce0af2c758-kube-api-access-6r6nv\") pod \"network-check-target-jtqz9\" (UID: \"bfde2ba8-6a5f-413f-9ce9-d9ce0af2c758\") " pod="openshift-network-diagnostics/network-check-target-jtqz9" Sep 16 07:27:51.216952 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:27:51.216904 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 07:27:51.216952 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:27:51.216926 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 07:27:51.216952 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:27:51.216940 2557 projected.go:196] Error preparing data for projected volume kube-api-access-6r6nv for pod openshift-network-diagnostics/network-check-target-jtqz9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 07:27:51.217128 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:27:51.217000 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bfde2ba8-6a5f-413f-9ce9-d9ce0af2c758-kube-api-access-6r6nv podName:bfde2ba8-6a5f-413f-9ce9-d9ce0af2c758 nodeName:}" failed. No retries permitted until 2026-09-16 07:27:53.21698024 +0000 UTC m=+6.243881256 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-6r6nv" (UniqueName: "kubernetes.io/projected/bfde2ba8-6a5f-413f-9ce9-d9ce0af2c758-kube-api-access-6r6nv") pod "network-check-target-jtqz9" (UID: "bfde2ba8-6a5f-413f-9ce9-d9ce0af2c758") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 07:27:51.444655 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:51.444561 2557 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 07:22:50 +0000 UTC" deadline="2028-03-18 07:05:43.16930537 +0000 UTC" Sep 16 07:27:51.445104 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:51.444608 2557 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13175h37m51.724702491s" Sep 16 07:27:51.520388 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:51.520359 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t8thj" Sep 16 07:27:51.520552 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:27:51.520509 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t8thj" podUID="90e5e23a-060b-4612-80f5-f9a600b6d476" Sep 16 07:27:52.519381 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:52.518728 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jtqz9" Sep 16 07:27:52.519381 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:52.518781 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kqc7r" Sep 16 07:27:52.519381 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:27:52.518872 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jtqz9" podUID="bfde2ba8-6a5f-413f-9ce9-d9ce0af2c758" Sep 16 07:27:52.519381 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:27:52.519338 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kqc7r" podUID="296c5dcf-403c-4a1f-9d6f-05ffb7d75e38" Sep 16 07:27:53.032850 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:53.032809 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/90e5e23a-060b-4612-80f5-f9a600b6d476-original-pull-secret\") pod \"global-pull-secret-syncer-t8thj\" (UID: \"90e5e23a-060b-4612-80f5-f9a600b6d476\") " pod="kube-system/global-pull-secret-syncer-t8thj" Sep 16 07:27:53.033035 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:27:53.032981 2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 07:27:53.033115 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:27:53.033051 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90e5e23a-060b-4612-80f5-f9a600b6d476-original-pull-secret podName:90e5e23a-060b-4612-80f5-f9a600b6d476 nodeName:}" failed. No retries permitted until 2026-09-16 07:27:57.033029747 +0000 UTC m=+10.059930755 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/90e5e23a-060b-4612-80f5-f9a600b6d476-original-pull-secret") pod "global-pull-secret-syncer-t8thj" (UID: "90e5e23a-060b-4612-80f5-f9a600b6d476") : object "kube-system"/"original-pull-secret" not registered Sep 16 07:27:53.133835 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:53.133797 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/296c5dcf-403c-4a1f-9d6f-05ffb7d75e38-metrics-certs\") pod \"network-metrics-daemon-kqc7r\" (UID: \"296c5dcf-403c-4a1f-9d6f-05ffb7d75e38\") " pod="openshift-multus/network-metrics-daemon-kqc7r" Sep 16 07:27:53.134018 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:27:53.133968 2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 07:27:53.134085 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:27:53.134045 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/296c5dcf-403c-4a1f-9d6f-05ffb7d75e38-metrics-certs podName:296c5dcf-403c-4a1f-9d6f-05ffb7d75e38 nodeName:}" failed. No retries permitted until 2026-09-16 07:27:57.134025264 +0000 UTC m=+10.160926279 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/296c5dcf-403c-4a1f-9d6f-05ffb7d75e38-metrics-certs") pod "network-metrics-daemon-kqc7r" (UID: "296c5dcf-403c-4a1f-9d6f-05ffb7d75e38") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 07:27:53.234783 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:53.234742 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6r6nv\" (UniqueName: \"kubernetes.io/projected/bfde2ba8-6a5f-413f-9ce9-d9ce0af2c758-kube-api-access-6r6nv\") pod \"network-check-target-jtqz9\" (UID: \"bfde2ba8-6a5f-413f-9ce9-d9ce0af2c758\") " pod="openshift-network-diagnostics/network-check-target-jtqz9" Sep 16 07:27:53.235031 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:27:53.234981 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 07:27:53.235031 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:27:53.235009 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 07:27:53.235031 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:27:53.235022 2557 projected.go:196] Error preparing data for projected volume kube-api-access-6r6nv for pod openshift-network-diagnostics/network-check-target-jtqz9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 07:27:53.235196 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:27:53.235082 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bfde2ba8-6a5f-413f-9ce9-d9ce0af2c758-kube-api-access-6r6nv podName:bfde2ba8-6a5f-413f-9ce9-d9ce0af2c758 nodeName:}" failed. No retries permitted until 2026-09-16 07:27:57.235064484 +0000 UTC m=+10.261965481 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-6r6nv" (UniqueName: "kubernetes.io/projected/bfde2ba8-6a5f-413f-9ce9-d9ce0af2c758-kube-api-access-6r6nv") pod "network-check-target-jtqz9" (UID: "bfde2ba8-6a5f-413f-9ce9-d9ce0af2c758") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 07:27:53.519533 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:53.519451 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t8thj" Sep 16 07:27:53.520002 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:27:53.519586 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t8thj" podUID="90e5e23a-060b-4612-80f5-f9a600b6d476" Sep 16 07:27:54.518851 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:54.518817 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jtqz9" Sep 16 07:27:54.519044 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:27:54.518957 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jtqz9" podUID="bfde2ba8-6a5f-413f-9ce9-d9ce0af2c758" Sep 16 07:27:54.519571 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:54.519390 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kqc7r" Sep 16 07:27:54.519571 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:27:54.519501 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kqc7r" podUID="296c5dcf-403c-4a1f-9d6f-05ffb7d75e38" Sep 16 07:27:55.518946 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:55.518901 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t8thj" Sep 16 07:27:55.519168 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:27:55.519080 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t8thj" podUID="90e5e23a-060b-4612-80f5-f9a600b6d476" Sep 16 07:27:56.518843 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:56.518801 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jtqz9" Sep 16 07:27:56.519415 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:56.518801 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kqc7r" Sep 16 07:27:56.519415 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:27:56.518925 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jtqz9" podUID="bfde2ba8-6a5f-413f-9ce9-d9ce0af2c758" Sep 16 07:27:56.519415 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:27:56.519028 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kqc7r" podUID="296c5dcf-403c-4a1f-9d6f-05ffb7d75e38" Sep 16 07:27:57.068251 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:57.068219 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/90e5e23a-060b-4612-80f5-f9a600b6d476-original-pull-secret\") pod \"global-pull-secret-syncer-t8thj\" (UID: \"90e5e23a-060b-4612-80f5-f9a600b6d476\") " pod="kube-system/global-pull-secret-syncer-t8thj" Sep 16 07:27:57.068438 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:27:57.068399 2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 07:27:57.068493 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:27:57.068477 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90e5e23a-060b-4612-80f5-f9a600b6d476-original-pull-secret podName:90e5e23a-060b-4612-80f5-f9a600b6d476 nodeName:}" failed. No retries permitted until 2026-09-16 07:28:05.06845436 +0000 UTC m=+18.095355353 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/90e5e23a-060b-4612-80f5-f9a600b6d476-original-pull-secret") pod "global-pull-secret-syncer-t8thj" (UID: "90e5e23a-060b-4612-80f5-f9a600b6d476") : object "kube-system"/"original-pull-secret" not registered Sep 16 07:27:57.169117 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:57.169079 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/296c5dcf-403c-4a1f-9d6f-05ffb7d75e38-metrics-certs\") pod \"network-metrics-daemon-kqc7r\" (UID: \"296c5dcf-403c-4a1f-9d6f-05ffb7d75e38\") " pod="openshift-multus/network-metrics-daemon-kqc7r" Sep 16 07:27:57.169335 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:27:57.169261 2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 07:27:57.169414 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:27:57.169353 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/296c5dcf-403c-4a1f-9d6f-05ffb7d75e38-metrics-certs podName:296c5dcf-403c-4a1f-9d6f-05ffb7d75e38 nodeName:}" failed. No retries permitted until 2026-09-16 07:28:05.169335871 +0000 UTC m=+18.196236881 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/296c5dcf-403c-4a1f-9d6f-05ffb7d75e38-metrics-certs") pod "network-metrics-daemon-kqc7r" (UID: "296c5dcf-403c-4a1f-9d6f-05ffb7d75e38") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 07:27:57.270255 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:57.270204 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6r6nv\" (UniqueName: \"kubernetes.io/projected/bfde2ba8-6a5f-413f-9ce9-d9ce0af2c758-kube-api-access-6r6nv\") pod \"network-check-target-jtqz9\" (UID: \"bfde2ba8-6a5f-413f-9ce9-d9ce0af2c758\") " pod="openshift-network-diagnostics/network-check-target-jtqz9" Sep 16 07:27:57.270504 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:27:57.270423 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 07:27:57.270504 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:27:57.270448 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 07:27:57.270504 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:27:57.270461 2557 projected.go:196] Error preparing data for projected volume kube-api-access-6r6nv for pod openshift-network-diagnostics/network-check-target-jtqz9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 07:27:57.270682 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:27:57.270518 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bfde2ba8-6a5f-413f-9ce9-d9ce0af2c758-kube-api-access-6r6nv podName:bfde2ba8-6a5f-413f-9ce9-d9ce0af2c758 nodeName:}" failed. No retries permitted until 2026-09-16 07:28:05.270500377 +0000 UTC m=+18.297401377 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-6r6nv" (UniqueName: "kubernetes.io/projected/bfde2ba8-6a5f-413f-9ce9-d9ce0af2c758-kube-api-access-6r6nv") pod "network-check-target-jtqz9" (UID: "bfde2ba8-6a5f-413f-9ce9-d9ce0af2c758") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 07:27:57.519514 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:57.519421 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t8thj" Sep 16 07:27:57.519942 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:27:57.519553 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t8thj" podUID="90e5e23a-060b-4612-80f5-f9a600b6d476" Sep 16 07:27:58.519255 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:58.519207 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jtqz9" Sep 16 07:27:58.519448 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:27:58.519371 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jtqz9" podUID="bfde2ba8-6a5f-413f-9ce9-d9ce0af2c758" Sep 16 07:27:58.519448 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:58.519422 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kqc7r" Sep 16 07:27:58.519576 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:27:58.519553 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kqc7r" podUID="296c5dcf-403c-4a1f-9d6f-05ffb7d75e38" Sep 16 07:27:59.518449 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:27:59.518415 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t8thj" Sep 16 07:27:59.518638 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:27:59.518567 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t8thj" podUID="90e5e23a-060b-4612-80f5-f9a600b6d476" Sep 16 07:28:00.518740 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:00.518706 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jtqz9" Sep 16 07:28:00.519091 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:28:00.518806 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jtqz9" podUID="bfde2ba8-6a5f-413f-9ce9-d9ce0af2c758" Sep 16 07:28:00.519091 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:00.518860 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kqc7r" Sep 16 07:28:00.519091 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:28:00.518956 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kqc7r" podUID="296c5dcf-403c-4a1f-9d6f-05ffb7d75e38" Sep 16 07:28:01.519203 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:01.519120 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t8thj" Sep 16 07:28:01.519659 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:28:01.519266 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t8thj" podUID="90e5e23a-060b-4612-80f5-f9a600b6d476" Sep 16 07:28:02.518870 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:02.518832 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jtqz9" Sep 16 07:28:02.519035 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:02.518847 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kqc7r" Sep 16 07:28:02.519035 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:28:02.518960 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jtqz9" podUID="bfde2ba8-6a5f-413f-9ce9-d9ce0af2c758" Sep 16 07:28:02.519126 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:28:02.519046 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kqc7r" podUID="296c5dcf-403c-4a1f-9d6f-05ffb7d75e38" Sep 16 07:28:03.518742 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:03.518667 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t8thj" Sep 16 07:28:03.519323 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:28:03.518793 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t8thj" podUID="90e5e23a-060b-4612-80f5-f9a600b6d476" Sep 16 07:28:04.519145 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:04.519112 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jtqz9" Sep 16 07:28:04.519569 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:28:04.519212 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jtqz9" podUID="bfde2ba8-6a5f-413f-9ce9-d9ce0af2c758" Sep 16 07:28:04.519569 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:04.519268 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kqc7r" Sep 16 07:28:04.519569 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:28:04.519359 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kqc7r" podUID="296c5dcf-403c-4a1f-9d6f-05ffb7d75e38" Sep 16 07:28:05.133843 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:05.133810 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/90e5e23a-060b-4612-80f5-f9a600b6d476-original-pull-secret\") pod \"global-pull-secret-syncer-t8thj\" (UID: \"90e5e23a-060b-4612-80f5-f9a600b6d476\") " pod="kube-system/global-pull-secret-syncer-t8thj" Sep 16 07:28:05.134031 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:28:05.133969 2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 07:28:05.134096 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:28:05.134044 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90e5e23a-060b-4612-80f5-f9a600b6d476-original-pull-secret podName:90e5e23a-060b-4612-80f5-f9a600b6d476 nodeName:}" failed. No retries permitted until 2026-09-16 07:28:21.134026316 +0000 UTC m=+34.160927309 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/90e5e23a-060b-4612-80f5-f9a600b6d476-original-pull-secret") pod "global-pull-secret-syncer-t8thj" (UID: "90e5e23a-060b-4612-80f5-f9a600b6d476") : object "kube-system"/"original-pull-secret" not registered Sep 16 07:28:05.234483 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:05.234455 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/296c5dcf-403c-4a1f-9d6f-05ffb7d75e38-metrics-certs\") pod \"network-metrics-daemon-kqc7r\" (UID: \"296c5dcf-403c-4a1f-9d6f-05ffb7d75e38\") " pod="openshift-multus/network-metrics-daemon-kqc7r" Sep 16 07:28:05.234631 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:28:05.234579 2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 07:28:05.234677 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:28:05.234645 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/296c5dcf-403c-4a1f-9d6f-05ffb7d75e38-metrics-certs podName:296c5dcf-403c-4a1f-9d6f-05ffb7d75e38 nodeName:}" failed. No retries permitted until 2026-09-16 07:28:21.234626784 +0000 UTC m=+34.261527779 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/296c5dcf-403c-4a1f-9d6f-05ffb7d75e38-metrics-certs") pod "network-metrics-daemon-kqc7r" (UID: "296c5dcf-403c-4a1f-9d6f-05ffb7d75e38") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 07:28:05.335137 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:05.335067 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6r6nv\" (UniqueName: \"kubernetes.io/projected/bfde2ba8-6a5f-413f-9ce9-d9ce0af2c758-kube-api-access-6r6nv\") pod \"network-check-target-jtqz9\" (UID: \"bfde2ba8-6a5f-413f-9ce9-d9ce0af2c758\") " pod="openshift-network-diagnostics/network-check-target-jtqz9" Sep 16 07:28:05.335302 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:28:05.335267 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 07:28:05.335302 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:28:05.335293 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 07:28:05.335392 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:28:05.335307 2557 projected.go:196] Error preparing data for projected volume kube-api-access-6r6nv for pod openshift-network-diagnostics/network-check-target-jtqz9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 07:28:05.335392 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:28:05.335370 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bfde2ba8-6a5f-413f-9ce9-d9ce0af2c758-kube-api-access-6r6nv podName:bfde2ba8-6a5f-413f-9ce9-d9ce0af2c758 nodeName:}" failed. No retries permitted until 2026-09-16 07:28:21.335351446 +0000 UTC m=+34.362252505 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-6r6nv" (UniqueName: "kubernetes.io/projected/bfde2ba8-6a5f-413f-9ce9-d9ce0af2c758-kube-api-access-6r6nv") pod "network-check-target-jtqz9" (UID: "bfde2ba8-6a5f-413f-9ce9-d9ce0af2c758") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 07:28:05.519414 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:05.519338 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t8thj" Sep 16 07:28:05.519801 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:28:05.519466 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t8thj" podUID="90e5e23a-060b-4612-80f5-f9a600b6d476" Sep 16 07:28:06.518562 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:06.518530 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jtqz9" Sep 16 07:28:06.518721 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:06.518530 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kqc7r" Sep 16 07:28:06.518721 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:28:06.518652 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jtqz9" podUID="bfde2ba8-6a5f-413f-9ce9-d9ce0af2c758" Sep 16 07:28:06.518825 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:28:06.518730 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kqc7r" podUID="296c5dcf-403c-4a1f-9d6f-05ffb7d75e38" Sep 16 07:28:07.519330 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:07.519298 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t8thj" Sep 16 07:28:07.519782 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:28:07.519408 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t8thj" podUID="90e5e23a-060b-4612-80f5-f9a600b6d476" Sep 16 07:28:08.518665 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:08.518428 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jtqz9" Sep 16 07:28:08.518665 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:08.518449 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kqc7r" Sep 16 07:28:08.518665 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:28:08.518541 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jtqz9" podUID="bfde2ba8-6a5f-413f-9ce9-d9ce0af2c758" Sep 16 07:28:08.518665 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:28:08.518636 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kqc7r" podUID="296c5dcf-403c-4a1f-9d6f-05ffb7d75e38" Sep 16 07:28:09.519719 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:09.519331 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t8thj" Sep 16 07:28:09.520487 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:28:09.519801 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t8thj" podUID="90e5e23a-060b-4612-80f5-f9a600b6d476" Sep 16 07:28:09.587647 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:09.587616 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zjxc4" event={"ID":"eb688a0e-2bd2-4df2-b475-7eba4237e526","Type":"ContainerStarted","Data":"fad9f01136fd03492fd774d4047ecd0163b39aa9ee970bb1332e5e2aad2b5939"} Sep 16 07:28:09.589279 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:09.589255 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vj2rd" event={"ID":"17757d39-4eae-4f8d-8c3a-679d3502d985","Type":"ContainerStarted","Data":"9c2723cda5a1508878675aa03a3d64a5655d206e6dc38cf38505a339f4732688"} Sep 16 07:28:09.590665 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:09.590641 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-lq72c" event={"ID":"1ae92448-74b5-4b28-b337-758f6884d09a","Type":"ContainerStarted","Data":"0a03d23b45a0e40f886fd84ea2cfc8e59d5258bc5fb014e78ee882f4b6efcab0"} Sep 16 07:28:09.591974 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:09.591956 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vbxvc" event={"ID":"89f6995b-d178-4fe9-9593-ba8dc8f49351","Type":"ContainerStarted","Data":"6902bd658c00df3129feeaf4c2dba787f4031e6606ac601b738ddb28f59a6d16"} Sep 16 07:28:09.594913 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:09.594894 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4ztgb" event={"ID":"8d5c17cc-f191-4354-9c73-ab7f25dd203b","Type":"ContainerStarted","Data":"9392e066eb48a40ae3bbc2c5827b2f7c574268a269b574b02c6138afe8253c17"} Sep 16 07:28:09.595014 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:09.594953 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4ztgb" event={"ID":"8d5c17cc-f191-4354-9c73-ab7f25dd203b","Type":"ContainerStarted","Data":"a0b088717fc0c78d462cb374d0ade5b54bb993e00979d973fc76c50d78e27eac"} Sep 16 07:28:09.595014 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:09.595001 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4ztgb" event={"ID":"8d5c17cc-f191-4354-9c73-ab7f25dd203b","Type":"ContainerStarted","Data":"35ad22d87b81f6a060e885966056d32ad7851dbca2fcab0793d637e6b2f8f440"} Sep 16 07:28:09.595114 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:09.595050 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4ztgb" event={"ID":"8d5c17cc-f191-4354-9c73-ab7f25dd203b","Type":"ContainerStarted","Data":"831f7b7ecd4a07d35176286d19ee23794a31a4658d2abf282f3e7b3a41d0c855"} Sep 16 07:28:09.595114 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:09.595099 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4ztgb" event={"ID":"8d5c17cc-f191-4354-9c73-ab7f25dd203b","Type":"ContainerStarted","Data":"7a4950f3bbd25670b79fef97b8687639e646f4d7e1bcbea6b830923d6087bbd6"} Sep 16 07:28:09.595196 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:09.595140 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4ztgb" event={"ID":"8d5c17cc-f191-4354-9c73-ab7f25dd203b","Type":"ContainerStarted","Data":"2e37005fd2c890829ce02171a27d3a01deb768918537ba6328b952fdede01c1a"} Sep 16 07:28:09.596335 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:09.596317 2557 generic.go:356] "Generic (PLEG): container finished" podID="5f7afb78-2955-49e0-adec-6e8aae7ae51a" containerID="1b7b8e03eb2dfa05186e7279b7c4a8bb5acee3d9fe68fdff8248d88dbd255590" exitCode=0 Sep 16 07:28:09.596412 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:09.596336 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qprhb" event={"ID":"5f7afb78-2955-49e0-adec-6e8aae7ae51a","Type":"ContainerDied","Data":"1b7b8e03eb2dfa05186e7279b7c4a8bb5acee3d9fe68fdff8248d88dbd255590"} Sep 16 07:28:09.597872 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:09.597753 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4cb95" event={"ID":"280250dc-6c09-4766-822c-b8bdf97e4a49","Type":"ContainerStarted","Data":"8d58be4b2dd25c85dc2b998a0576ea9ed659c00db0aa0800c0c8931acb1345e2"} Sep 16 07:28:09.599217 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:09.599199 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-8gnk6" event={"ID":"20f6d954-1104-49b9-ab10-32e0de4ef8e0","Type":"ContainerStarted","Data":"872375a9642853b25c9e4916440341e039c36b199ab2628305b21ae8737a968d"} Sep 16 07:28:09.600583 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:09.600561 2557 generic.go:356] "Generic (PLEG): container finished" podID="8ac46b15b636b4c144bce2b4a1b3535a" containerID="3fa5c472312fe5f89ffb924d6fb387f4664fb99d11066e72447198b51d3adb96" exitCode=0 Sep 16 07:28:09.600669 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:09.600637 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-221.ec2.internal" event={"ID":"8ac46b15b636b4c144bce2b4a1b3535a","Type":"ContainerDied","Data":"3fa5c472312fe5f89ffb924d6fb387f4664fb99d11066e72447198b51d3adb96"} Sep 16 07:28:09.602147 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:09.602128 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-221.ec2.internal" event={"ID":"3825bcd696c373ea98c5e3521a26b861","Type":"ContainerStarted","Data":"cdb69a40a90030257c6757e8abc3637d7dbafec0266016ba6d35601618358e06"} Sep 16 07:28:09.616892 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:09.616846 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-zjxc4" podStartSLOduration=2.821362488 podStartE2EDuration="21.61682765s" totalImagesPullingTime="18.795465162s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 07:27:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:28:09.604812963 +0000 UTC m=+22.631714003" watchObservedRunningTime="2026-09-16 07:28:09.61682765 +0000 UTC m=+22.643728663" Sep 16 07:28:09.617291 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:09.617258 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-221.ec2.internal" podStartSLOduration=20.617229551 podStartE2EDuration="20.617229551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 07:27:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:28:09.617048975 +0000 UTC m=+22.643949988" watchObservedRunningTime="2026-09-16 07:28:09.617229551 +0000 UTC m=+22.644130565" Sep 16 07:28:09.631214 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:09.631179 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-8gnk6" podStartSLOduration=2.8092848889999997 podStartE2EDuration="21.631169561s" totalImagesPullingTime="18.821884672s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 07:27:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:28:09.630713107 +0000 UTC m=+22.657614139" watchObservedRunningTime="2026-09-16 07:28:09.631169561 +0000 UTC m=+22.658070572" Sep 16 07:28:09.646244 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:09.646202 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-4cb95" podStartSLOduration=2.780312322 podStartE2EDuration="21.646191487s" totalImagesPullingTime="18.865879165s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 07:27:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:28:09.645646383 +0000 UTC m=+22.672547417" watchObservedRunningTime="2026-09-16 07:28:09.646191487 +0000 UTC m=+22.673092499" Sep 16 07:28:09.659866 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:09.659826 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-lq72c" podStartSLOduration=3.145647687 podStartE2EDuration="21.659810143s" totalImagesPullingTime="18.514162456s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:27:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:28:09.659132139 +0000 UTC m=+22.686033172" watchObservedRunningTime="2026-09-16 07:28:09.659810143 +0000 UTC m=+22.686711157" Sep 16 07:28:09.708730 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:09.708685 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-vbxvc" podStartSLOduration=3.239960926 podStartE2EDuration="21.708671711s" totalImagesPullingTime="18.468710785s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 07:27:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:28:09.708555195 +0000 UTC m=+22.735456234" watchObservedRunningTime="2026-09-16 07:28:09.708671711 +0000 UTC m=+22.735572724" Sep 16 07:28:09.726709 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:09.726686 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-lq72c" Sep 16 07:28:09.727292 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:09.727277 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-lq72c" Sep 16 07:28:09.927304 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:09.927229 2557 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 16 07:28:10.454931 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:10.454838 2557 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-16T07:28:09.927261214Z","UUID":"8e129fb0-8509-4d0b-abb6-991aa69497e0","Handler":null,"Name":"","Endpoint":""} Sep 16 07:28:10.456757 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:10.456612 2557 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 16 07:28:10.456757 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:10.456647 2557 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 16 07:28:10.518943 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:10.518916 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jtqz9" Sep 16 07:28:10.519081 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:28:10.519032 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jtqz9" podUID="bfde2ba8-6a5f-413f-9ce9-d9ce0af2c758" Sep 16 07:28:10.519081 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:10.519088 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kqc7r" Sep 16 07:28:10.519296 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:28:10.519178 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kqc7r" podUID="296c5dcf-403c-4a1f-9d6f-05ffb7d75e38" Sep 16 07:28:10.607224 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:10.607167 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vj2rd" event={"ID":"17757d39-4eae-4f8d-8c3a-679d3502d985","Type":"ContainerStarted","Data":"dacd593dcc600c6276f813bad510e07ea15ddf8a8966b61efab6e20c2d7cb076"} Sep 16 07:28:10.609379 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:10.609353 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-221.ec2.internal" event={"ID":"8ac46b15b636b4c144bce2b4a1b3535a","Type":"ContainerStarted","Data":"24cdd2e59c4fa89645879c7b48616e6faf10c4836947a8079fda8d8c49f60930"} Sep 16 07:28:10.611330 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:10.611306 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-fhnn7" event={"ID":"53ab9fb8-54fd-478d-9b2b-f69c8aac0d02","Type":"ContainerStarted","Data":"00a9a6b25f9cd50d71e8cadb732b69124144b1fc2df0a02a87ca066c177527b5"} Sep 16 07:28:10.612387 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:10.612266 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-lq72c" Sep 16 07:28:10.612745 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:10.612724 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-lq72c" Sep 16 07:28:10.625173 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:10.625126 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-221.ec2.internal" podStartSLOduration=21.625110161 podStartE2EDuration="21.625110161s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 07:27:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:28:10.624615279 +0000 UTC m=+23.651516314" watchObservedRunningTime="2026-09-16 07:28:10.625110161 +0000 UTC m=+23.652011176" Sep 16 07:28:10.650670 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:10.650625 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-fhnn7" podStartSLOduration=3.900222657 podStartE2EDuration="22.650613337s" totalImagesPullingTime="18.75039068s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 07:27:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:28:10.650393296 +0000 UTC m=+23.677294334" watchObservedRunningTime="2026-09-16 07:28:10.650613337 +0000 UTC m=+23.677514354" Sep 16 07:28:11.518674 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:11.518450 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t8thj" Sep 16 07:28:11.518804 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:28:11.518705 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t8thj" podUID="90e5e23a-060b-4612-80f5-f9a600b6d476" Sep 16 07:28:11.615462 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:11.615427 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vj2rd" event={"ID":"17757d39-4eae-4f8d-8c3a-679d3502d985","Type":"ContainerStarted","Data":"ac6a49a796acbf133ceba87d090ad65e94aa99fc458d0c8f8c22cf3e09168938"} Sep 16 07:28:11.622280 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:11.622252 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4ztgb" event={"ID":"8d5c17cc-f191-4354-9c73-ab7f25dd203b","Type":"ContainerStarted","Data":"7a0954f56aa5bb6e3c88276ed5cf3ae2ca608fedc7c0fb73b363da355fbe5b15"} Sep 16 07:28:11.635845 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:11.635806 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vj2rd" podStartSLOduration=3.123459826 podStartE2EDuration="23.635792556s" totalImagesPullingTime="20.51233273s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 07:27:48 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:28:11.635100571 +0000 UTC m=+24.662001604" watchObservedRunningTime="2026-09-16 07:28:11.635792556 +0000 UTC m=+24.662693550" Sep 16 07:28:12.518821 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:12.518790 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kqc7r" Sep 16 07:28:12.519023 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:28:12.518907 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kqc7r" podUID="296c5dcf-403c-4a1f-9d6f-05ffb7d75e38" Sep 16 07:28:12.519023 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:12.518961 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jtqz9" Sep 16 07:28:12.519143 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:28:12.519061 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jtqz9" podUID="bfde2ba8-6a5f-413f-9ce9-d9ce0af2c758" Sep 16 07:28:13.518600 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:13.518529 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t8thj" Sep 16 07:28:13.519035 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:28:13.518654 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t8thj" podUID="90e5e23a-060b-4612-80f5-f9a600b6d476" Sep 16 07:28:14.518848 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:14.518822 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jtqz9" Sep 16 07:28:14.519213 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:14.518822 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kqc7r" Sep 16 07:28:14.519213 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:28:14.518919 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jtqz9" podUID="bfde2ba8-6a5f-413f-9ce9-d9ce0af2c758" Sep 16 07:28:14.519213 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:28:14.518994 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kqc7r" podUID="296c5dcf-403c-4a1f-9d6f-05ffb7d75e38" Sep 16 07:28:15.519422 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:15.519160 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t8thj" Sep 16 07:28:15.519899 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:28:15.519477 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t8thj" podUID="90e5e23a-060b-4612-80f5-f9a600b6d476" Sep 16 07:28:15.634425 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:15.634399 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4ztgb" event={"ID":"8d5c17cc-f191-4354-9c73-ab7f25dd203b","Type":"ContainerStarted","Data":"4634d4bce74f68a59a49aeb341acb59d2ce2fe0771f08e4ba7c5af9e647c44bf"} Sep 16 07:28:15.634661 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:15.634641 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4ztgb" Sep 16 07:28:15.636403 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:15.636381 2557 generic.go:356] "Generic (PLEG): container finished" podID="5f7afb78-2955-49e0-adec-6e8aae7ae51a" containerID="85448d4090fa68e7872993a325eb66d9dae8c14212d5956ba850e012659c5e2a" exitCode=0 Sep 16 07:28:15.636500 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:15.636423 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qprhb" event={"ID":"5f7afb78-2955-49e0-adec-6e8aae7ae51a","Type":"ContainerDied","Data":"85448d4090fa68e7872993a325eb66d9dae8c14212d5956ba850e012659c5e2a"} Sep 16 07:28:15.652218 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:15.650148 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4ztgb" Sep 16 07:28:15.660651 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:15.660616 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-4ztgb" podStartSLOduration=8.86942862 podStartE2EDuration="27.660606338s" totalImagesPullingTime="18.791177718s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 07:27:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:28:15.660403056 +0000 UTC m=+28.687304091" watchObservedRunningTime="2026-09-16 07:28:15.660606338 +0000 UTC m=+28.687507350" Sep 16 07:28:16.518537 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:16.518508 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jtqz9" Sep 16 07:28:16.518686 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:16.518519 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kqc7r" Sep 16 07:28:16.518686 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:28:16.518608 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jtqz9" podUID="bfde2ba8-6a5f-413f-9ce9-d9ce0af2c758" Sep 16 07:28:16.518767 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:28:16.518686 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kqc7r" podUID="296c5dcf-403c-4a1f-9d6f-05ffb7d75e38" Sep 16 07:28:16.570168 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:16.570141 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-jtqz9"] Sep 16 07:28:16.570977 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:16.570960 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-t8thj"] Sep 16 07:28:16.571059 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:16.571046 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t8thj" Sep 16 07:28:16.571144 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:28:16.571126 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t8thj" podUID="90e5e23a-060b-4612-80f5-f9a600b6d476" Sep 16 07:28:16.573627 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:16.573608 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-kqc7r"] Sep 16 07:28:16.638523 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:16.638504 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kqc7r" Sep 16 07:28:16.638523 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:16.638519 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jtqz9" Sep 16 07:28:16.638675 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:28:16.638598 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kqc7r" podUID="296c5dcf-403c-4a1f-9d6f-05ffb7d75e38" Sep 16 07:28:16.638981 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:28:16.638953 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jtqz9" podUID="bfde2ba8-6a5f-413f-9ce9-d9ce0af2c758" Sep 16 07:28:16.639054 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:16.638978 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4ztgb" Sep 16 07:28:16.639054 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:16.639036 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4ztgb" Sep 16 07:28:16.653359 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:16.653343 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4ztgb" Sep 16 07:28:17.642638 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:17.642474 2557 generic.go:356] "Generic (PLEG): container finished" podID="5f7afb78-2955-49e0-adec-6e8aae7ae51a" containerID="ea8bdb8c11eadfabd3e692611139d7be1878b054b52f2d2183778b1824252143" exitCode=0 Sep 16 07:28:17.643006 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:17.642557 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qprhb" event={"ID":"5f7afb78-2955-49e0-adec-6e8aae7ae51a","Type":"ContainerDied","Data":"ea8bdb8c11eadfabd3e692611139d7be1878b054b52f2d2183778b1824252143"} Sep 16 07:28:18.518922 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:18.518893 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t8thj" Sep 16 07:28:18.519089 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:18.518938 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jtqz9" Sep 16 07:28:18.519089 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:28:18.519006 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jtqz9" podUID="bfde2ba8-6a5f-413f-9ce9-d9ce0af2c758" Sep 16 07:28:18.519089 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:18.519009 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kqc7r" Sep 16 07:28:18.519089 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:28:18.519068 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t8thj" podUID="90e5e23a-060b-4612-80f5-f9a600b6d476" Sep 16 07:28:18.519293 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:28:18.519144 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kqc7r" podUID="296c5dcf-403c-4a1f-9d6f-05ffb7d75e38" Sep 16 07:28:19.649715 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:19.649679 2557 generic.go:356] "Generic (PLEG): container finished" podID="5f7afb78-2955-49e0-adec-6e8aae7ae51a" containerID="d814d1e4dc99ac11f07326a02e60760e1ae59fea2621b617a7637f4da454958c" exitCode=0 Sep 16 07:28:19.650109 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:19.649728 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qprhb" event={"ID":"5f7afb78-2955-49e0-adec-6e8aae7ae51a","Type":"ContainerDied","Data":"d814d1e4dc99ac11f07326a02e60760e1ae59fea2621b617a7637f4da454958c"} Sep 16 07:28:20.519391 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:20.519357 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t8thj" Sep 16 07:28:20.519556 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:20.519358 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kqc7r" Sep 16 07:28:20.519556 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:28:20.519485 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t8thj" podUID="90e5e23a-060b-4612-80f5-f9a600b6d476" Sep 16 07:28:20.519645 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:20.519360 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jtqz9" Sep 16 07:28:20.519689 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:28:20.519578 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kqc7r" podUID="296c5dcf-403c-4a1f-9d6f-05ffb7d75e38" Sep 16 07:28:20.519689 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:28:20.519671 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jtqz9" podUID="bfde2ba8-6a5f-413f-9ce9-d9ce0af2c758" Sep 16 07:28:21.146213 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:21.146176 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/90e5e23a-060b-4612-80f5-f9a600b6d476-original-pull-secret\") pod \"global-pull-secret-syncer-t8thj\" (UID: \"90e5e23a-060b-4612-80f5-f9a600b6d476\") " pod="kube-system/global-pull-secret-syncer-t8thj" Sep 16 07:28:21.146659 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:28:21.146302 2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 07:28:21.146659 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:28:21.146356 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90e5e23a-060b-4612-80f5-f9a600b6d476-original-pull-secret podName:90e5e23a-060b-4612-80f5-f9a600b6d476 nodeName:}" failed. No retries permitted until 2026-09-16 07:28:53.146342594 +0000 UTC m=+66.173243587 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/90e5e23a-060b-4612-80f5-f9a600b6d476-original-pull-secret") pod "global-pull-secret-syncer-t8thj" (UID: "90e5e23a-060b-4612-80f5-f9a600b6d476") : object "kube-system"/"original-pull-secret" not registered Sep 16 07:28:21.235101 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:21.235072 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-221.ec2.internal" event="NodeReady" Sep 16 07:28:21.235285 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:21.235228 2557 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 16 07:28:21.247446 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:21.247413 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/296c5dcf-403c-4a1f-9d6f-05ffb7d75e38-metrics-certs\") pod \"network-metrics-daemon-kqc7r\" (UID: \"296c5dcf-403c-4a1f-9d6f-05ffb7d75e38\") " pod="openshift-multus/network-metrics-daemon-kqc7r" Sep 16 07:28:21.247599 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:28:21.247567 2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 07:28:21.247658 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:28:21.247625 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/296c5dcf-403c-4a1f-9d6f-05ffb7d75e38-metrics-certs podName:296c5dcf-403c-4a1f-9d6f-05ffb7d75e38 nodeName:}" failed. No retries permitted until 2026-09-16 07:28:53.247607853 +0000 UTC m=+66.274508859 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/296c5dcf-403c-4a1f-9d6f-05ffb7d75e38-metrics-certs") pod "network-metrics-daemon-kqc7r" (UID: "296c5dcf-403c-4a1f-9d6f-05ffb7d75e38") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 07:28:21.276593 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:21.276555 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-v8n9m"] Sep 16 07:28:21.307708 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:21.307671 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-lprn9"] Sep 16 07:28:21.307876 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:21.307818 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-v8n9m" Sep 16 07:28:21.310071 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:21.310043 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 16 07:28:21.310215 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:21.310135 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-8grdh\"" Sep 16 07:28:21.310215 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:21.310184 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 16 07:28:21.337849 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:21.337774 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-v8n9m"] Sep 16 07:28:21.337849 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:21.337819 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-lprn9"] Sep 16 07:28:21.338033 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:21.337918 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-lprn9" Sep 16 07:28:21.340552 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:21.340527 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 16 07:28:21.341099 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:21.340678 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 16 07:28:21.341099 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:21.340908 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 16 07:28:21.341099 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:21.340936 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-jmsns\"" Sep 16 07:28:21.348253 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:21.348221 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6r6nv\" (UniqueName: \"kubernetes.io/projected/bfde2ba8-6a5f-413f-9ce9-d9ce0af2c758-kube-api-access-6r6nv\") pod \"network-check-target-jtqz9\" (UID: \"bfde2ba8-6a5f-413f-9ce9-d9ce0af2c758\") " pod="openshift-network-diagnostics/network-check-target-jtqz9" Sep 16 07:28:21.348367 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:28:21.348349 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 07:28:21.348413 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:28:21.348373 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 07:28:21.348413 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:28:21.348383 2557 projected.go:196] Error preparing data for projected volume kube-api-access-6r6nv for pod openshift-network-diagnostics/network-check-target-jtqz9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 07:28:21.348478 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:28:21.348430 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bfde2ba8-6a5f-413f-9ce9-d9ce0af2c758-kube-api-access-6r6nv podName:bfde2ba8-6a5f-413f-9ce9-d9ce0af2c758 nodeName:}" failed. No retries permitted until 2026-09-16 07:28:53.348416342 +0000 UTC m=+66.375317336 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-6r6nv" (UniqueName: "kubernetes.io/projected/bfde2ba8-6a5f-413f-9ce9-d9ce0af2c758-kube-api-access-6r6nv") pod "network-check-target-jtqz9" (UID: "bfde2ba8-6a5f-413f-9ce9-d9ce0af2c758") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 07:28:21.448849 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:21.448808 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dbc6f54b-f7e7-4ede-b9a2-58f81e7a3f4c-metrics-tls\") pod \"dns-default-v8n9m\" (UID: \"dbc6f54b-f7e7-4ede-b9a2-58f81e7a3f4c\") " pod="openshift-dns/dns-default-v8n9m" Sep 16 07:28:21.448849 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:21.448851 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7c9bb910-6eda-4d97-9d79-7d78fd9abb58-cert\") pod \"ingress-canary-lprn9\" (UID: \"7c9bb910-6eda-4d97-9d79-7d78fd9abb58\") " pod="openshift-ingress-canary/ingress-canary-lprn9" Sep 16 07:28:21.449062 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:21.448880 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/dbc6f54b-f7e7-4ede-b9a2-58f81e7a3f4c-config-volume\") pod \"dns-default-v8n9m\" (UID: \"dbc6f54b-f7e7-4ede-b9a2-58f81e7a3f4c\") " pod="openshift-dns/dns-default-v8n9m" Sep 16 07:28:21.449062 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:21.448958 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbwtv\" (UniqueName: \"kubernetes.io/projected/7c9bb910-6eda-4d97-9d79-7d78fd9abb58-kube-api-access-kbwtv\") pod \"ingress-canary-lprn9\" (UID: \"7c9bb910-6eda-4d97-9d79-7d78fd9abb58\") " pod="openshift-ingress-canary/ingress-canary-lprn9" Sep 16 07:28:21.449062 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:21.448983 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbc6f54b-f7e7-4ede-b9a2-58f81e7a3f4c-tmp-dir\") pod \"dns-default-v8n9m\" (UID: \"dbc6f54b-f7e7-4ede-b9a2-58f81e7a3f4c\") " pod="openshift-dns/dns-default-v8n9m" Sep 16 07:28:21.449062 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:21.448998 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdbrw\" (UniqueName: \"kubernetes.io/projected/dbc6f54b-f7e7-4ede-b9a2-58f81e7a3f4c-kube-api-access-kdbrw\") pod \"dns-default-v8n9m\" (UID: \"dbc6f54b-f7e7-4ede-b9a2-58f81e7a3f4c\") " pod="openshift-dns/dns-default-v8n9m" Sep 16 07:28:21.549376 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:21.549345 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbwtv\" (UniqueName: \"kubernetes.io/projected/7c9bb910-6eda-4d97-9d79-7d78fd9abb58-kube-api-access-kbwtv\") pod \"ingress-canary-lprn9\" (UID: \"7c9bb910-6eda-4d97-9d79-7d78fd9abb58\") " pod="openshift-ingress-canary/ingress-canary-lprn9" Sep 16 07:28:21.549376 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:21.549378 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbc6f54b-f7e7-4ede-b9a2-58f81e7a3f4c-tmp-dir\") pod \"dns-default-v8n9m\" (UID: \"dbc6f54b-f7e7-4ede-b9a2-58f81e7a3f4c\") " pod="openshift-dns/dns-default-v8n9m" Sep 16 07:28:21.549625 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:21.549395 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdbrw\" (UniqueName: \"kubernetes.io/projected/dbc6f54b-f7e7-4ede-b9a2-58f81e7a3f4c-kube-api-access-kdbrw\") pod \"dns-default-v8n9m\" (UID: \"dbc6f54b-f7e7-4ede-b9a2-58f81e7a3f4c\") " pod="openshift-dns/dns-default-v8n9m" Sep 16 07:28:21.549625 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:21.549438 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dbc6f54b-f7e7-4ede-b9a2-58f81e7a3f4c-metrics-tls\") pod \"dns-default-v8n9m\" (UID: \"dbc6f54b-f7e7-4ede-b9a2-58f81e7a3f4c\") " pod="openshift-dns/dns-default-v8n9m" Sep 16 07:28:21.549625 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:21.549467 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7c9bb910-6eda-4d97-9d79-7d78fd9abb58-cert\") pod \"ingress-canary-lprn9\" (UID: \"7c9bb910-6eda-4d97-9d79-7d78fd9abb58\") " pod="openshift-ingress-canary/ingress-canary-lprn9" Sep 16 07:28:21.549625 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:21.549499 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/dbc6f54b-f7e7-4ede-b9a2-58f81e7a3f4c-config-volume\") pod \"dns-default-v8n9m\" (UID: \"dbc6f54b-f7e7-4ede-b9a2-58f81e7a3f4c\") " pod="openshift-dns/dns-default-v8n9m" Sep 16 07:28:21.549625 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:28:21.549556 2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 07:28:21.549625 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:28:21.549627 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbc6f54b-f7e7-4ede-b9a2-58f81e7a3f4c-metrics-tls podName:dbc6f54b-f7e7-4ede-b9a2-58f81e7a3f4c nodeName:}" failed. No retries permitted until 2026-09-16 07:28:22.049607138 +0000 UTC m=+35.076508146 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/dbc6f54b-f7e7-4ede-b9a2-58f81e7a3f4c-metrics-tls") pod "dns-default-v8n9m" (UID: "dbc6f54b-f7e7-4ede-b9a2-58f81e7a3f4c") : secret "dns-default-metrics-tls" not found Sep 16 07:28:21.549899 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:21.549800 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbc6f54b-f7e7-4ede-b9a2-58f81e7a3f4c-tmp-dir\") pod \"dns-default-v8n9m\" (UID: \"dbc6f54b-f7e7-4ede-b9a2-58f81e7a3f4c\") " pod="openshift-dns/dns-default-v8n9m" Sep 16 07:28:21.550059 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:21.550040 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/dbc6f54b-f7e7-4ede-b9a2-58f81e7a3f4c-config-volume\") pod \"dns-default-v8n9m\" (UID: \"dbc6f54b-f7e7-4ede-b9a2-58f81e7a3f4c\") " pod="openshift-dns/dns-default-v8n9m" Sep 16 07:28:21.554291 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:21.554269 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7c9bb910-6eda-4d97-9d79-7d78fd9abb58-cert\") pod \"ingress-canary-lprn9\" (UID: \"7c9bb910-6eda-4d97-9d79-7d78fd9abb58\") " pod="openshift-ingress-canary/ingress-canary-lprn9" Sep 16 07:28:21.558641 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:21.558617 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbwtv\" (UniqueName: \"kubernetes.io/projected/7c9bb910-6eda-4d97-9d79-7d78fd9abb58-kube-api-access-kbwtv\") pod \"ingress-canary-lprn9\" (UID: \"7c9bb910-6eda-4d97-9d79-7d78fd9abb58\") " pod="openshift-ingress-canary/ingress-canary-lprn9" Sep 16 07:28:21.559043 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:21.559022 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdbrw\" (UniqueName: \"kubernetes.io/projected/dbc6f54b-f7e7-4ede-b9a2-58f81e7a3f4c-kube-api-access-kdbrw\") pod \"dns-default-v8n9m\" (UID: \"dbc6f54b-f7e7-4ede-b9a2-58f81e7a3f4c\") " pod="openshift-dns/dns-default-v8n9m" Sep 16 07:28:21.647886 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:21.647850 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-lprn9" Sep 16 07:28:21.979133 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:21.978817 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-lprn9"] Sep 16 07:28:21.985421 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:28:21.985386 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c9bb910_6eda_4d97_9d79_7d78fd9abb58.slice/crio-b9ff960f72d66353bfda5b870a653d821f6bec49039d55d24b9f106661335095 WatchSource:0}: Error finding container b9ff960f72d66353bfda5b870a653d821f6bec49039d55d24b9f106661335095: Status 404 returned error can't find the container with id b9ff960f72d66353bfda5b870a653d821f6bec49039d55d24b9f106661335095 Sep 16 07:28:22.053353 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:22.053315 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dbc6f54b-f7e7-4ede-b9a2-58f81e7a3f4c-metrics-tls\") pod \"dns-default-v8n9m\" (UID: \"dbc6f54b-f7e7-4ede-b9a2-58f81e7a3f4c\") " pod="openshift-dns/dns-default-v8n9m" Sep 16 07:28:22.053541 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:28:22.053478 2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 07:28:22.053608 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:28:22.053556 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbc6f54b-f7e7-4ede-b9a2-58f81e7a3f4c-metrics-tls podName:dbc6f54b-f7e7-4ede-b9a2-58f81e7a3f4c nodeName:}" failed. No retries permitted until 2026-09-16 07:28:23.053535075 +0000 UTC m=+36.080436082 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/dbc6f54b-f7e7-4ede-b9a2-58f81e7a3f4c-metrics-tls") pod "dns-default-v8n9m" (UID: "dbc6f54b-f7e7-4ede-b9a2-58f81e7a3f4c") : secret "dns-default-metrics-tls" not found Sep 16 07:28:22.518474 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:22.518423 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kqc7r" Sep 16 07:28:22.518932 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:22.518446 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jtqz9" Sep 16 07:28:22.518932 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:22.518450 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t8thj" Sep 16 07:28:22.521104 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:22.521082 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 16 07:28:22.521250 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:22.521130 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 16 07:28:22.521250 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:22.521226 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 16 07:28:22.522062 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:22.522042 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 16 07:28:22.522205 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:22.522188 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-wbnpf\"" Sep 16 07:28:22.522297 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:22.522258 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xjqjb\"" Sep 16 07:28:22.659351 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:22.659301 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-lprn9" event={"ID":"7c9bb910-6eda-4d97-9d79-7d78fd9abb58","Type":"ContainerStarted","Data":"b9ff960f72d66353bfda5b870a653d821f6bec49039d55d24b9f106661335095"} Sep 16 07:28:23.062233 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:23.062197 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dbc6f54b-f7e7-4ede-b9a2-58f81e7a3f4c-metrics-tls\") pod \"dns-default-v8n9m\" (UID: \"dbc6f54b-f7e7-4ede-b9a2-58f81e7a3f4c\") " pod="openshift-dns/dns-default-v8n9m" Sep 16 07:28:23.062432 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:28:23.062382 2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 07:28:23.062475 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:28:23.062466 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbc6f54b-f7e7-4ede-b9a2-58f81e7a3f4c-metrics-tls podName:dbc6f54b-f7e7-4ede-b9a2-58f81e7a3f4c nodeName:}" failed. No retries permitted until 2026-09-16 07:28:25.062444821 +0000 UTC m=+38.089345814 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/dbc6f54b-f7e7-4ede-b9a2-58f81e7a3f4c-metrics-tls") pod "dns-default-v8n9m" (UID: "dbc6f54b-f7e7-4ede-b9a2-58f81e7a3f4c") : secret "dns-default-metrics-tls" not found Sep 16 07:28:25.078746 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:25.078703 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dbc6f54b-f7e7-4ede-b9a2-58f81e7a3f4c-metrics-tls\") pod \"dns-default-v8n9m\" (UID: \"dbc6f54b-f7e7-4ede-b9a2-58f81e7a3f4c\") " pod="openshift-dns/dns-default-v8n9m" Sep 16 07:28:25.079202 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:28:25.078849 2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 07:28:25.079202 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:28:25.078906 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbc6f54b-f7e7-4ede-b9a2-58f81e7a3f4c-metrics-tls podName:dbc6f54b-f7e7-4ede-b9a2-58f81e7a3f4c nodeName:}" failed. No retries permitted until 2026-09-16 07:28:29.07889051 +0000 UTC m=+42.105791503 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/dbc6f54b-f7e7-4ede-b9a2-58f81e7a3f4c-metrics-tls") pod "dns-default-v8n9m" (UID: "dbc6f54b-f7e7-4ede-b9a2-58f81e7a3f4c") : secret "dns-default-metrics-tls" not found Sep 16 07:28:25.667994 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:25.667961 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-lprn9" event={"ID":"7c9bb910-6eda-4d97-9d79-7d78fd9abb58","Type":"ContainerStarted","Data":"d90efc1f075a64442b4b3aa0d37805c1f1b599071e9f93e61ef776315bab760b"} Sep 16 07:28:25.670739 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:25.670710 2557 generic.go:356] "Generic (PLEG): container finished" podID="5f7afb78-2955-49e0-adec-6e8aae7ae51a" containerID="f89b6500823a8c6d86b64975a023443c5b07ad5a3173a28525aa9772be46446b" exitCode=0 Sep 16 07:28:25.670879 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:25.670752 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qprhb" event={"ID":"5f7afb78-2955-49e0-adec-6e8aae7ae51a","Type":"ContainerDied","Data":"f89b6500823a8c6d86b64975a023443c5b07ad5a3173a28525aa9772be46446b"} Sep 16 07:28:25.683356 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:25.683271 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-lprn9" podStartSLOduration=1.5112479250000002 podStartE2EDuration="4.683251599s" totalImagesPullingTime="3.172003674s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:28:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:28:25.682766902 +0000 UTC m=+38.709667946" watchObservedRunningTime="2026-09-16 07:28:25.683251599 +0000 UTC m=+38.710152614" Sep 16 07:28:26.676452 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:26.676413 2557 generic.go:356] "Generic (PLEG): container finished" podID="5f7afb78-2955-49e0-adec-6e8aae7ae51a" containerID="56284651b1245bf3c0407e710310e3742cbaf8ce721a587b16a71d0356c32658" exitCode=0 Sep 16 07:28:26.677078 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:26.676500 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qprhb" event={"ID":"5f7afb78-2955-49e0-adec-6e8aae7ae51a","Type":"ContainerDied","Data":"56284651b1245bf3c0407e710310e3742cbaf8ce721a587b16a71d0356c32658"} Sep 16 07:28:27.681973 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:27.681939 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qprhb" event={"ID":"5f7afb78-2955-49e0-adec-6e8aae7ae51a","Type":"ContainerStarted","Data":"b26f09e95655de4b6fafb54e1c5a937a34e4a9efad9f7cbd2bc51fd3b98624c2"} Sep 16 07:28:27.702698 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:27.702640 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-qprhb" podStartSLOduration=8.170789238 podStartE2EDuration="39.702623869s" totalImagesPullingTime="31.531834631s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 07:27:48 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:28:27.701465726 +0000 UTC m=+40.728366759" watchObservedRunningTime="2026-09-16 07:28:27.702623869 +0000 UTC m=+40.729524880" Sep 16 07:28:29.107754 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:29.107698 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dbc6f54b-f7e7-4ede-b9a2-58f81e7a3f4c-metrics-tls\") pod \"dns-default-v8n9m\" (UID: \"dbc6f54b-f7e7-4ede-b9a2-58f81e7a3f4c\") " pod="openshift-dns/dns-default-v8n9m" Sep 16 07:28:29.108184 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:28:29.107859 2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 07:28:29.108184 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:28:29.107932 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbc6f54b-f7e7-4ede-b9a2-58f81e7a3f4c-metrics-tls podName:dbc6f54b-f7e7-4ede-b9a2-58f81e7a3f4c nodeName:}" failed. No retries permitted until 2026-09-16 07:28:37.107914798 +0000 UTC m=+50.134815789 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/dbc6f54b-f7e7-4ede-b9a2-58f81e7a3f4c-metrics-tls") pod "dns-default-v8n9m" (UID: "dbc6f54b-f7e7-4ede-b9a2-58f81e7a3f4c") : secret "dns-default-metrics-tls" not found Sep 16 07:28:37.164283 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:37.164218 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dbc6f54b-f7e7-4ede-b9a2-58f81e7a3f4c-metrics-tls\") pod \"dns-default-v8n9m\" (UID: \"dbc6f54b-f7e7-4ede-b9a2-58f81e7a3f4c\") " pod="openshift-dns/dns-default-v8n9m" Sep 16 07:28:37.164748 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:28:37.164350 2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 07:28:37.164748 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:28:37.164402 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbc6f54b-f7e7-4ede-b9a2-58f81e7a3f4c-metrics-tls podName:dbc6f54b-f7e7-4ede-b9a2-58f81e7a3f4c nodeName:}" failed. No retries permitted until 2026-09-16 07:28:53.164389694 +0000 UTC m=+66.191290684 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/dbc6f54b-f7e7-4ede-b9a2-58f81e7a3f4c-metrics-tls") pod "dns-default-v8n9m" (UID: "dbc6f54b-f7e7-4ede-b9a2-58f81e7a3f4c") : secret "dns-default-metrics-tls" not found Sep 16 07:28:48.666938 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:48.666902 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4ztgb" Sep 16 07:28:53.175116 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:53.175073 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dbc6f54b-f7e7-4ede-b9a2-58f81e7a3f4c-metrics-tls\") pod \"dns-default-v8n9m\" (UID: \"dbc6f54b-f7e7-4ede-b9a2-58f81e7a3f4c\") " pod="openshift-dns/dns-default-v8n9m" Sep 16 07:28:53.175116 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:53.175121 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/90e5e23a-060b-4612-80f5-f9a600b6d476-original-pull-secret\") pod \"global-pull-secret-syncer-t8thj\" (UID: \"90e5e23a-060b-4612-80f5-f9a600b6d476\") " pod="kube-system/global-pull-secret-syncer-t8thj" Sep 16 07:28:53.175569 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:28:53.175230 2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 07:28:53.175569 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:28:53.175333 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbc6f54b-f7e7-4ede-b9a2-58f81e7a3f4c-metrics-tls podName:dbc6f54b-f7e7-4ede-b9a2-58f81e7a3f4c nodeName:}" failed. No retries permitted until 2026-09-16 07:29:25.175315886 +0000 UTC m=+98.202216885 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/dbc6f54b-f7e7-4ede-b9a2-58f81e7a3f4c-metrics-tls") pod "dns-default-v8n9m" (UID: "dbc6f54b-f7e7-4ede-b9a2-58f81e7a3f4c") : secret "dns-default-metrics-tls" not found Sep 16 07:28:53.177626 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:53.177600 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 16 07:28:53.188920 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:53.188890 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/90e5e23a-060b-4612-80f5-f9a600b6d476-original-pull-secret\") pod \"global-pull-secret-syncer-t8thj\" (UID: \"90e5e23a-060b-4612-80f5-f9a600b6d476\") " pod="kube-system/global-pull-secret-syncer-t8thj" Sep 16 07:28:53.276298 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:53.276257 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/296c5dcf-403c-4a1f-9d6f-05ffb7d75e38-metrics-certs\") pod \"network-metrics-daemon-kqc7r\" (UID: \"296c5dcf-403c-4a1f-9d6f-05ffb7d75e38\") " pod="openshift-multus/network-metrics-daemon-kqc7r" Sep 16 07:28:53.278652 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:53.278631 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 16 07:28:53.287526 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:28:53.287492 2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 16 07:28:53.287588 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:28:53.287583 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/296c5dcf-403c-4a1f-9d6f-05ffb7d75e38-metrics-certs podName:296c5dcf-403c-4a1f-9d6f-05ffb7d75e38 nodeName:}" failed. No retries permitted until 2026-09-16 07:29:57.28756598 +0000 UTC m=+130.314466970 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/296c5dcf-403c-4a1f-9d6f-05ffb7d75e38-metrics-certs") pod "network-metrics-daemon-kqc7r" (UID: "296c5dcf-403c-4a1f-9d6f-05ffb7d75e38") : secret "metrics-daemon-secret" not found Sep 16 07:28:53.377383 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:53.377345 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6r6nv\" (UniqueName: \"kubernetes.io/projected/bfde2ba8-6a5f-413f-9ce9-d9ce0af2c758-kube-api-access-6r6nv\") pod \"network-check-target-jtqz9\" (UID: \"bfde2ba8-6a5f-413f-9ce9-d9ce0af2c758\") " pod="openshift-network-diagnostics/network-check-target-jtqz9" Sep 16 07:28:53.379599 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:53.379581 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 16 07:28:53.390160 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:53.390137 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 16 07:28:53.401452 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:53.401424 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6r6nv\" (UniqueName: \"kubernetes.io/projected/bfde2ba8-6a5f-413f-9ce9-d9ce0af2c758-kube-api-access-6r6nv\") pod \"network-check-target-jtqz9\" (UID: \"bfde2ba8-6a5f-413f-9ce9-d9ce0af2c758\") " pod="openshift-network-diagnostics/network-check-target-jtqz9" Sep 16 07:28:53.436879 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:53.436796 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-wbnpf\"" Sep 16 07:28:53.439777 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:53.439761 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t8thj" Sep 16 07:28:53.445198 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:53.445180 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jtqz9" Sep 16 07:28:53.709086 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:53.708990 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-t8thj"] Sep 16 07:28:53.709642 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:53.709618 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-jtqz9"] Sep 16 07:28:53.712409 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:28:53.712387 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90e5e23a_060b_4612_80f5_f9a600b6d476.slice/crio-f4816924412b67364dacdbcf29333747091dfcd4fb53e072bae550e8b0528784 WatchSource:0}: Error finding container f4816924412b67364dacdbcf29333747091dfcd4fb53e072bae550e8b0528784: Status 404 returned error can't find the container with id f4816924412b67364dacdbcf29333747091dfcd4fb53e072bae550e8b0528784 Sep 16 07:28:53.712859 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:28:53.712842 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfde2ba8_6a5f_413f_9ce9_d9ce0af2c758.slice/crio-d9476e0b6b1190b9e60e4b0f359d197169ecfbf84a4f7a7112b397bd1832644b WatchSource:0}: Error finding container d9476e0b6b1190b9e60e4b0f359d197169ecfbf84a4f7a7112b397bd1832644b: Status 404 returned error can't find the container with id d9476e0b6b1190b9e60e4b0f359d197169ecfbf84a4f7a7112b397bd1832644b Sep 16 07:28:53.744583 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:53.744550 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-jtqz9" event={"ID":"bfde2ba8-6a5f-413f-9ce9-d9ce0af2c758","Type":"ContainerStarted","Data":"d9476e0b6b1190b9e60e4b0f359d197169ecfbf84a4f7a7112b397bd1832644b"} Sep 16 07:28:53.745601 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:53.745571 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-t8thj" event={"ID":"90e5e23a-060b-4612-80f5-f9a600b6d476","Type":"ContainerStarted","Data":"f4816924412b67364dacdbcf29333747091dfcd4fb53e072bae550e8b0528784"} Sep 16 07:28:58.759293 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:58.759222 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-jtqz9" event={"ID":"bfde2ba8-6a5f-413f-9ce9-d9ce0af2c758","Type":"ContainerStarted","Data":"ada9e56027a3a890df90e5b517c1a956f4001ba1c31fa34408bf019faede8405"} Sep 16 07:28:58.760603 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:58.760579 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-t8thj" event={"ID":"90e5e23a-060b-4612-80f5-f9a600b6d476","Type":"ContainerStarted","Data":"e125c2f44a26c8402c7112ce8e45bd0f0b5e664c89f6ebb01f43680ded90d81d"} Sep 16 07:28:58.773479 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:58.773435 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-jtqz9" podStartSLOduration=66.168835033 podStartE2EDuration="1m10.773420641s" totalImagesPullingTime="4.604585608s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:27:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:28:58.773211049 +0000 UTC m=+71.800112081" watchObservedRunningTime="2026-09-16 07:28:58.773420641 +0000 UTC m=+71.800321653" Sep 16 07:28:58.786601 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:58.786546 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-t8thj" podStartSLOduration=66.176702552 podStartE2EDuration="1m10.786529452s" totalImagesPullingTime="4.6098269s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 07:27:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:28:58.785711059 +0000 UTC m=+71.812612072" watchObservedRunningTime="2026-09-16 07:28:58.786529452 +0000 UTC m=+71.813430465" Sep 16 07:28:59.763416 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:28:59.763386 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-jtqz9" Sep 16 07:29:25.192846 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:29:25.192809 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dbc6f54b-f7e7-4ede-b9a2-58f81e7a3f4c-metrics-tls\") pod \"dns-default-v8n9m\" (UID: \"dbc6f54b-f7e7-4ede-b9a2-58f81e7a3f4c\") " pod="openshift-dns/dns-default-v8n9m" Sep 16 07:29:25.193348 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:29:25.192937 2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 07:29:25.193348 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:29:25.192987 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbc6f54b-f7e7-4ede-b9a2-58f81e7a3f4c-metrics-tls podName:dbc6f54b-f7e7-4ede-b9a2-58f81e7a3f4c nodeName:}" failed. No retries permitted until 2026-09-16 07:30:29.192974555 +0000 UTC m=+162.219875545 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/dbc6f54b-f7e7-4ede-b9a2-58f81e7a3f4c-metrics-tls") pod "dns-default-v8n9m" (UID: "dbc6f54b-f7e7-4ede-b9a2-58f81e7a3f4c") : secret "dns-default-metrics-tls" not found Sep 16 07:29:26.027370 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:29:26.027328 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75fbf854f4-frbn5"] Sep 16 07:29:26.031588 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:29:26.031563 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75fbf854f4-frbn5" Sep 16 07:29:26.035313 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:29:26.035289 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-qxdpf\"" Sep 16 07:29:26.035747 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:29:26.035730 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Sep 16 07:29:26.036120 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:29:26.036089 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Sep 16 07:29:26.036167 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:29:26.036132 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Sep 16 07:29:26.036212 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:29:26.036180 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Sep 16 07:29:26.041661 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:29:26.041634 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b4fbfd475-5xvpw"] Sep 16 07:29:26.044408 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:29:26.044391 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b4fbfd475-5xvpw" Sep 16 07:29:26.050457 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:29:26.050435 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75fbf854f4-frbn5"] Sep 16 07:29:26.050722 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:29:26.050709 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Sep 16 07:29:26.051147 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:29:26.051131 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Sep 16 07:29:26.051253 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:29:26.051221 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Sep 16 07:29:26.052151 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:29:26.052135 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Sep 16 07:29:26.061488 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:29:26.061459 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b4fbfd475-5xvpw"] Sep 16 07:29:26.099098 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:29:26.099069 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhh25\" (UniqueName: \"kubernetes.io/projected/445e9de2-283b-4d3d-81b9-480fb2de1d2b-kube-api-access-rhh25\") pod \"cluster-proxy-proxy-agent-6b4fbfd475-5xvpw\" (UID: \"445e9de2-283b-4d3d-81b9-480fb2de1d2b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b4fbfd475-5xvpw" Sep 16 07:29:26.099201 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:29:26.099103 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/445e9de2-283b-4d3d-81b9-480fb2de1d2b-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6b4fbfd475-5xvpw\" (UID: \"445e9de2-283b-4d3d-81b9-480fb2de1d2b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b4fbfd475-5xvpw" Sep 16 07:29:26.099201 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:29:26.099118 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8dd9f4c8-f461-410b-a852-13fc43b4d407-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-75fbf854f4-frbn5\" (UID: \"8dd9f4c8-f461-410b-a852-13fc43b4d407\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75fbf854f4-frbn5" Sep 16 07:29:26.099201 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:29:26.099149 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/445e9de2-283b-4d3d-81b9-480fb2de1d2b-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6b4fbfd475-5xvpw\" (UID: \"445e9de2-283b-4d3d-81b9-480fb2de1d2b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b4fbfd475-5xvpw" Sep 16 07:29:26.099356 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:29:26.099202 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/445e9de2-283b-4d3d-81b9-480fb2de1d2b-ca\") pod \"cluster-proxy-proxy-agent-6b4fbfd475-5xvpw\" (UID: \"445e9de2-283b-4d3d-81b9-480fb2de1d2b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b4fbfd475-5xvpw" Sep 16 07:29:26.099356 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:29:26.099232 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/445e9de2-283b-4d3d-81b9-480fb2de1d2b-hub\") pod \"cluster-proxy-proxy-agent-6b4fbfd475-5xvpw\" (UID: \"445e9de2-283b-4d3d-81b9-480fb2de1d2b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b4fbfd475-5xvpw" Sep 16 07:29:26.099356 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:29:26.099298 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knhj2\" (UniqueName: \"kubernetes.io/projected/8dd9f4c8-f461-410b-a852-13fc43b4d407-kube-api-access-knhj2\") pod \"managed-serviceaccount-addon-agent-75fbf854f4-frbn5\" (UID: \"8dd9f4c8-f461-410b-a852-13fc43b4d407\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75fbf854f4-frbn5" Sep 16 07:29:26.099356 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:29:26.099316 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/445e9de2-283b-4d3d-81b9-480fb2de1d2b-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6b4fbfd475-5xvpw\" (UID: \"445e9de2-283b-4d3d-81b9-480fb2de1d2b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b4fbfd475-5xvpw" Sep 16 07:29:26.199824 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:29:26.199800 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/445e9de2-283b-4d3d-81b9-480fb2de1d2b-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6b4fbfd475-5xvpw\" (UID: \"445e9de2-283b-4d3d-81b9-480fb2de1d2b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b4fbfd475-5xvpw" Sep 16 07:29:26.200119 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:29:26.199827 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/445e9de2-283b-4d3d-81b9-480fb2de1d2b-ca\") pod \"cluster-proxy-proxy-agent-6b4fbfd475-5xvpw\" (UID: \"445e9de2-283b-4d3d-81b9-480fb2de1d2b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b4fbfd475-5xvpw" Sep 16 07:29:26.200119 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:29:26.199842 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/445e9de2-283b-4d3d-81b9-480fb2de1d2b-hub\") pod \"cluster-proxy-proxy-agent-6b4fbfd475-5xvpw\" (UID: \"445e9de2-283b-4d3d-81b9-480fb2de1d2b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b4fbfd475-5xvpw" Sep 16 07:29:26.200119 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:29:26.199874 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knhj2\" (UniqueName: \"kubernetes.io/projected/8dd9f4c8-f461-410b-a852-13fc43b4d407-kube-api-access-knhj2\") pod \"managed-serviceaccount-addon-agent-75fbf854f4-frbn5\" (UID: \"8dd9f4c8-f461-410b-a852-13fc43b4d407\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75fbf854f4-frbn5" Sep 16 07:29:26.200119 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:29:26.199890 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/445e9de2-283b-4d3d-81b9-480fb2de1d2b-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6b4fbfd475-5xvpw\" (UID: \"445e9de2-283b-4d3d-81b9-480fb2de1d2b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b4fbfd475-5xvpw" Sep 16 07:29:26.200119 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:29:26.199916 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhh25\" (UniqueName: \"kubernetes.io/projected/445e9de2-283b-4d3d-81b9-480fb2de1d2b-kube-api-access-rhh25\") pod \"cluster-proxy-proxy-agent-6b4fbfd475-5xvpw\" (UID: \"445e9de2-283b-4d3d-81b9-480fb2de1d2b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b4fbfd475-5xvpw" Sep 16 07:29:26.200119 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:29:26.199951 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/445e9de2-283b-4d3d-81b9-480fb2de1d2b-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6b4fbfd475-5xvpw\" (UID: \"445e9de2-283b-4d3d-81b9-480fb2de1d2b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b4fbfd475-5xvpw" Sep 16 07:29:26.200119 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:29:26.199974 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8dd9f4c8-f461-410b-a852-13fc43b4d407-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-75fbf854f4-frbn5\" (UID: \"8dd9f4c8-f461-410b-a852-13fc43b4d407\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75fbf854f4-frbn5" Sep 16 07:29:26.200740 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:29:26.200712 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/445e9de2-283b-4d3d-81b9-480fb2de1d2b-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6b4fbfd475-5xvpw\" (UID: \"445e9de2-283b-4d3d-81b9-480fb2de1d2b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b4fbfd475-5xvpw" Sep 16 07:29:26.202544 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:29:26.202511 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/445e9de2-283b-4d3d-81b9-480fb2de1d2b-hub\") pod \"cluster-proxy-proxy-agent-6b4fbfd475-5xvpw\" (UID: \"445e9de2-283b-4d3d-81b9-480fb2de1d2b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b4fbfd475-5xvpw" Sep 16 07:29:26.202628 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:29:26.202613 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/445e9de2-283b-4d3d-81b9-480fb2de1d2b-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6b4fbfd475-5xvpw\" (UID: \"445e9de2-283b-4d3d-81b9-480fb2de1d2b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b4fbfd475-5xvpw" Sep 16 07:29:26.202788 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:29:26.202771 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/445e9de2-283b-4d3d-81b9-480fb2de1d2b-ca\") pod \"cluster-proxy-proxy-agent-6b4fbfd475-5xvpw\" (UID: \"445e9de2-283b-4d3d-81b9-480fb2de1d2b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b4fbfd475-5xvpw" Sep 16 07:29:26.202828 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:29:26.202809 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8dd9f4c8-f461-410b-a852-13fc43b4d407-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-75fbf854f4-frbn5\" (UID: \"8dd9f4c8-f461-410b-a852-13fc43b4d407\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75fbf854f4-frbn5" Sep 16 07:29:26.203084 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:29:26.203062 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/445e9de2-283b-4d3d-81b9-480fb2de1d2b-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6b4fbfd475-5xvpw\" (UID: \"445e9de2-283b-4d3d-81b9-480fb2de1d2b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b4fbfd475-5xvpw" Sep 16 07:29:26.208088 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:29:26.208066 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-knhj2\" (UniqueName: \"kubernetes.io/projected/8dd9f4c8-f461-410b-a852-13fc43b4d407-kube-api-access-knhj2\") pod \"managed-serviceaccount-addon-agent-75fbf854f4-frbn5\" (UID: \"8dd9f4c8-f461-410b-a852-13fc43b4d407\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75fbf854f4-frbn5" Sep 16 07:29:26.208549 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:29:26.208529 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhh25\" (UniqueName: \"kubernetes.io/projected/445e9de2-283b-4d3d-81b9-480fb2de1d2b-kube-api-access-rhh25\") pod \"cluster-proxy-proxy-agent-6b4fbfd475-5xvpw\" (UID: \"445e9de2-283b-4d3d-81b9-480fb2de1d2b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b4fbfd475-5xvpw" Sep 16 07:29:26.355679 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:29:26.355621 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75fbf854f4-frbn5" Sep 16 07:29:26.366811 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:29:26.366795 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b4fbfd475-5xvpw" Sep 16 07:29:26.624717 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:29:26.624642 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75fbf854f4-frbn5"] Sep 16 07:29:26.628125 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:29:26.628099 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8dd9f4c8_f461_410b_a852_13fc43b4d407.slice/crio-f6cf80ccad7c597cdaf4906536a8d6007d4b512e43dd3ae24326dd6bdb665afe WatchSource:0}: Error finding container f6cf80ccad7c597cdaf4906536a8d6007d4b512e43dd3ae24326dd6bdb665afe: Status 404 returned error can't find the container with id f6cf80ccad7c597cdaf4906536a8d6007d4b512e43dd3ae24326dd6bdb665afe Sep 16 07:29:26.646049 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:29:26.646022 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b4fbfd475-5xvpw"] Sep 16 07:29:26.648495 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:29:26.648468 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod445e9de2_283b_4d3d_81b9_480fb2de1d2b.slice/crio-2e250cf14adca5d9a227a9db758b5d178a8c69269c3f3db1b347c6be50350c1b WatchSource:0}: Error finding container 2e250cf14adca5d9a227a9db758b5d178a8c69269c3f3db1b347c6be50350c1b: Status 404 returned error can't find the container with id 2e250cf14adca5d9a227a9db758b5d178a8c69269c3f3db1b347c6be50350c1b Sep 16 07:29:26.829634 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:29:26.829595 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b4fbfd475-5xvpw" event={"ID":"445e9de2-283b-4d3d-81b9-480fb2de1d2b","Type":"ContainerStarted","Data":"2e250cf14adca5d9a227a9db758b5d178a8c69269c3f3db1b347c6be50350c1b"} Sep 16 07:29:26.830707 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:29:26.830683 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75fbf854f4-frbn5" event={"ID":"8dd9f4c8-f461-410b-a852-13fc43b4d407","Type":"ContainerStarted","Data":"f6cf80ccad7c597cdaf4906536a8d6007d4b512e43dd3ae24326dd6bdb665afe"} Sep 16 07:29:29.839437 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:29:29.839396 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75fbf854f4-frbn5" event={"ID":"8dd9f4c8-f461-410b-a852-13fc43b4d407","Type":"ContainerStarted","Data":"369a446c5b5f4015d3ea7f5cf41358ab06c92c07f1ed57081223bc95b6cf1125"} Sep 16 07:29:29.855199 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:29:29.855146 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75fbf854f4-frbn5" podStartSLOduration=1.262264274 podStartE2EDuration="3.855126237s" totalImagesPullingTime="2.592861963s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:29:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:29:29.853571383 +0000 UTC m=+102.880472398" watchObservedRunningTime="2026-09-16 07:29:29.855126237 +0000 UTC m=+102.882027251" Sep 16 07:29:30.769082 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:29:30.769052 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-jtqz9" Sep 16 07:29:32.848633 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:29:32.848599 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b4fbfd475-5xvpw" event={"ID":"445e9de2-283b-4d3d-81b9-480fb2de1d2b","Type":"ContainerStarted","Data":"fb96d2e2b10e2adee65c13ea7aa33ddff7cfe554e7ee787afe288675a9c77575"} Sep 16 07:29:32.849015 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:29:32.848644 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b4fbfd475-5xvpw" event={"ID":"445e9de2-283b-4d3d-81b9-480fb2de1d2b","Type":"ContainerStarted","Data":"f13756809a41cdfd575ba23c599ea17d0895238927f332f43a818918891e6249"} Sep 16 07:29:32.849015 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:29:32.848658 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b4fbfd475-5xvpw" event={"ID":"445e9de2-283b-4d3d-81b9-480fb2de1d2b","Type":"ContainerStarted","Data":"141d551bf7217fe99d8c647e4f1b1897db3bcaace346e9e359606bccda2af72e"} Sep 16 07:29:32.866365 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:29:32.866311 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b4fbfd475-5xvpw" podStartSLOduration=1.217874556 podStartE2EDuration="6.86629623s" totalImagesPullingTime="5.648421674s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:29:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:29:32.865919958 +0000 UTC m=+105.892820969" watchObservedRunningTime="2026-09-16 07:29:32.86629623 +0000 UTC m=+105.893197244" Sep 16 07:29:57.305548 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:29:57.305506 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/296c5dcf-403c-4a1f-9d6f-05ffb7d75e38-metrics-certs\") pod \"network-metrics-daemon-kqc7r\" (UID: \"296c5dcf-403c-4a1f-9d6f-05ffb7d75e38\") " pod="openshift-multus/network-metrics-daemon-kqc7r" Sep 16 07:29:57.306081 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:29:57.305627 2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 16 07:29:57.306081 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:29:57.305688 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/296c5dcf-403c-4a1f-9d6f-05ffb7d75e38-metrics-certs podName:296c5dcf-403c-4a1f-9d6f-05ffb7d75e38 nodeName:}" failed. No retries permitted until 2026-09-16 07:31:59.305670279 +0000 UTC m=+252.332571268 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/296c5dcf-403c-4a1f-9d6f-05ffb7d75e38-metrics-certs") pod "network-metrics-daemon-kqc7r" (UID: "296c5dcf-403c-4a1f-9d6f-05ffb7d75e38") : secret "metrics-daemon-secret" not found Sep 16 07:30:24.318141 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:30:24.318097 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-v8n9m" podUID="dbc6f54b-f7e7-4ede-b9a2-58f81e7a3f4c" Sep 16 07:30:24.999934 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:30:24.999903 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-v8n9m" Sep 16 07:30:25.528992 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:30:25.528957 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-kqc7r" podUID="296c5dcf-403c-4a1f-9d6f-05ffb7d75e38" Sep 16 07:30:29.208438 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:30:29.208405 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dbc6f54b-f7e7-4ede-b9a2-58f81e7a3f4c-metrics-tls\") pod \"dns-default-v8n9m\" (UID: \"dbc6f54b-f7e7-4ede-b9a2-58f81e7a3f4c\") " pod="openshift-dns/dns-default-v8n9m" Sep 16 07:30:29.208796 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:30:29.208518 2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 07:30:29.208796 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:30:29.208576 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbc6f54b-f7e7-4ede-b9a2-58f81e7a3f4c-metrics-tls podName:dbc6f54b-f7e7-4ede-b9a2-58f81e7a3f4c nodeName:}" failed. No retries permitted until 2026-09-16 07:32:31.208562431 +0000 UTC m=+284.235463431 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/dbc6f54b-f7e7-4ede-b9a2-58f81e7a3f4c-metrics-tls") pod "dns-default-v8n9m" (UID: "dbc6f54b-f7e7-4ede-b9a2-58f81e7a3f4c") : secret "dns-default-metrics-tls" not found Sep 16 07:30:38.519413 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:30:38.519364 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kqc7r" Sep 16 07:31:30.184945 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:31:30.184908 2557 generic.go:356] "Generic (PLEG): container finished" podID="8dd9f4c8-f461-410b-a852-13fc43b4d407" containerID="369a446c5b5f4015d3ea7f5cf41358ab06c92c07f1ed57081223bc95b6cf1125" exitCode=255 Sep 16 07:31:30.185341 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:31:30.184962 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75fbf854f4-frbn5" event={"ID":"8dd9f4c8-f461-410b-a852-13fc43b4d407","Type":"ContainerDied","Data":"369a446c5b5f4015d3ea7f5cf41358ab06c92c07f1ed57081223bc95b6cf1125"} Sep 16 07:31:30.185341 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:31:30.185230 2557 scope.go:117] "RemoveContainer" containerID="369a446c5b5f4015d3ea7f5cf41358ab06c92c07f1ed57081223bc95b6cf1125" Sep 16 07:31:31.189604 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:31:31.189576 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75fbf854f4-frbn5" event={"ID":"8dd9f4c8-f461-410b-a852-13fc43b4d407","Type":"ContainerStarted","Data":"bbdcad8abf053de39fba714d8c3b4b3b3d88ec9d48903d43eec2d33baf219c1b"} Sep 16 07:31:59.051869 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:31:59.051835 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-xq64b"] Sep 16 07:31:59.054963 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:31:59.054944 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-xq64b" Sep 16 07:31:59.058569 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:31:59.058542 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 16 07:31:59.058684 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:31:59.058569 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 16 07:31:59.058868 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:31:59.058852 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 16 07:31:59.059557 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:31:59.059535 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-qw2t8\"" Sep 16 07:31:59.059671 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:31:59.059539 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 16 07:31:59.066546 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:31:59.066520 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-xq64b"] Sep 16 07:31:59.201097 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:31:59.201073 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/44a442da-5d1a-4b97-8697-3ce085773de6-data-volume\") pod \"insights-runtime-extractor-xq64b\" (UID: \"44a442da-5d1a-4b97-8697-3ce085773de6\") " pod="openshift-insights/insights-runtime-extractor-xq64b" Sep 16 07:31:59.201210 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:31:59.201101 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/44a442da-5d1a-4b97-8697-3ce085773de6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xq64b\" (UID: \"44a442da-5d1a-4b97-8697-3ce085773de6\") " pod="openshift-insights/insights-runtime-extractor-xq64b" Sep 16 07:31:59.201210 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:31:59.201128 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/44a442da-5d1a-4b97-8697-3ce085773de6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xq64b\" (UID: \"44a442da-5d1a-4b97-8697-3ce085773de6\") " pod="openshift-insights/insights-runtime-extractor-xq64b" Sep 16 07:31:59.201210 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:31:59.201156 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8478t\" (UniqueName: \"kubernetes.io/projected/44a442da-5d1a-4b97-8697-3ce085773de6-kube-api-access-8478t\") pod \"insights-runtime-extractor-xq64b\" (UID: \"44a442da-5d1a-4b97-8697-3ce085773de6\") " pod="openshift-insights/insights-runtime-extractor-xq64b" Sep 16 07:31:59.201356 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:31:59.201220 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/44a442da-5d1a-4b97-8697-3ce085773de6-crio-socket\") pod \"insights-runtime-extractor-xq64b\" (UID: \"44a442da-5d1a-4b97-8697-3ce085773de6\") " pod="openshift-insights/insights-runtime-extractor-xq64b" Sep 16 07:31:59.301619 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:31:59.301598 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/44a442da-5d1a-4b97-8697-3ce085773de6-crio-socket\") pod \"insights-runtime-extractor-xq64b\" (UID: \"44a442da-5d1a-4b97-8697-3ce085773de6\") " pod="openshift-insights/insights-runtime-extractor-xq64b" Sep 16 07:31:59.301743 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:31:59.301640 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/44a442da-5d1a-4b97-8697-3ce085773de6-data-volume\") pod \"insights-runtime-extractor-xq64b\" (UID: \"44a442da-5d1a-4b97-8697-3ce085773de6\") " pod="openshift-insights/insights-runtime-extractor-xq64b" Sep 16 07:31:59.301743 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:31:59.301658 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/44a442da-5d1a-4b97-8697-3ce085773de6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xq64b\" (UID: \"44a442da-5d1a-4b97-8697-3ce085773de6\") " pod="openshift-insights/insights-runtime-extractor-xq64b" Sep 16 07:31:59.301743 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:31:59.301684 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/44a442da-5d1a-4b97-8697-3ce085773de6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xq64b\" (UID: \"44a442da-5d1a-4b97-8697-3ce085773de6\") " pod="openshift-insights/insights-runtime-extractor-xq64b" Sep 16 07:31:59.301743 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:31:59.301724 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8478t\" (UniqueName: \"kubernetes.io/projected/44a442da-5d1a-4b97-8697-3ce085773de6-kube-api-access-8478t\") pod \"insights-runtime-extractor-xq64b\" (UID: \"44a442da-5d1a-4b97-8697-3ce085773de6\") " pod="openshift-insights/insights-runtime-extractor-xq64b" Sep 16 07:31:59.301743 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:31:59.301730 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/44a442da-5d1a-4b97-8697-3ce085773de6-crio-socket\") pod \"insights-runtime-extractor-xq64b\" (UID: \"44a442da-5d1a-4b97-8697-3ce085773de6\") " pod="openshift-insights/insights-runtime-extractor-xq64b" Sep 16 07:31:59.302010 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:31:59.301765 2557 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 16 07:31:59.302010 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:31:59.301830 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/44a442da-5d1a-4b97-8697-3ce085773de6-kube-rbac-proxy-cm podName:44a442da-5d1a-4b97-8697-3ce085773de6 nodeName:}" failed. No retries permitted until 2026-09-16 07:31:59.801812785 +0000 UTC m=+252.828713774 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/44a442da-5d1a-4b97-8697-3ce085773de6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xq64b" (UID: "44a442da-5d1a-4b97-8697-3ce085773de6") : configmap "kube-rbac-proxy" not found Sep 16 07:31:59.302010 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:31:59.301831 2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 16 07:31:59.302010 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:31:59.301914 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44a442da-5d1a-4b97-8697-3ce085773de6-insights-runtime-extractor-tls podName:44a442da-5d1a-4b97-8697-3ce085773de6 nodeName:}" failed. No retries permitted until 2026-09-16 07:31:59.801893806 +0000 UTC m=+252.828794799 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/44a442da-5d1a-4b97-8697-3ce085773de6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xq64b" (UID: "44a442da-5d1a-4b97-8697-3ce085773de6") : secret "insights-runtime-extractor-tls" not found Sep 16 07:31:59.302010 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:31:59.301974 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/44a442da-5d1a-4b97-8697-3ce085773de6-data-volume\") pod \"insights-runtime-extractor-xq64b\" (UID: \"44a442da-5d1a-4b97-8697-3ce085773de6\") " pod="openshift-insights/insights-runtime-extractor-xq64b" Sep 16 07:31:59.310090 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:31:59.310065 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8478t\" (UniqueName: \"kubernetes.io/projected/44a442da-5d1a-4b97-8697-3ce085773de6-kube-api-access-8478t\") pod \"insights-runtime-extractor-xq64b\" (UID: \"44a442da-5d1a-4b97-8697-3ce085773de6\") " pod="openshift-insights/insights-runtime-extractor-xq64b" Sep 16 07:31:59.402531 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:31:59.402500 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/296c5dcf-403c-4a1f-9d6f-05ffb7d75e38-metrics-certs\") pod \"network-metrics-daemon-kqc7r\" (UID: \"296c5dcf-403c-4a1f-9d6f-05ffb7d75e38\") " pod="openshift-multus/network-metrics-daemon-kqc7r" Sep 16 07:31:59.402657 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:31:59.402627 2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 16 07:31:59.402705 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:31:59.402695 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/296c5dcf-403c-4a1f-9d6f-05ffb7d75e38-metrics-certs podName:296c5dcf-403c-4a1f-9d6f-05ffb7d75e38 nodeName:}" failed. No retries permitted until 2026-09-16 07:34:01.402680949 +0000 UTC m=+374.429581943 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/296c5dcf-403c-4a1f-9d6f-05ffb7d75e38-metrics-certs") pod "network-metrics-daemon-kqc7r" (UID: "296c5dcf-403c-4a1f-9d6f-05ffb7d75e38") : secret "metrics-daemon-secret" not found Sep 16 07:31:59.805464 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:31:59.805438 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/44a442da-5d1a-4b97-8697-3ce085773de6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xq64b\" (UID: \"44a442da-5d1a-4b97-8697-3ce085773de6\") " pod="openshift-insights/insights-runtime-extractor-xq64b" Sep 16 07:31:59.805584 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:31:59.805526 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/44a442da-5d1a-4b97-8697-3ce085773de6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xq64b\" (UID: \"44a442da-5d1a-4b97-8697-3ce085773de6\") " pod="openshift-insights/insights-runtime-extractor-xq64b" Sep 16 07:31:59.805637 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:31:59.805584 2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 16 07:31:59.805637 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:31:59.805595 2557 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 16 07:31:59.805721 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:31:59.805647 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/44a442da-5d1a-4b97-8697-3ce085773de6-kube-rbac-proxy-cm podName:44a442da-5d1a-4b97-8697-3ce085773de6 nodeName:}" failed. No retries permitted until 2026-09-16 07:32:00.805632177 +0000 UTC m=+253.832533166 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/44a442da-5d1a-4b97-8697-3ce085773de6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xq64b" (UID: "44a442da-5d1a-4b97-8697-3ce085773de6") : configmap "kube-rbac-proxy" not found Sep 16 07:31:59.805721 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:31:59.805660 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44a442da-5d1a-4b97-8697-3ce085773de6-insights-runtime-extractor-tls podName:44a442da-5d1a-4b97-8697-3ce085773de6 nodeName:}" failed. No retries permitted until 2026-09-16 07:32:00.805653329 +0000 UTC m=+253.832554318 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/44a442da-5d1a-4b97-8697-3ce085773de6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xq64b" (UID: "44a442da-5d1a-4b97-8697-3ce085773de6") : secret "insights-runtime-extractor-tls" not found Sep 16 07:32:00.767540 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:00.767506 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-8gnk6_20f6d954-1104-49b9-ab10-32e0de4ef8e0/dns-node-resolver/0.log" Sep 16 07:32:00.810717 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:00.810671 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/44a442da-5d1a-4b97-8697-3ce085773de6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xq64b\" (UID: \"44a442da-5d1a-4b97-8697-3ce085773de6\") " pod="openshift-insights/insights-runtime-extractor-xq64b" Sep 16 07:32:00.810859 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:00.810726 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/44a442da-5d1a-4b97-8697-3ce085773de6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xq64b\" (UID: \"44a442da-5d1a-4b97-8697-3ce085773de6\") " pod="openshift-insights/insights-runtime-extractor-xq64b" Sep 16 07:32:00.810859 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:32:00.810779 2557 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 16 07:32:00.810859 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:32:00.810827 2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 16 07:32:00.810859 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:32:00.810838 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/44a442da-5d1a-4b97-8697-3ce085773de6-kube-rbac-proxy-cm podName:44a442da-5d1a-4b97-8697-3ce085773de6 nodeName:}" failed. No retries permitted until 2026-09-16 07:32:02.81082449 +0000 UTC m=+255.837725480 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/44a442da-5d1a-4b97-8697-3ce085773de6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xq64b" (UID: "44a442da-5d1a-4b97-8697-3ce085773de6") : configmap "kube-rbac-proxy" not found Sep 16 07:32:00.811040 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:32:00.810897 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44a442da-5d1a-4b97-8697-3ce085773de6-insights-runtime-extractor-tls podName:44a442da-5d1a-4b97-8697-3ce085773de6 nodeName:}" failed. No retries permitted until 2026-09-16 07:32:02.810879664 +0000 UTC m=+255.837780697 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/44a442da-5d1a-4b97-8697-3ce085773de6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xq64b" (UID: "44a442da-5d1a-4b97-8697-3ce085773de6") : secret "insights-runtime-extractor-tls" not found Sep 16 07:32:01.967760 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:01.967724 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-vbxvc_89f6995b-d178-4fe9-9593-ba8dc8f49351/node-ca/0.log" Sep 16 07:32:02.567188 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:02.567154 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-lprn9_7c9bb910-6eda-4d97-9d79-7d78fd9abb58/serve-healthcheck-canary/0.log" Sep 16 07:32:02.824191 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:02.824127 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/44a442da-5d1a-4b97-8697-3ce085773de6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xq64b\" (UID: \"44a442da-5d1a-4b97-8697-3ce085773de6\") " pod="openshift-insights/insights-runtime-extractor-xq64b" Sep 16 07:32:02.824191 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:02.824160 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/44a442da-5d1a-4b97-8697-3ce085773de6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xq64b\" (UID: \"44a442da-5d1a-4b97-8697-3ce085773de6\") " pod="openshift-insights/insights-runtime-extractor-xq64b" Sep 16 07:32:02.824386 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:32:02.824257 2557 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 16 07:32:02.824386 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:32:02.824283 2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 16 07:32:02.824386 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:32:02.824327 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/44a442da-5d1a-4b97-8697-3ce085773de6-kube-rbac-proxy-cm podName:44a442da-5d1a-4b97-8697-3ce085773de6 nodeName:}" failed. No retries permitted until 2026-09-16 07:32:06.824308967 +0000 UTC m=+259.851209988 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/44a442da-5d1a-4b97-8697-3ce085773de6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xq64b" (UID: "44a442da-5d1a-4b97-8697-3ce085773de6") : configmap "kube-rbac-proxy" not found Sep 16 07:32:02.824386 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:32:02.824343 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44a442da-5d1a-4b97-8697-3ce085773de6-insights-runtime-extractor-tls podName:44a442da-5d1a-4b97-8697-3ce085773de6 nodeName:}" failed. No retries permitted until 2026-09-16 07:32:06.824335882 +0000 UTC m=+259.851236871 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/44a442da-5d1a-4b97-8697-3ce085773de6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xq64b" (UID: "44a442da-5d1a-4b97-8697-3ce085773de6") : secret "insights-runtime-extractor-tls" not found Sep 16 07:32:06.852748 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:06.852688 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/44a442da-5d1a-4b97-8697-3ce085773de6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xq64b\" (UID: \"44a442da-5d1a-4b97-8697-3ce085773de6\") " pod="openshift-insights/insights-runtime-extractor-xq64b" Sep 16 07:32:06.852748 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:06.852750 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/44a442da-5d1a-4b97-8697-3ce085773de6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xq64b\" (UID: \"44a442da-5d1a-4b97-8697-3ce085773de6\") " pod="openshift-insights/insights-runtime-extractor-xq64b" Sep 16 07:32:06.853289 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:32:06.852818 2557 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 16 07:32:06.853289 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:32:06.852856 2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 16 07:32:06.853289 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:32:06.852902 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/44a442da-5d1a-4b97-8697-3ce085773de6-kube-rbac-proxy-cm podName:44a442da-5d1a-4b97-8697-3ce085773de6 nodeName:}" failed. No retries permitted until 2026-09-16 07:32:14.852882185 +0000 UTC m=+267.879783200 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/44a442da-5d1a-4b97-8697-3ce085773de6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xq64b" (UID: "44a442da-5d1a-4b97-8697-3ce085773de6") : configmap "kube-rbac-proxy" not found Sep 16 07:32:06.853289 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:32:06.852932 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44a442da-5d1a-4b97-8697-3ce085773de6-insights-runtime-extractor-tls podName:44a442da-5d1a-4b97-8697-3ce085773de6 nodeName:}" failed. No retries permitted until 2026-09-16 07:32:14.852920695 +0000 UTC m=+267.879821698 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/44a442da-5d1a-4b97-8697-3ce085773de6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xq64b" (UID: "44a442da-5d1a-4b97-8697-3ce085773de6") : secret "insights-runtime-extractor-tls" not found Sep 16 07:32:14.909052 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:14.909011 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/44a442da-5d1a-4b97-8697-3ce085773de6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xq64b\" (UID: \"44a442da-5d1a-4b97-8697-3ce085773de6\") " pod="openshift-insights/insights-runtime-extractor-xq64b" Sep 16 07:32:14.909052 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:14.909054 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/44a442da-5d1a-4b97-8697-3ce085773de6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xq64b\" (UID: \"44a442da-5d1a-4b97-8697-3ce085773de6\") " pod="openshift-insights/insights-runtime-extractor-xq64b" Sep 16 07:32:14.909595 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:32:14.909159 2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 16 07:32:14.909595 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:32:14.909212 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44a442da-5d1a-4b97-8697-3ce085773de6-insights-runtime-extractor-tls podName:44a442da-5d1a-4b97-8697-3ce085773de6 nodeName:}" failed. No retries permitted until 2026-09-16 07:32:30.909199525 +0000 UTC m=+283.936100515 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/44a442da-5d1a-4b97-8697-3ce085773de6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xq64b" (UID: "44a442da-5d1a-4b97-8697-3ce085773de6") : secret "insights-runtime-extractor-tls" not found Sep 16 07:32:14.909595 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:14.909560 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/44a442da-5d1a-4b97-8697-3ce085773de6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xq64b\" (UID: \"44a442da-5d1a-4b97-8697-3ce085773de6\") " pod="openshift-insights/insights-runtime-extractor-xq64b" Sep 16 07:32:28.000633 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:32:28.000598 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-v8n9m" podUID="dbc6f54b-f7e7-4ede-b9a2-58f81e7a3f4c" Sep 16 07:32:28.353195 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:28.353120 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-v8n9m" Sep 16 07:32:29.384701 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:29.384669 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-9686b8b46-nklrk"] Sep 16 07:32:29.387743 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:29.387725 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9686b8b46-nklrk" Sep 16 07:32:29.390317 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:29.390293 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 16 07:32:29.391095 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:29.391079 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 16 07:32:29.391148 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:29.391120 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 16 07:32:29.391419 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:29.391388 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 16 07:32:29.391419 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:29.391402 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 16 07:32:29.391556 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:29.391395 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 16 07:32:29.391556 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:29.391432 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-rxmq4\"" Sep 16 07:32:29.391556 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:29.391396 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 16 07:32:29.397055 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:29.397033 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-9686b8b46-nklrk"] Sep 16 07:32:29.401518 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:29.401498 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/88feefad-338d-4fde-b835-c7aba8b1c875-service-ca\") pod \"console-9686b8b46-nklrk\" (UID: \"88feefad-338d-4fde-b835-c7aba8b1c875\") " pod="openshift-console/console-9686b8b46-nklrk" Sep 16 07:32:29.401601 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:29.401529 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/88feefad-338d-4fde-b835-c7aba8b1c875-console-oauth-config\") pod \"console-9686b8b46-nklrk\" (UID: \"88feefad-338d-4fde-b835-c7aba8b1c875\") " pod="openshift-console/console-9686b8b46-nklrk" Sep 16 07:32:29.401601 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:29.401544 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zqpp\" (UniqueName: \"kubernetes.io/projected/88feefad-338d-4fde-b835-c7aba8b1c875-kube-api-access-6zqpp\") pod \"console-9686b8b46-nklrk\" (UID: \"88feefad-338d-4fde-b835-c7aba8b1c875\") " pod="openshift-console/console-9686b8b46-nklrk" Sep 16 07:32:29.401601 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:29.401571 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/88feefad-338d-4fde-b835-c7aba8b1c875-console-config\") pod \"console-9686b8b46-nklrk\" (UID: \"88feefad-338d-4fde-b835-c7aba8b1c875\") " pod="openshift-console/console-9686b8b46-nklrk" Sep 16 07:32:29.401712 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:29.401628 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/88feefad-338d-4fde-b835-c7aba8b1c875-oauth-serving-cert\") pod \"console-9686b8b46-nklrk\" (UID: \"88feefad-338d-4fde-b835-c7aba8b1c875\") " pod="openshift-console/console-9686b8b46-nklrk" Sep 16 07:32:29.401749 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:29.401714 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/88feefad-338d-4fde-b835-c7aba8b1c875-console-serving-cert\") pod \"console-9686b8b46-nklrk\" (UID: \"88feefad-338d-4fde-b835-c7aba8b1c875\") " pod="openshift-console/console-9686b8b46-nklrk" Sep 16 07:32:29.502317 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:29.502293 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/88feefad-338d-4fde-b835-c7aba8b1c875-console-config\") pod \"console-9686b8b46-nklrk\" (UID: \"88feefad-338d-4fde-b835-c7aba8b1c875\") " pod="openshift-console/console-9686b8b46-nklrk" Sep 16 07:32:29.502434 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:29.502328 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/88feefad-338d-4fde-b835-c7aba8b1c875-oauth-serving-cert\") pod \"console-9686b8b46-nklrk\" (UID: \"88feefad-338d-4fde-b835-c7aba8b1c875\") " pod="openshift-console/console-9686b8b46-nklrk" Sep 16 07:32:29.502434 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:29.502357 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/88feefad-338d-4fde-b835-c7aba8b1c875-console-serving-cert\") pod \"console-9686b8b46-nklrk\" (UID: \"88feefad-338d-4fde-b835-c7aba8b1c875\") " pod="openshift-console/console-9686b8b46-nklrk" Sep 16 07:32:29.502434 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:29.502386 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/88feefad-338d-4fde-b835-c7aba8b1c875-service-ca\") pod \"console-9686b8b46-nklrk\" (UID: \"88feefad-338d-4fde-b835-c7aba8b1c875\") " pod="openshift-console/console-9686b8b46-nklrk" Sep 16 07:32:29.502434 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:29.502404 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/88feefad-338d-4fde-b835-c7aba8b1c875-console-oauth-config\") pod \"console-9686b8b46-nklrk\" (UID: \"88feefad-338d-4fde-b835-c7aba8b1c875\") " pod="openshift-console/console-9686b8b46-nklrk" Sep 16 07:32:29.502434 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:29.502417 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zqpp\" (UniqueName: \"kubernetes.io/projected/88feefad-338d-4fde-b835-c7aba8b1c875-kube-api-access-6zqpp\") pod \"console-9686b8b46-nklrk\" (UID: \"88feefad-338d-4fde-b835-c7aba8b1c875\") " pod="openshift-console/console-9686b8b46-nklrk" Sep 16 07:32:29.503024 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:29.503000 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/88feefad-338d-4fde-b835-c7aba8b1c875-console-config\") pod \"console-9686b8b46-nklrk\" (UID: \"88feefad-338d-4fde-b835-c7aba8b1c875\") " pod="openshift-console/console-9686b8b46-nklrk" Sep 16 07:32:29.503115 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:29.503032 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/88feefad-338d-4fde-b835-c7aba8b1c875-service-ca\") pod \"console-9686b8b46-nklrk\" (UID: \"88feefad-338d-4fde-b835-c7aba8b1c875\") " pod="openshift-console/console-9686b8b46-nklrk" Sep 16 07:32:29.503115 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:29.503050 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/88feefad-338d-4fde-b835-c7aba8b1c875-oauth-serving-cert\") pod \"console-9686b8b46-nklrk\" (UID: \"88feefad-338d-4fde-b835-c7aba8b1c875\") " pod="openshift-console/console-9686b8b46-nklrk" Sep 16 07:32:29.504724 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:29.504701 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/88feefad-338d-4fde-b835-c7aba8b1c875-console-serving-cert\") pod \"console-9686b8b46-nklrk\" (UID: \"88feefad-338d-4fde-b835-c7aba8b1c875\") " pod="openshift-console/console-9686b8b46-nklrk" Sep 16 07:32:29.504800 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:29.504724 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/88feefad-338d-4fde-b835-c7aba8b1c875-console-oauth-config\") pod \"console-9686b8b46-nklrk\" (UID: \"88feefad-338d-4fde-b835-c7aba8b1c875\") " pod="openshift-console/console-9686b8b46-nklrk" Sep 16 07:32:29.510900 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:29.510882 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zqpp\" (UniqueName: \"kubernetes.io/projected/88feefad-338d-4fde-b835-c7aba8b1c875-kube-api-access-6zqpp\") pod \"console-9686b8b46-nklrk\" (UID: \"88feefad-338d-4fde-b835-c7aba8b1c875\") " pod="openshift-console/console-9686b8b46-nklrk" Sep 16 07:32:29.697412 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:29.697381 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9686b8b46-nklrk" Sep 16 07:32:29.970745 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:29.970684 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-9686b8b46-nklrk"] Sep 16 07:32:29.973338 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:32:29.973306 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88feefad_338d_4fde_b835_c7aba8b1c875.slice/crio-57032f7f199ff2ba2011c2d06855b490e9533d36bdfa784812c33e5c173f82b8 WatchSource:0}: Error finding container 57032f7f199ff2ba2011c2d06855b490e9533d36bdfa784812c33e5c173f82b8: Status 404 returned error can't find the container with id 57032f7f199ff2ba2011c2d06855b490e9533d36bdfa784812c33e5c173f82b8 Sep 16 07:32:30.359472 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:30.359395 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9686b8b46-nklrk" event={"ID":"88feefad-338d-4fde-b835-c7aba8b1c875","Type":"ContainerStarted","Data":"57032f7f199ff2ba2011c2d06855b490e9533d36bdfa784812c33e5c173f82b8"} Sep 16 07:32:30.911369 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:30.911332 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/44a442da-5d1a-4b97-8697-3ce085773de6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xq64b\" (UID: \"44a442da-5d1a-4b97-8697-3ce085773de6\") " pod="openshift-insights/insights-runtime-extractor-xq64b" Sep 16 07:32:30.913760 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:30.913737 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/44a442da-5d1a-4b97-8697-3ce085773de6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xq64b\" (UID: \"44a442da-5d1a-4b97-8697-3ce085773de6\") " pod="openshift-insights/insights-runtime-extractor-xq64b" Sep 16 07:32:31.164647 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:31.164570 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-xq64b" Sep 16 07:32:31.212953 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:31.212916 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dbc6f54b-f7e7-4ede-b9a2-58f81e7a3f4c-metrics-tls\") pod \"dns-default-v8n9m\" (UID: \"dbc6f54b-f7e7-4ede-b9a2-58f81e7a3f4c\") " pod="openshift-dns/dns-default-v8n9m" Sep 16 07:32:31.215876 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:31.215786 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dbc6f54b-f7e7-4ede-b9a2-58f81e7a3f4c-metrics-tls\") pod \"dns-default-v8n9m\" (UID: \"dbc6f54b-f7e7-4ede-b9a2-58f81e7a3f4c\") " pod="openshift-dns/dns-default-v8n9m" Sep 16 07:32:31.356372 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:31.356345 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-8grdh\"" Sep 16 07:32:31.364321 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:31.364295 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-v8n9m" Sep 16 07:32:31.442401 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:31.442368 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-xq64b"] Sep 16 07:32:31.444901 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:32:31.444874 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44a442da_5d1a_4b97_8697_3ce085773de6.slice/crio-a55835ca9d66f2b8da61406175220d130d22dd145bffd80c3d10901cef766c0e WatchSource:0}: Error finding container a55835ca9d66f2b8da61406175220d130d22dd145bffd80c3d10901cef766c0e: Status 404 returned error can't find the container with id a55835ca9d66f2b8da61406175220d130d22dd145bffd80c3d10901cef766c0e Sep 16 07:32:31.645091 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:31.645064 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-v8n9m"] Sep 16 07:32:31.647931 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:32:31.647906 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbc6f54b_f7e7_4ede_b9a2_58f81e7a3f4c.slice/crio-c4a6ceec334b93a43e23d85d182b06013ac8c813059f1569dee1dc81e9881aeb WatchSource:0}: Error finding container c4a6ceec334b93a43e23d85d182b06013ac8c813059f1569dee1dc81e9881aeb: Status 404 returned error can't find the container with id c4a6ceec334b93a43e23d85d182b06013ac8c813059f1569dee1dc81e9881aeb Sep 16 07:32:32.367836 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:32.367792 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-v8n9m" event={"ID":"dbc6f54b-f7e7-4ede-b9a2-58f81e7a3f4c","Type":"ContainerStarted","Data":"c4a6ceec334b93a43e23d85d182b06013ac8c813059f1569dee1dc81e9881aeb"} Sep 16 07:32:32.369473 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:32.369449 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xq64b" event={"ID":"44a442da-5d1a-4b97-8697-3ce085773de6","Type":"ContainerStarted","Data":"d510019a4327471f951651fb4ab38babc4173bf60086de280906dcb5c3068faa"} Sep 16 07:32:32.369592 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:32.369500 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xq64b" event={"ID":"44a442da-5d1a-4b97-8697-3ce085773de6","Type":"ContainerStarted","Data":"a55835ca9d66f2b8da61406175220d130d22dd145bffd80c3d10901cef766c0e"} Sep 16 07:32:33.381314 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:33.381220 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9686b8b46-nklrk" event={"ID":"88feefad-338d-4fde-b835-c7aba8b1c875","Type":"ContainerStarted","Data":"7001bf5b847388a46f0b4cdfdaf52230a53c2800444ed333ae766288727e167b"} Sep 16 07:32:33.383791 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:33.383757 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xq64b" event={"ID":"44a442da-5d1a-4b97-8697-3ce085773de6","Type":"ContainerStarted","Data":"4636a77aab5768a8de40b5f8b77e0d5cb8dd126390e1c90f03d5b3e8d96db7ac"} Sep 16 07:32:33.399558 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:33.399514 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-9686b8b46-nklrk" podStartSLOduration=1.676698568 podStartE2EDuration="4.399497679s" totalImagesPullingTime="2.722799111s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:32:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:32:33.399045964 +0000 UTC m=+286.425946991" watchObservedRunningTime="2026-09-16 07:32:33.399497679 +0000 UTC m=+286.426398691" Sep 16 07:32:35.391704 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:35.391664 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xq64b" event={"ID":"44a442da-5d1a-4b97-8697-3ce085773de6","Type":"ContainerStarted","Data":"9ed8297739a757c8083698921613f4a2222e241f52ee9f6722a4e9fd185bb899"} Sep 16 07:32:35.393569 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:35.393541 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-v8n9m" event={"ID":"dbc6f54b-f7e7-4ede-b9a2-58f81e7a3f4c","Type":"ContainerStarted","Data":"29996a695eed671b5eda9c8d5b56eb29f08b8a33150594c8f13453182fd5c90d"} Sep 16 07:32:35.393682 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:35.393596 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-v8n9m" event={"ID":"dbc6f54b-f7e7-4ede-b9a2-58f81e7a3f4c","Type":"ContainerStarted","Data":"d19dacecc262f7f5c87e2916fbf486c51ed8a2761e2960c331f29558f46c5ca7"} Sep 16 07:32:35.393729 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:35.393679 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-v8n9m" Sep 16 07:32:35.409538 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:35.409495 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-xq64b" podStartSLOduration=33.619789936 podStartE2EDuration="36.40948046s" totalImagesPullingTime="2.789690524s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 07:31:59 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:32:35.408907399 +0000 UTC m=+288.435808424" watchObservedRunningTime="2026-09-16 07:32:35.40948046 +0000 UTC m=+288.436381465" Sep 16 07:32:35.426497 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:35.426443 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-v8n9m" podStartSLOduration=251.720518398 podStartE2EDuration="4m14.426428445s" totalImagesPullingTime="2.705910047s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:28:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:32:35.425121292 +0000 UTC m=+288.452022317" watchObservedRunningTime="2026-09-16 07:32:35.426428445 +0000 UTC m=+288.453329456" Sep 16 07:32:37.940459 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:37.940419 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-mkpdq"] Sep 16 07:32:37.943852 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:37.943822 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-mkpdq" Sep 16 07:32:37.946320 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:37.946291 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Sep 16 07:32:37.946581 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:37.946549 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-zhzvx\"" Sep 16 07:32:37.946581 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:37.946564 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Sep 16 07:32:37.946680 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:37.946574 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 16 07:32:37.947694 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:37.947676 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Sep 16 07:32:37.947771 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:37.947678 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 16 07:32:37.947771 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:37.947714 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 16 07:32:37.958848 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:37.958825 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-mkpdq"] Sep 16 07:32:37.959990 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:37.959970 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/5e8b982e-e811-49dd-9ca7-fcdbd0cf36d1-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-mkpdq\" (UID: \"5e8b982e-e811-49dd-9ca7-fcdbd0cf36d1\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-mkpdq" Sep 16 07:32:37.960048 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:37.960005 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5e8b982e-e811-49dd-9ca7-fcdbd0cf36d1-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-mkpdq\" (UID: \"5e8b982e-e811-49dd-9ca7-fcdbd0cf36d1\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-mkpdq" Sep 16 07:32:37.960048 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:37.960035 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88n6p\" (UniqueName: \"kubernetes.io/projected/5e8b982e-e811-49dd-9ca7-fcdbd0cf36d1-kube-api-access-88n6p\") pod \"kube-state-metrics-59fbdd77d-mkpdq\" (UID: \"5e8b982e-e811-49dd-9ca7-fcdbd0cf36d1\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-mkpdq" Sep 16 07:32:37.960119 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:37.960058 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5e8b982e-e811-49dd-9ca7-fcdbd0cf36d1-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-mkpdq\" (UID: \"5e8b982e-e811-49dd-9ca7-fcdbd0cf36d1\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-mkpdq" Sep 16 07:32:37.960119 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:37.960080 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/5e8b982e-e811-49dd-9ca7-fcdbd0cf36d1-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-mkpdq\" (UID: \"5e8b982e-e811-49dd-9ca7-fcdbd0cf36d1\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-mkpdq" Sep 16 07:32:37.960248 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:37.960209 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5e8b982e-e811-49dd-9ca7-fcdbd0cf36d1-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-mkpdq\" (UID: \"5e8b982e-e811-49dd-9ca7-fcdbd0cf36d1\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-mkpdq" Sep 16 07:32:37.964294 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:37.964263 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-5tdrc"] Sep 16 07:32:37.967599 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:37.967580 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-5tdrc" Sep 16 07:32:37.970387 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:37.970367 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 16 07:32:37.970483 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:37.970367 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 16 07:32:37.970610 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:37.970592 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 16 07:32:37.970859 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:37.970844 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-d7lb2\"" Sep 16 07:32:38.060459 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:38.060429 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5e8b982e-e811-49dd-9ca7-fcdbd0cf36d1-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-mkpdq\" (UID: \"5e8b982e-e811-49dd-9ca7-fcdbd0cf36d1\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-mkpdq" Sep 16 07:32:38.060459 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:38.060457 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e0c3b327-c203-42b9-92ab-7ee31f7153ba-root\") pod \"node-exporter-5tdrc\" (UID: \"e0c3b327-c203-42b9-92ab-7ee31f7153ba\") " pod="openshift-monitoring/node-exporter-5tdrc" Sep 16 07:32:38.060646 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:38.060474 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e0c3b327-c203-42b9-92ab-7ee31f7153ba-metrics-client-ca\") pod \"node-exporter-5tdrc\" (UID: \"e0c3b327-c203-42b9-92ab-7ee31f7153ba\") " pod="openshift-monitoring/node-exporter-5tdrc" Sep 16 07:32:38.060646 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:38.060499 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88n6p\" (UniqueName: \"kubernetes.io/projected/5e8b982e-e811-49dd-9ca7-fcdbd0cf36d1-kube-api-access-88n6p\") pod \"kube-state-metrics-59fbdd77d-mkpdq\" (UID: \"5e8b982e-e811-49dd-9ca7-fcdbd0cf36d1\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-mkpdq" Sep 16 07:32:38.060646 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:38.060551 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e0c3b327-c203-42b9-92ab-7ee31f7153ba-sys\") pod \"node-exporter-5tdrc\" (UID: \"e0c3b327-c203-42b9-92ab-7ee31f7153ba\") " pod="openshift-monitoring/node-exporter-5tdrc" Sep 16 07:32:38.060646 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:38.060578 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5e8b982e-e811-49dd-9ca7-fcdbd0cf36d1-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-mkpdq\" (UID: \"5e8b982e-e811-49dd-9ca7-fcdbd0cf36d1\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-mkpdq" Sep 16 07:32:38.060646 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:38.060636 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/5e8b982e-e811-49dd-9ca7-fcdbd0cf36d1-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-mkpdq\" (UID: \"5e8b982e-e811-49dd-9ca7-fcdbd0cf36d1\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-mkpdq" Sep 16 07:32:38.060897 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:38.060670 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e0c3b327-c203-42b9-92ab-7ee31f7153ba-node-exporter-textfile\") pod \"node-exporter-5tdrc\" (UID: \"e0c3b327-c203-42b9-92ab-7ee31f7153ba\") " pod="openshift-monitoring/node-exporter-5tdrc" Sep 16 07:32:38.060897 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:38.060689 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e0c3b327-c203-42b9-92ab-7ee31f7153ba-node-exporter-wtmp\") pod \"node-exporter-5tdrc\" (UID: \"e0c3b327-c203-42b9-92ab-7ee31f7153ba\") " pod="openshift-monitoring/node-exporter-5tdrc" Sep 16 07:32:38.060897 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:38.060704 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e0c3b327-c203-42b9-92ab-7ee31f7153ba-node-exporter-accelerators-collector-config\") pod \"node-exporter-5tdrc\" (UID: \"e0c3b327-c203-42b9-92ab-7ee31f7153ba\") " pod="openshift-monitoring/node-exporter-5tdrc" Sep 16 07:32:38.060897 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:38.060738 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5e8b982e-e811-49dd-9ca7-fcdbd0cf36d1-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-mkpdq\" (UID: \"5e8b982e-e811-49dd-9ca7-fcdbd0cf36d1\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-mkpdq" Sep 16 07:32:38.060897 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:38.060769 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/5e8b982e-e811-49dd-9ca7-fcdbd0cf36d1-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-mkpdq\" (UID: \"5e8b982e-e811-49dd-9ca7-fcdbd0cf36d1\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-mkpdq" Sep 16 07:32:38.060897 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:38.060785 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e0c3b327-c203-42b9-92ab-7ee31f7153ba-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5tdrc\" (UID: \"e0c3b327-c203-42b9-92ab-7ee31f7153ba\") " pod="openshift-monitoring/node-exporter-5tdrc" Sep 16 07:32:38.060897 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:38.060801 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e0c3b327-c203-42b9-92ab-7ee31f7153ba-node-exporter-tls\") pod \"node-exporter-5tdrc\" (UID: \"e0c3b327-c203-42b9-92ab-7ee31f7153ba\") " pod="openshift-monitoring/node-exporter-5tdrc" Sep 16 07:32:38.060897 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:38.060836 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8lzk\" (UniqueName: \"kubernetes.io/projected/e0c3b327-c203-42b9-92ab-7ee31f7153ba-kube-api-access-n8lzk\") pod \"node-exporter-5tdrc\" (UID: \"e0c3b327-c203-42b9-92ab-7ee31f7153ba\") " pod="openshift-monitoring/node-exporter-5tdrc" Sep 16 07:32:38.061292 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:38.061152 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/5e8b982e-e811-49dd-9ca7-fcdbd0cf36d1-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-mkpdq\" (UID: \"5e8b982e-e811-49dd-9ca7-fcdbd0cf36d1\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-mkpdq" Sep 16 07:32:38.061372 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:38.061352 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/5e8b982e-e811-49dd-9ca7-fcdbd0cf36d1-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-mkpdq\" (UID: \"5e8b982e-e811-49dd-9ca7-fcdbd0cf36d1\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-mkpdq" Sep 16 07:32:38.061457 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:38.061438 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5e8b982e-e811-49dd-9ca7-fcdbd0cf36d1-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-mkpdq\" (UID: \"5e8b982e-e811-49dd-9ca7-fcdbd0cf36d1\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-mkpdq" Sep 16 07:32:38.062690 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:38.062668 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5e8b982e-e811-49dd-9ca7-fcdbd0cf36d1-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-mkpdq\" (UID: \"5e8b982e-e811-49dd-9ca7-fcdbd0cf36d1\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-mkpdq" Sep 16 07:32:38.062796 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:38.062780 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5e8b982e-e811-49dd-9ca7-fcdbd0cf36d1-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-mkpdq\" (UID: \"5e8b982e-e811-49dd-9ca7-fcdbd0cf36d1\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-mkpdq" Sep 16 07:32:38.072643 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:38.072619 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-88n6p\" (UniqueName: \"kubernetes.io/projected/5e8b982e-e811-49dd-9ca7-fcdbd0cf36d1-kube-api-access-88n6p\") pod \"kube-state-metrics-59fbdd77d-mkpdq\" (UID: \"5e8b982e-e811-49dd-9ca7-fcdbd0cf36d1\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-mkpdq" Sep 16 07:32:38.161720 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:38.161700 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e0c3b327-c203-42b9-92ab-7ee31f7153ba-node-exporter-textfile\") pod \"node-exporter-5tdrc\" (UID: \"e0c3b327-c203-42b9-92ab-7ee31f7153ba\") " pod="openshift-monitoring/node-exporter-5tdrc" Sep 16 07:32:38.161799 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:38.161725 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e0c3b327-c203-42b9-92ab-7ee31f7153ba-node-exporter-wtmp\") pod \"node-exporter-5tdrc\" (UID: \"e0c3b327-c203-42b9-92ab-7ee31f7153ba\") " pod="openshift-monitoring/node-exporter-5tdrc" Sep 16 07:32:38.161799 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:38.161741 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e0c3b327-c203-42b9-92ab-7ee31f7153ba-node-exporter-accelerators-collector-config\") pod \"node-exporter-5tdrc\" (UID: \"e0c3b327-c203-42b9-92ab-7ee31f7153ba\") " pod="openshift-monitoring/node-exporter-5tdrc" Sep 16 07:32:38.161799 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:38.161766 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e0c3b327-c203-42b9-92ab-7ee31f7153ba-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5tdrc\" (UID: \"e0c3b327-c203-42b9-92ab-7ee31f7153ba\") " pod="openshift-monitoring/node-exporter-5tdrc" Sep 16 07:32:38.161799 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:38.161789 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e0c3b327-c203-42b9-92ab-7ee31f7153ba-node-exporter-tls\") pod \"node-exporter-5tdrc\" (UID: \"e0c3b327-c203-42b9-92ab-7ee31f7153ba\") " pod="openshift-monitoring/node-exporter-5tdrc" Sep 16 07:32:38.161986 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:38.161820 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8lzk\" (UniqueName: \"kubernetes.io/projected/e0c3b327-c203-42b9-92ab-7ee31f7153ba-kube-api-access-n8lzk\") pod \"node-exporter-5tdrc\" (UID: \"e0c3b327-c203-42b9-92ab-7ee31f7153ba\") " pod="openshift-monitoring/node-exporter-5tdrc" Sep 16 07:32:38.161986 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:38.161890 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e0c3b327-c203-42b9-92ab-7ee31f7153ba-node-exporter-wtmp\") pod \"node-exporter-5tdrc\" (UID: \"e0c3b327-c203-42b9-92ab-7ee31f7153ba\") " pod="openshift-monitoring/node-exporter-5tdrc" Sep 16 07:32:38.161986 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:38.161932 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e0c3b327-c203-42b9-92ab-7ee31f7153ba-root\") pod \"node-exporter-5tdrc\" (UID: \"e0c3b327-c203-42b9-92ab-7ee31f7153ba\") " pod="openshift-monitoring/node-exporter-5tdrc" Sep 16 07:32:38.161986 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:32:38.161947 2557 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Sep 16 07:32:38.161986 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:38.161957 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e0c3b327-c203-42b9-92ab-7ee31f7153ba-metrics-client-ca\") pod \"node-exporter-5tdrc\" (UID: \"e0c3b327-c203-42b9-92ab-7ee31f7153ba\") " pod="openshift-monitoring/node-exporter-5tdrc" Sep 16 07:32:38.162207 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:32:38.162003 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0c3b327-c203-42b9-92ab-7ee31f7153ba-node-exporter-tls podName:e0c3b327-c203-42b9-92ab-7ee31f7153ba nodeName:}" failed. No retries permitted until 2026-09-16 07:32:38.661986832 +0000 UTC m=+291.688887834 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/e0c3b327-c203-42b9-92ab-7ee31f7153ba-node-exporter-tls") pod "node-exporter-5tdrc" (UID: "e0c3b327-c203-42b9-92ab-7ee31f7153ba") : secret "node-exporter-tls" not found Sep 16 07:32:38.162207 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:38.161999 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e0c3b327-c203-42b9-92ab-7ee31f7153ba-root\") pod \"node-exporter-5tdrc\" (UID: \"e0c3b327-c203-42b9-92ab-7ee31f7153ba\") " pod="openshift-monitoring/node-exporter-5tdrc" Sep 16 07:32:38.162207 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:38.162033 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e0c3b327-c203-42b9-92ab-7ee31f7153ba-node-exporter-textfile\") pod \"node-exporter-5tdrc\" (UID: \"e0c3b327-c203-42b9-92ab-7ee31f7153ba\") " pod="openshift-monitoring/node-exporter-5tdrc" Sep 16 07:32:38.162207 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:38.162052 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e0c3b327-c203-42b9-92ab-7ee31f7153ba-sys\") pod \"node-exporter-5tdrc\" (UID: \"e0c3b327-c203-42b9-92ab-7ee31f7153ba\") " pod="openshift-monitoring/node-exporter-5tdrc" Sep 16 07:32:38.162207 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:38.162132 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e0c3b327-c203-42b9-92ab-7ee31f7153ba-sys\") pod \"node-exporter-5tdrc\" (UID: \"e0c3b327-c203-42b9-92ab-7ee31f7153ba\") " pod="openshift-monitoring/node-exporter-5tdrc" Sep 16 07:32:38.162408 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:38.162354 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e0c3b327-c203-42b9-92ab-7ee31f7153ba-node-exporter-accelerators-collector-config\") pod \"node-exporter-5tdrc\" (UID: \"e0c3b327-c203-42b9-92ab-7ee31f7153ba\") " pod="openshift-monitoring/node-exporter-5tdrc" Sep 16 07:32:38.162458 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:38.162443 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e0c3b327-c203-42b9-92ab-7ee31f7153ba-metrics-client-ca\") pod \"node-exporter-5tdrc\" (UID: \"e0c3b327-c203-42b9-92ab-7ee31f7153ba\") " pod="openshift-monitoring/node-exporter-5tdrc" Sep 16 07:32:38.163688 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:38.163668 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e0c3b327-c203-42b9-92ab-7ee31f7153ba-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5tdrc\" (UID: \"e0c3b327-c203-42b9-92ab-7ee31f7153ba\") " pod="openshift-monitoring/node-exporter-5tdrc" Sep 16 07:32:38.176960 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:38.176941 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8lzk\" (UniqueName: \"kubernetes.io/projected/e0c3b327-c203-42b9-92ab-7ee31f7153ba-kube-api-access-n8lzk\") pod \"node-exporter-5tdrc\" (UID: \"e0c3b327-c203-42b9-92ab-7ee31f7153ba\") " pod="openshift-monitoring/node-exporter-5tdrc" Sep 16 07:32:38.254008 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:38.253955 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-mkpdq" Sep 16 07:32:38.528136 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:38.528079 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-mkpdq"] Sep 16 07:32:38.530724 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:32:38.530702 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e8b982e_e811_49dd_9ca7_fcdbd0cf36d1.slice/crio-f35a1f05e835178aeab54d95696a8c3e892b2c39cf679a6c86e89a428a7132c6 WatchSource:0}: Error finding container f35a1f05e835178aeab54d95696a8c3e892b2c39cf679a6c86e89a428a7132c6: Status 404 returned error can't find the container with id f35a1f05e835178aeab54d95696a8c3e892b2c39cf679a6c86e89a428a7132c6 Sep 16 07:32:38.666276 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:38.666250 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e0c3b327-c203-42b9-92ab-7ee31f7153ba-node-exporter-tls\") pod \"node-exporter-5tdrc\" (UID: \"e0c3b327-c203-42b9-92ab-7ee31f7153ba\") " pod="openshift-monitoring/node-exporter-5tdrc" Sep 16 07:32:38.668201 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:38.668185 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e0c3b327-c203-42b9-92ab-7ee31f7153ba-node-exporter-tls\") pod \"node-exporter-5tdrc\" (UID: \"e0c3b327-c203-42b9-92ab-7ee31f7153ba\") " pod="openshift-monitoring/node-exporter-5tdrc" Sep 16 07:32:38.880170 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:38.880118 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-5tdrc" Sep 16 07:32:38.890735 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:32:38.890712 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0c3b327_c203_42b9_92ab_7ee31f7153ba.slice/crio-410f0c62f0b742b8135c354618ddccfbbe9110cdb88323a7ae41e9155315be83 WatchSource:0}: Error finding container 410f0c62f0b742b8135c354618ddccfbbe9110cdb88323a7ae41e9155315be83: Status 404 returned error can't find the container with id 410f0c62f0b742b8135c354618ddccfbbe9110cdb88323a7ae41e9155315be83 Sep 16 07:32:39.407529 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:39.407492 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5tdrc" event={"ID":"e0c3b327-c203-42b9-92ab-7ee31f7153ba","Type":"ContainerStarted","Data":"410f0c62f0b742b8135c354618ddccfbbe9110cdb88323a7ae41e9155315be83"} Sep 16 07:32:39.408607 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:39.408588 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-mkpdq" event={"ID":"5e8b982e-e811-49dd-9ca7-fcdbd0cf36d1","Type":"ContainerStarted","Data":"f35a1f05e835178aeab54d95696a8c3e892b2c39cf679a6c86e89a428a7132c6"} Sep 16 07:32:39.697719 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:39.697640 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-9686b8b46-nklrk" Sep 16 07:32:39.697866 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:39.697727 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-9686b8b46-nklrk" Sep 16 07:32:39.702870 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:39.702851 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-9686b8b46-nklrk" Sep 16 07:32:40.414181 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:40.414150 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-mkpdq" event={"ID":"5e8b982e-e811-49dd-9ca7-fcdbd0cf36d1","Type":"ContainerStarted","Data":"6d7cb48461eea1fab1338798681a9543f032e044fd0dfbb537e59104a3797764"} Sep 16 07:32:40.414636 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:40.414211 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-mkpdq" event={"ID":"5e8b982e-e811-49dd-9ca7-fcdbd0cf36d1","Type":"ContainerStarted","Data":"e220131f79dd003f5eaab8cb5b543a7f992a4d95e250c009781d2751a9ad4728"} Sep 16 07:32:40.414636 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:40.414257 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-mkpdq" event={"ID":"5e8b982e-e811-49dd-9ca7-fcdbd0cf36d1","Type":"ContainerStarted","Data":"9ea365b20df1137595f98ff7eafdb35cc67fc57e1cc4738ca509d126a9d33adb"} Sep 16 07:32:40.415858 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:40.415831 2557 generic.go:356] "Generic (PLEG): container finished" podID="e0c3b327-c203-42b9-92ab-7ee31f7153ba" containerID="4310e59751eb0cc2d037dc7fed5d1005e7f3882c0d6be2b757edde77c386e96e" exitCode=0 Sep 16 07:32:40.415977 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:40.415889 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5tdrc" event={"ID":"e0c3b327-c203-42b9-92ab-7ee31f7153ba","Type":"ContainerDied","Data":"4310e59751eb0cc2d037dc7fed5d1005e7f3882c0d6be2b757edde77c386e96e"} Sep 16 07:32:40.420963 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:40.420941 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-9686b8b46-nklrk" Sep 16 07:32:40.433769 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:40.433728 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-mkpdq" podStartSLOduration=1.9288596139999998 podStartE2EDuration="3.433713495s" totalImagesPullingTime="1.504853881s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:32:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:32:40.432786211 +0000 UTC m=+293.459687407" watchObservedRunningTime="2026-09-16 07:32:40.433713495 +0000 UTC m=+293.460614519" Sep 16 07:32:41.056686 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:41.056661 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-7dc978598b-7j5h4"] Sep 16 07:32:41.062037 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:41.062011 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7dc978598b-7j5h4" Sep 16 07:32:41.064650 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:41.064624 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Sep 16 07:32:41.064814 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:41.064771 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Sep 16 07:32:41.065023 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:41.065004 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-3snkpb0muhkad\"" Sep 16 07:32:41.065023 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:41.065012 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Sep 16 07:32:41.065170 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:41.065108 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Sep 16 07:32:41.065170 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:41.065111 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-flcs6\"" Sep 16 07:32:41.065170 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:41.065155 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Sep 16 07:32:41.073183 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:41.073162 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7dc978598b-7j5h4"] Sep 16 07:32:41.085243 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:41.085220 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7ca2b45d-2c2d-4d42-8f53-285ff2febfdc-metrics-client-ca\") pod \"thanos-querier-7dc978598b-7j5h4\" (UID: \"7ca2b45d-2c2d-4d42-8f53-285ff2febfdc\") " pod="openshift-monitoring/thanos-querier-7dc978598b-7j5h4" Sep 16 07:32:41.085330 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:41.085297 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7ca2b45d-2c2d-4d42-8f53-285ff2febfdc-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7dc978598b-7j5h4\" (UID: \"7ca2b45d-2c2d-4d42-8f53-285ff2febfdc\") " pod="openshift-monitoring/thanos-querier-7dc978598b-7j5h4" Sep 16 07:32:41.085330 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:41.085323 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7ca2b45d-2c2d-4d42-8f53-285ff2febfdc-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7dc978598b-7j5h4\" (UID: \"7ca2b45d-2c2d-4d42-8f53-285ff2febfdc\") " pod="openshift-monitoring/thanos-querier-7dc978598b-7j5h4" Sep 16 07:32:41.085400 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:41.085340 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/7ca2b45d-2c2d-4d42-8f53-285ff2febfdc-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7dc978598b-7j5h4\" (UID: \"7ca2b45d-2c2d-4d42-8f53-285ff2febfdc\") " pod="openshift-monitoring/thanos-querier-7dc978598b-7j5h4" Sep 16 07:32:41.085400 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:41.085380 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/7ca2b45d-2c2d-4d42-8f53-285ff2febfdc-secret-thanos-querier-tls\") pod \"thanos-querier-7dc978598b-7j5h4\" (UID: \"7ca2b45d-2c2d-4d42-8f53-285ff2febfdc\") " pod="openshift-monitoring/thanos-querier-7dc978598b-7j5h4" Sep 16 07:32:41.085467 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:41.085421 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-675p6\" (UniqueName: \"kubernetes.io/projected/7ca2b45d-2c2d-4d42-8f53-285ff2febfdc-kube-api-access-675p6\") pod \"thanos-querier-7dc978598b-7j5h4\" (UID: \"7ca2b45d-2c2d-4d42-8f53-285ff2febfdc\") " pod="openshift-monitoring/thanos-querier-7dc978598b-7j5h4" Sep 16 07:32:41.085467 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:41.085447 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/7ca2b45d-2c2d-4d42-8f53-285ff2febfdc-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7dc978598b-7j5h4\" (UID: \"7ca2b45d-2c2d-4d42-8f53-285ff2febfdc\") " pod="openshift-monitoring/thanos-querier-7dc978598b-7j5h4" Sep 16 07:32:41.085543 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:41.085470 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7ca2b45d-2c2d-4d42-8f53-285ff2febfdc-secret-grpc-tls\") pod \"thanos-querier-7dc978598b-7j5h4\" (UID: \"7ca2b45d-2c2d-4d42-8f53-285ff2febfdc\") " pod="openshift-monitoring/thanos-querier-7dc978598b-7j5h4" Sep 16 07:32:41.186101 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:41.186081 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7ca2b45d-2c2d-4d42-8f53-285ff2febfdc-metrics-client-ca\") pod \"thanos-querier-7dc978598b-7j5h4\" (UID: \"7ca2b45d-2c2d-4d42-8f53-285ff2febfdc\") " pod="openshift-monitoring/thanos-querier-7dc978598b-7j5h4" Sep 16 07:32:41.186182 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:41.186109 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7ca2b45d-2c2d-4d42-8f53-285ff2febfdc-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7dc978598b-7j5h4\" (UID: \"7ca2b45d-2c2d-4d42-8f53-285ff2febfdc\") " pod="openshift-monitoring/thanos-querier-7dc978598b-7j5h4" Sep 16 07:32:41.186182 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:41.186128 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7ca2b45d-2c2d-4d42-8f53-285ff2febfdc-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7dc978598b-7j5h4\" (UID: \"7ca2b45d-2c2d-4d42-8f53-285ff2febfdc\") " pod="openshift-monitoring/thanos-querier-7dc978598b-7j5h4" Sep 16 07:32:41.186182 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:41.186145 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/7ca2b45d-2c2d-4d42-8f53-285ff2febfdc-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7dc978598b-7j5h4\" (UID: \"7ca2b45d-2c2d-4d42-8f53-285ff2febfdc\") " pod="openshift-monitoring/thanos-querier-7dc978598b-7j5h4" Sep 16 07:32:41.186182 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:41.186166 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/7ca2b45d-2c2d-4d42-8f53-285ff2febfdc-secret-thanos-querier-tls\") pod \"thanos-querier-7dc978598b-7j5h4\" (UID: \"7ca2b45d-2c2d-4d42-8f53-285ff2febfdc\") " pod="openshift-monitoring/thanos-querier-7dc978598b-7j5h4" Sep 16 07:32:41.186400 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:41.186217 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-675p6\" (UniqueName: \"kubernetes.io/projected/7ca2b45d-2c2d-4d42-8f53-285ff2febfdc-kube-api-access-675p6\") pod \"thanos-querier-7dc978598b-7j5h4\" (UID: \"7ca2b45d-2c2d-4d42-8f53-285ff2febfdc\") " pod="openshift-monitoring/thanos-querier-7dc978598b-7j5h4" Sep 16 07:32:41.186400 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:41.186264 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/7ca2b45d-2c2d-4d42-8f53-285ff2febfdc-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7dc978598b-7j5h4\" (UID: \"7ca2b45d-2c2d-4d42-8f53-285ff2febfdc\") " pod="openshift-monitoring/thanos-querier-7dc978598b-7j5h4" Sep 16 07:32:41.186400 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:41.186291 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7ca2b45d-2c2d-4d42-8f53-285ff2febfdc-secret-grpc-tls\") pod \"thanos-querier-7dc978598b-7j5h4\" (UID: \"7ca2b45d-2c2d-4d42-8f53-285ff2febfdc\") " pod="openshift-monitoring/thanos-querier-7dc978598b-7j5h4" Sep 16 07:32:41.186913 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:41.186894 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7ca2b45d-2c2d-4d42-8f53-285ff2febfdc-metrics-client-ca\") pod \"thanos-querier-7dc978598b-7j5h4\" (UID: \"7ca2b45d-2c2d-4d42-8f53-285ff2febfdc\") " pod="openshift-monitoring/thanos-querier-7dc978598b-7j5h4" Sep 16 07:32:41.189282 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:41.189258 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/7ca2b45d-2c2d-4d42-8f53-285ff2febfdc-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7dc978598b-7j5h4\" (UID: \"7ca2b45d-2c2d-4d42-8f53-285ff2febfdc\") " pod="openshift-monitoring/thanos-querier-7dc978598b-7j5h4" Sep 16 07:32:41.189282 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:41.189268 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7ca2b45d-2c2d-4d42-8f53-285ff2febfdc-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7dc978598b-7j5h4\" (UID: \"7ca2b45d-2c2d-4d42-8f53-285ff2febfdc\") " pod="openshift-monitoring/thanos-querier-7dc978598b-7j5h4" Sep 16 07:32:41.189470 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:41.189446 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/7ca2b45d-2c2d-4d42-8f53-285ff2febfdc-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7dc978598b-7j5h4\" (UID: \"7ca2b45d-2c2d-4d42-8f53-285ff2febfdc\") " pod="openshift-monitoring/thanos-querier-7dc978598b-7j5h4" Sep 16 07:32:41.189533 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:41.189468 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7ca2b45d-2c2d-4d42-8f53-285ff2febfdc-secret-grpc-tls\") pod \"thanos-querier-7dc978598b-7j5h4\" (UID: \"7ca2b45d-2c2d-4d42-8f53-285ff2febfdc\") " pod="openshift-monitoring/thanos-querier-7dc978598b-7j5h4" Sep 16 07:32:41.189533 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:41.189511 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/7ca2b45d-2c2d-4d42-8f53-285ff2febfdc-secret-thanos-querier-tls\") pod \"thanos-querier-7dc978598b-7j5h4\" (UID: \"7ca2b45d-2c2d-4d42-8f53-285ff2febfdc\") " pod="openshift-monitoring/thanos-querier-7dc978598b-7j5h4" Sep 16 07:32:41.189533 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:41.189525 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7ca2b45d-2c2d-4d42-8f53-285ff2febfdc-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7dc978598b-7j5h4\" (UID: \"7ca2b45d-2c2d-4d42-8f53-285ff2febfdc\") " pod="openshift-monitoring/thanos-querier-7dc978598b-7j5h4" Sep 16 07:32:41.193640 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:41.193618 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-675p6\" (UniqueName: \"kubernetes.io/projected/7ca2b45d-2c2d-4d42-8f53-285ff2febfdc-kube-api-access-675p6\") pod \"thanos-querier-7dc978598b-7j5h4\" (UID: \"7ca2b45d-2c2d-4d42-8f53-285ff2febfdc\") " pod="openshift-monitoring/thanos-querier-7dc978598b-7j5h4" Sep 16 07:32:41.370932 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:41.370863 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7dc978598b-7j5h4" Sep 16 07:32:41.428301 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:41.425360 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5tdrc" event={"ID":"e0c3b327-c203-42b9-92ab-7ee31f7153ba","Type":"ContainerStarted","Data":"7838bf714d30417c196cb84685143d6bab6d1c1e14e548a75ab4a1354469a1da"} Sep 16 07:32:41.428301 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:41.425468 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5tdrc" event={"ID":"e0c3b327-c203-42b9-92ab-7ee31f7153ba","Type":"ContainerStarted","Data":"6e7d4b845cf792b294188aed8d043d6e656a0dc9e2e06e02a60d2f5b596184bb"} Sep 16 07:32:41.448999 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:41.448948 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-5tdrc" podStartSLOduration=3.230466066 podStartE2EDuration="4.44893436s" totalImagesPullingTime="1.146940713s" totalInitContainerRuntime="71.527581ms" isStatefulPod=true podCreationTimestamp="2026-09-16 07:32:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:32:41.447867573 +0000 UTC m=+294.474768608" watchObservedRunningTime="2026-09-16 07:32:41.44893436 +0000 UTC m=+294.475835370" Sep 16 07:32:41.520280 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:32:41.520253 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-kqc7r" podUID="296c5dcf-403c-4a1f-9d6f-05ffb7d75e38" Sep 16 07:32:41.648255 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:41.648211 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7dc978598b-7j5h4"] Sep 16 07:32:41.651406 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:32:41.651375 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ca2b45d_2c2d_4d42_8f53_285ff2febfdc.slice/crio-5321ba89005b451e341652dca15e03df50806a207691faf7dfe3091dba5deec6 WatchSource:0}: Error finding container 5321ba89005b451e341652dca15e03df50806a207691faf7dfe3091dba5deec6: Status 404 returned error can't find the container with id 5321ba89005b451e341652dca15e03df50806a207691faf7dfe3091dba5deec6 Sep 16 07:32:42.429972 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:42.429940 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7dc978598b-7j5h4" event={"ID":"7ca2b45d-2c2d-4d42-8f53-285ff2febfdc","Type":"ContainerStarted","Data":"5321ba89005b451e341652dca15e03df50806a207691faf7dfe3091dba5deec6"} Sep 16 07:32:42.734808 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:42.734735 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-tzgsd"] Sep 16 07:32:42.738061 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:42.738043 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-tzgsd" Sep 16 07:32:42.740164 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:42.740135 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-x7c2q\"" Sep 16 07:32:42.740295 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:42.740262 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Sep 16 07:32:42.749394 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:42.749374 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-tzgsd"] Sep 16 07:32:42.799664 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:42.799643 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdp9c\" (UniqueName: \"kubernetes.io/projected/d31607fb-50b7-4280-a7bc-b7242a48207c-kube-api-access-zdp9c\") pod \"monitoring-plugin-5d85778b9d-tzgsd\" (UID: \"d31607fb-50b7-4280-a7bc-b7242a48207c\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-tzgsd" Sep 16 07:32:42.799785 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:42.799672 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d31607fb-50b7-4280-a7bc-b7242a48207c-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-tzgsd\" (UID: \"d31607fb-50b7-4280-a7bc-b7242a48207c\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-tzgsd" Sep 16 07:32:42.900815 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:42.900784 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d31607fb-50b7-4280-a7bc-b7242a48207c-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-tzgsd\" (UID: \"d31607fb-50b7-4280-a7bc-b7242a48207c\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-tzgsd" Sep 16 07:32:42.900925 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:42.900897 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdp9c\" (UniqueName: \"kubernetes.io/projected/d31607fb-50b7-4280-a7bc-b7242a48207c-kube-api-access-zdp9c\") pod \"monitoring-plugin-5d85778b9d-tzgsd\" (UID: \"d31607fb-50b7-4280-a7bc-b7242a48207c\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-tzgsd" Sep 16 07:32:42.900972 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:32:42.900927 2557 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Sep 16 07:32:42.901022 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:32:42.900980 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d31607fb-50b7-4280-a7bc-b7242a48207c-monitoring-plugin-cert podName:d31607fb-50b7-4280-a7bc-b7242a48207c nodeName:}" failed. No retries permitted until 2026-09-16 07:32:43.400965077 +0000 UTC m=+296.427866066 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/d31607fb-50b7-4280-a7bc-b7242a48207c-monitoring-plugin-cert") pod "monitoring-plugin-5d85778b9d-tzgsd" (UID: "d31607fb-50b7-4280-a7bc-b7242a48207c") : secret "monitoring-plugin-cert" not found Sep 16 07:32:42.911616 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:42.911591 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdp9c\" (UniqueName: \"kubernetes.io/projected/d31607fb-50b7-4280-a7bc-b7242a48207c-kube-api-access-zdp9c\") pod \"monitoring-plugin-5d85778b9d-tzgsd\" (UID: \"d31607fb-50b7-4280-a7bc-b7242a48207c\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-tzgsd" Sep 16 07:32:43.406220 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:43.406190 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d31607fb-50b7-4280-a7bc-b7242a48207c-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-tzgsd\" (UID: \"d31607fb-50b7-4280-a7bc-b7242a48207c\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-tzgsd" Sep 16 07:32:43.408354 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:43.408332 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d31607fb-50b7-4280-a7bc-b7242a48207c-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-tzgsd\" (UID: \"d31607fb-50b7-4280-a7bc-b7242a48207c\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-tzgsd" Sep 16 07:32:43.648309 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:43.648274 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-tzgsd" Sep 16 07:32:43.922148 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:43.919618 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-tzgsd"] Sep 16 07:32:44.439433 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:44.439398 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7dc978598b-7j5h4" event={"ID":"7ca2b45d-2c2d-4d42-8f53-285ff2febfdc","Type":"ContainerStarted","Data":"9fc350e8ebe8e68025d00b6eb84e78369c0804d8dda2a51fa4ec022b6395d3ad"} Sep 16 07:32:44.439604 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:44.439477 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7dc978598b-7j5h4" event={"ID":"7ca2b45d-2c2d-4d42-8f53-285ff2febfdc","Type":"ContainerStarted","Data":"d8be69a1b41e6c76835c06e27748439af9d1eec7a998105ecac1cdaeb7dc7715"} Sep 16 07:32:44.439604 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:44.439506 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7dc978598b-7j5h4" event={"ID":"7ca2b45d-2c2d-4d42-8f53-285ff2febfdc","Type":"ContainerStarted","Data":"cdaa1f23d49102bb3f8dd6402719a0397f5127a750e05062fcf94dba0ac41d15"} Sep 16 07:32:44.440771 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:44.440738 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-tzgsd" event={"ID":"d31607fb-50b7-4280-a7bc-b7242a48207c","Type":"ContainerStarted","Data":"b6c6b25019457c19ff3d41c6fb1fa0aafc45eb941a309063f68da515b86ebe57"} Sep 16 07:32:45.398717 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:45.398695 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-v8n9m" Sep 16 07:32:45.447823 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:45.447786 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7dc978598b-7j5h4" event={"ID":"7ca2b45d-2c2d-4d42-8f53-285ff2febfdc","Type":"ContainerStarted","Data":"5fb93b7e7f51cfe30804215a3489cb819514ea3841604d572b3f1124959ed686"} Sep 16 07:32:45.447959 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:45.447867 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7dc978598b-7j5h4" event={"ID":"7ca2b45d-2c2d-4d42-8f53-285ff2febfdc","Type":"ContainerStarted","Data":"6f6351d34c240427fe153c4441638aee23a0d9a0f2f52a43303489ef0165aa55"} Sep 16 07:32:45.447959 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:45.447910 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7dc978598b-7j5h4" event={"ID":"7ca2b45d-2c2d-4d42-8f53-285ff2febfdc","Type":"ContainerStarted","Data":"104ea4125f4996e0bef12b1aefb79a38103077080655aac9dd561bf236063342"} Sep 16 07:32:45.447959 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:45.447947 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-7dc978598b-7j5h4" Sep 16 07:32:45.449401 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:45.449382 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-tzgsd" event={"ID":"d31607fb-50b7-4280-a7bc-b7242a48207c","Type":"ContainerStarted","Data":"fb27c45b68debc4b682a710e63f9d29337a74b63dd3c82428dfa1cb91fdd7103"} Sep 16 07:32:45.449498 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:45.449480 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-tzgsd" Sep 16 07:32:45.453835 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:45.453817 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-tzgsd" Sep 16 07:32:45.471299 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:45.471261 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-7dc978598b-7j5h4" podStartSLOduration=1.77580096 podStartE2EDuration="4.471230846s" totalImagesPullingTime="2.695429886s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:32:41 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:32:45.470008979 +0000 UTC m=+298.496910006" watchObservedRunningTime="2026-09-16 07:32:45.471230846 +0000 UTC m=+298.498131861" Sep 16 07:32:45.485880 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:45.485846 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-tzgsd" podStartSLOduration=2.084462769 podStartE2EDuration="3.485833526s" totalImagesPullingTime="1.401370757s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:32:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:32:45.484716143 +0000 UTC m=+298.511617171" watchObservedRunningTime="2026-09-16 07:32:45.485833526 +0000 UTC m=+298.512734535" Sep 16 07:32:47.412050 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:47.412027 2557 kubelet.go:1648] "Image garbage collection succeeded" Sep 16 07:32:51.459066 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:51.459032 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-7dc978598b-7j5h4" Sep 16 07:32:53.518719 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:53.518688 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kqc7r" Sep 16 07:32:53.922555 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:32:53.922519 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-9686b8b46-nklrk"] Sep 16 07:33:18.941599 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:33:18.941527 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-9686b8b46-nklrk" podUID="88feefad-338d-4fde-b835-c7aba8b1c875" containerName="console" containerID="cri-o://7001bf5b847388a46f0b4cdfdaf52230a53c2800444ed333ae766288727e167b" gracePeriod=15 Sep 16 07:33:19.171726 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:33:19.171700 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9686b8b46-nklrk" Sep 16 07:33:19.244699 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:33:19.244648 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/88feefad-338d-4fde-b835-c7aba8b1c875-console-serving-cert\") pod \"88feefad-338d-4fde-b835-c7aba8b1c875\" (UID: \"88feefad-338d-4fde-b835-c7aba8b1c875\") " Sep 16 07:33:19.244699 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:33:19.244677 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6zqpp\" (UniqueName: \"kubernetes.io/projected/88feefad-338d-4fde-b835-c7aba8b1c875-kube-api-access-6zqpp\") pod \"88feefad-338d-4fde-b835-c7aba8b1c875\" (UID: \"88feefad-338d-4fde-b835-c7aba8b1c875\") " Sep 16 07:33:19.244873 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:33:19.244704 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/88feefad-338d-4fde-b835-c7aba8b1c875-console-oauth-config\") pod \"88feefad-338d-4fde-b835-c7aba8b1c875\" (UID: \"88feefad-338d-4fde-b835-c7aba8b1c875\") " Sep 16 07:33:19.244873 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:33:19.244720 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/88feefad-338d-4fde-b835-c7aba8b1c875-service-ca\") pod \"88feefad-338d-4fde-b835-c7aba8b1c875\" (UID: \"88feefad-338d-4fde-b835-c7aba8b1c875\") " Sep 16 07:33:19.244873 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:33:19.244748 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/88feefad-338d-4fde-b835-c7aba8b1c875-console-config\") pod \"88feefad-338d-4fde-b835-c7aba8b1c875\" (UID: \"88feefad-338d-4fde-b835-c7aba8b1c875\") " Sep 16 07:33:19.244873 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:33:19.244826 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/88feefad-338d-4fde-b835-c7aba8b1c875-oauth-serving-cert\") pod \"88feefad-338d-4fde-b835-c7aba8b1c875\" (UID: \"88feefad-338d-4fde-b835-c7aba8b1c875\") " Sep 16 07:33:19.245141 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:33:19.245105 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88feefad-338d-4fde-b835-c7aba8b1c875-service-ca" (OuterVolumeSpecName: "service-ca") pod "88feefad-338d-4fde-b835-c7aba8b1c875" (UID: "88feefad-338d-4fde-b835-c7aba8b1c875"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 07:33:19.245276 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:33:19.245157 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88feefad-338d-4fde-b835-c7aba8b1c875-console-config" (OuterVolumeSpecName: "console-config") pod "88feefad-338d-4fde-b835-c7aba8b1c875" (UID: "88feefad-338d-4fde-b835-c7aba8b1c875"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 07:33:19.245276 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:33:19.245216 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88feefad-338d-4fde-b835-c7aba8b1c875-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "88feefad-338d-4fde-b835-c7aba8b1c875" (UID: "88feefad-338d-4fde-b835-c7aba8b1c875"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 07:33:19.246799 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:33:19.246779 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88feefad-338d-4fde-b835-c7aba8b1c875-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "88feefad-338d-4fde-b835-c7aba8b1c875" (UID: "88feefad-338d-4fde-b835-c7aba8b1c875"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 07:33:19.247209 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:33:19.247187 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88feefad-338d-4fde-b835-c7aba8b1c875-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "88feefad-338d-4fde-b835-c7aba8b1c875" (UID: "88feefad-338d-4fde-b835-c7aba8b1c875"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 07:33:19.247271 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:33:19.247198 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88feefad-338d-4fde-b835-c7aba8b1c875-kube-api-access-6zqpp" (OuterVolumeSpecName: "kube-api-access-6zqpp") pod "88feefad-338d-4fde-b835-c7aba8b1c875" (UID: "88feefad-338d-4fde-b835-c7aba8b1c875"). InnerVolumeSpecName "kube-api-access-6zqpp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 07:33:19.345820 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:33:19.345800 2557 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/88feefad-338d-4fde-b835-c7aba8b1c875-oauth-serving-cert\") on node \"ip-10-0-141-221.ec2.internal\" DevicePath \"\"" Sep 16 07:33:19.345820 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:33:19.345819 2557 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/88feefad-338d-4fde-b835-c7aba8b1c875-console-serving-cert\") on node \"ip-10-0-141-221.ec2.internal\" DevicePath \"\"" Sep 16 07:33:19.345928 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:33:19.345828 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zqpp\" (UniqueName: \"kubernetes.io/projected/88feefad-338d-4fde-b835-c7aba8b1c875-kube-api-access-6zqpp\") on node \"ip-10-0-141-221.ec2.internal\" DevicePath \"\"" Sep 16 07:33:19.345928 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:33:19.345837 2557 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/88feefad-338d-4fde-b835-c7aba8b1c875-console-oauth-config\") on node \"ip-10-0-141-221.ec2.internal\" DevicePath \"\"" Sep 16 07:33:19.345928 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:33:19.345845 2557 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/88feefad-338d-4fde-b835-c7aba8b1c875-service-ca\") on node \"ip-10-0-141-221.ec2.internal\" DevicePath \"\"" Sep 16 07:33:19.345928 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:33:19.345853 2557 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/88feefad-338d-4fde-b835-c7aba8b1c875-console-config\") on node \"ip-10-0-141-221.ec2.internal\" DevicePath \"\"" Sep 16 07:33:19.567962 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:33:19.567935 2557 generic.go:356] "Generic (PLEG): container finished" podID="88feefad-338d-4fde-b835-c7aba8b1c875" containerID="7001bf5b847388a46f0b4cdfdaf52230a53c2800444ed333ae766288727e167b" exitCode=0 Sep 16 07:33:19.568039 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:33:19.567975 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9686b8b46-nklrk" Sep 16 07:33:19.568039 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:33:19.568005 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9686b8b46-nklrk" event={"ID":"88feefad-338d-4fde-b835-c7aba8b1c875","Type":"ContainerDied","Data":"7001bf5b847388a46f0b4cdfdaf52230a53c2800444ed333ae766288727e167b"} Sep 16 07:33:19.568116 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:33:19.568040 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9686b8b46-nklrk" event={"ID":"88feefad-338d-4fde-b835-c7aba8b1c875","Type":"ContainerDied","Data":"57032f7f199ff2ba2011c2d06855b490e9533d36bdfa784812c33e5c173f82b8"} Sep 16 07:33:19.568116 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:33:19.568070 2557 scope.go:117] "RemoveContainer" containerID="7001bf5b847388a46f0b4cdfdaf52230a53c2800444ed333ae766288727e167b" Sep 16 07:33:19.576433 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:33:19.576418 2557 scope.go:117] "RemoveContainer" containerID="7001bf5b847388a46f0b4cdfdaf52230a53c2800444ed333ae766288727e167b" Sep 16 07:33:19.576708 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:33:19.576691 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7001bf5b847388a46f0b4cdfdaf52230a53c2800444ed333ae766288727e167b\": container with ID starting with 7001bf5b847388a46f0b4cdfdaf52230a53c2800444ed333ae766288727e167b not found: ID does not exist" containerID="7001bf5b847388a46f0b4cdfdaf52230a53c2800444ed333ae766288727e167b" Sep 16 07:33:19.576751 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:33:19.576715 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7001bf5b847388a46f0b4cdfdaf52230a53c2800444ed333ae766288727e167b"} err="failed to get container status \"7001bf5b847388a46f0b4cdfdaf52230a53c2800444ed333ae766288727e167b\": rpc error: code = NotFound desc = could not find container \"7001bf5b847388a46f0b4cdfdaf52230a53c2800444ed333ae766288727e167b\": container with ID starting with 7001bf5b847388a46f0b4cdfdaf52230a53c2800444ed333ae766288727e167b not found: ID does not exist" Sep 16 07:33:19.583921 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:33:19.583899 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-9686b8b46-nklrk"] Sep 16 07:33:19.587120 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:33:19.587099 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-9686b8b46-nklrk"] Sep 16 07:33:21.523210 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:33:21.523175 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88feefad-338d-4fde-b835-c7aba8b1c875" path="/var/lib/kubelet/pods/88feefad-338d-4fde-b835-c7aba8b1c875/volumes" Sep 16 07:33:36.367959 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:33:36.367917 2557 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b4fbfd475-5xvpw" podUID="445e9de2-283b-4d3d-81b9-480fb2de1d2b" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Sep 16 07:33:46.368159 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:33:46.368066 2557 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b4fbfd475-5xvpw" podUID="445e9de2-283b-4d3d-81b9-480fb2de1d2b" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Sep 16 07:33:56.368102 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:33:56.368052 2557 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b4fbfd475-5xvpw" podUID="445e9de2-283b-4d3d-81b9-480fb2de1d2b" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Sep 16 07:33:56.368725 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:33:56.368129 2557 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b4fbfd475-5xvpw" Sep 16 07:33:56.368725 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:33:56.368618 2557 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"fb96d2e2b10e2adee65c13ea7aa33ddff7cfe554e7ee787afe288675a9c77575"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b4fbfd475-5xvpw" containerMessage="Container service-proxy failed liveness probe, will be restarted" Sep 16 07:33:56.368725 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:33:56.368662 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b4fbfd475-5xvpw" podUID="445e9de2-283b-4d3d-81b9-480fb2de1d2b" containerName="service-proxy" containerID="cri-o://fb96d2e2b10e2adee65c13ea7aa33ddff7cfe554e7ee787afe288675a9c77575" gracePeriod=30 Sep 16 07:33:56.697332 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:33:56.697308 2557 generic.go:356] "Generic (PLEG): container finished" podID="445e9de2-283b-4d3d-81b9-480fb2de1d2b" containerID="fb96d2e2b10e2adee65c13ea7aa33ddff7cfe554e7ee787afe288675a9c77575" exitCode=2 Sep 16 07:33:56.697457 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:33:56.697375 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b4fbfd475-5xvpw" event={"ID":"445e9de2-283b-4d3d-81b9-480fb2de1d2b","Type":"ContainerDied","Data":"fb96d2e2b10e2adee65c13ea7aa33ddff7cfe554e7ee787afe288675a9c77575"} Sep 16 07:33:56.697457 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:33:56.697404 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b4fbfd475-5xvpw" event={"ID":"445e9de2-283b-4d3d-81b9-480fb2de1d2b","Type":"ContainerStarted","Data":"5772793b94358089347f6301ac932808e116492d48600cdb0f588399dd8afe70"} Sep 16 07:34:01.423481 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:34:01.423446 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/296c5dcf-403c-4a1f-9d6f-05ffb7d75e38-metrics-certs\") pod \"network-metrics-daemon-kqc7r\" (UID: \"296c5dcf-403c-4a1f-9d6f-05ffb7d75e38\") " pod="openshift-multus/network-metrics-daemon-kqc7r" Sep 16 07:34:01.425536 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:34:01.425517 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/296c5dcf-403c-4a1f-9d6f-05ffb7d75e38-metrics-certs\") pod \"network-metrics-daemon-kqc7r\" (UID: \"296c5dcf-403c-4a1f-9d6f-05ffb7d75e38\") " pod="openshift-multus/network-metrics-daemon-kqc7r" Sep 16 07:34:01.621992 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:34:01.621966 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xjqjb\"" Sep 16 07:34:01.630060 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:34:01.630035 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kqc7r" Sep 16 07:34:01.894524 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:34:01.894497 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-kqc7r"] Sep 16 07:34:01.897786 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:34:01.897763 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod296c5dcf_403c_4a1f_9d6f_05ffb7d75e38.slice/crio-040c4b78954a5f7f7bfc21adbc6a22892a9c902c72c9df87b69a2c25fd5d1ec3 WatchSource:0}: Error finding container 040c4b78954a5f7f7bfc21adbc6a22892a9c902c72c9df87b69a2c25fd5d1ec3: Status 404 returned error can't find the container with id 040c4b78954a5f7f7bfc21adbc6a22892a9c902c72c9df87b69a2c25fd5d1ec3 Sep 16 07:34:01.900023 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:34:01.900009 2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 16 07:34:02.718808 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:34:02.718775 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kqc7r" event={"ID":"296c5dcf-403c-4a1f-9d6f-05ffb7d75e38","Type":"ContainerStarted","Data":"040c4b78954a5f7f7bfc21adbc6a22892a9c902c72c9df87b69a2c25fd5d1ec3"} Sep 16 07:34:03.723370 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:34:03.723336 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kqc7r" event={"ID":"296c5dcf-403c-4a1f-9d6f-05ffb7d75e38","Type":"ContainerStarted","Data":"4bd0f29f3456ee9aea42c978af11f0abc2643dabb91be85d961e9bda3f27e766"} Sep 16 07:34:03.723713 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:34:03.723395 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kqc7r" event={"ID":"296c5dcf-403c-4a1f-9d6f-05ffb7d75e38","Type":"ContainerStarted","Data":"df18782a862c6251913969cc6c9508d3b20fa9b22be230206441c4affa53d657"} Sep 16 07:34:03.739856 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:34:03.739813 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-kqc7r" podStartSLOduration=374.652114167 podStartE2EDuration="6m15.739799923s" totalImagesPullingTime="1.087685756s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:27:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:34:03.739011981 +0000 UTC m=+376.765913015" watchObservedRunningTime="2026-09-16 07:34:03.739799923 +0000 UTC m=+376.766700934" Sep 16 07:34:05.495003 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:34:05.494968 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-85946dbcb-7trx5"] Sep 16 07:34:05.495366 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:34:05.495263 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88feefad-338d-4fde-b835-c7aba8b1c875" containerName="console" Sep 16 07:34:05.495366 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:34:05.495279 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="88feefad-338d-4fde-b835-c7aba8b1c875" containerName="console" Sep 16 07:34:05.495366 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:34:05.495329 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="88feefad-338d-4fde-b835-c7aba8b1c875" containerName="console" Sep 16 07:34:05.498147 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:34:05.498129 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85946dbcb-7trx5" Sep 16 07:34:05.500440 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:34:05.500397 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 16 07:34:05.501094 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:34:05.501075 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 16 07:34:05.501182 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:34:05.501101 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 16 07:34:05.501342 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:34:05.501325 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 16 07:34:05.501425 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:34:05.501328 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 16 07:34:05.501425 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:34:05.501329 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 16 07:34:05.501742 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:34:05.501727 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-rxmq4\"" Sep 16 07:34:05.501887 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:34:05.501872 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 16 07:34:05.506409 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:34:05.506388 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Sep 16 07:34:05.510487 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:34:05.510465 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-85946dbcb-7trx5"] Sep 16 07:34:05.651872 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:34:05.651849 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6cd88575-e813-4844-80b2-c24c3e55acb7-console-serving-cert\") pod \"console-85946dbcb-7trx5\" (UID: \"6cd88575-e813-4844-80b2-c24c3e55acb7\") " pod="openshift-console/console-85946dbcb-7trx5" Sep 16 07:34:05.651986 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:34:05.651877 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6cd88575-e813-4844-80b2-c24c3e55acb7-service-ca\") pod \"console-85946dbcb-7trx5\" (UID: \"6cd88575-e813-4844-80b2-c24c3e55acb7\") " pod="openshift-console/console-85946dbcb-7trx5" Sep 16 07:34:05.651986 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:34:05.651903 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npvhw\" (UniqueName: \"kubernetes.io/projected/6cd88575-e813-4844-80b2-c24c3e55acb7-kube-api-access-npvhw\") pod \"console-85946dbcb-7trx5\" (UID: \"6cd88575-e813-4844-80b2-c24c3e55acb7\") " pod="openshift-console/console-85946dbcb-7trx5" Sep 16 07:34:05.651986 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:34:05.651923 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6cd88575-e813-4844-80b2-c24c3e55acb7-console-config\") pod \"console-85946dbcb-7trx5\" (UID: \"6cd88575-e813-4844-80b2-c24c3e55acb7\") " pod="openshift-console/console-85946dbcb-7trx5" Sep 16 07:34:05.652087 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:34:05.651996 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6cd88575-e813-4844-80b2-c24c3e55acb7-console-oauth-config\") pod \"console-85946dbcb-7trx5\" (UID: \"6cd88575-e813-4844-80b2-c24c3e55acb7\") " pod="openshift-console/console-85946dbcb-7trx5" Sep 16 07:34:05.652087 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:34:05.652020 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6cd88575-e813-4844-80b2-c24c3e55acb7-trusted-ca-bundle\") pod \"console-85946dbcb-7trx5\" (UID: \"6cd88575-e813-4844-80b2-c24c3e55acb7\") " pod="openshift-console/console-85946dbcb-7trx5" Sep 16 07:34:05.652087 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:34:05.652035 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6cd88575-e813-4844-80b2-c24c3e55acb7-oauth-serving-cert\") pod \"console-85946dbcb-7trx5\" (UID: \"6cd88575-e813-4844-80b2-c24c3e55acb7\") " pod="openshift-console/console-85946dbcb-7trx5" Sep 16 07:34:05.752770 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:34:05.752717 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6cd88575-e813-4844-80b2-c24c3e55acb7-console-config\") pod \"console-85946dbcb-7trx5\" (UID: \"6cd88575-e813-4844-80b2-c24c3e55acb7\") " pod="openshift-console/console-85946dbcb-7trx5" Sep 16 07:34:05.752770 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:34:05.752748 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6cd88575-e813-4844-80b2-c24c3e55acb7-console-oauth-config\") pod \"console-85946dbcb-7trx5\" (UID: \"6cd88575-e813-4844-80b2-c24c3e55acb7\") " pod="openshift-console/console-85946dbcb-7trx5" Sep 16 07:34:05.752770 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:34:05.752765 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6cd88575-e813-4844-80b2-c24c3e55acb7-trusted-ca-bundle\") pod \"console-85946dbcb-7trx5\" (UID: \"6cd88575-e813-4844-80b2-c24c3e55acb7\") " pod="openshift-console/console-85946dbcb-7trx5" Sep 16 07:34:05.752924 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:34:05.752788 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6cd88575-e813-4844-80b2-c24c3e55acb7-oauth-serving-cert\") pod \"console-85946dbcb-7trx5\" (UID: \"6cd88575-e813-4844-80b2-c24c3e55acb7\") " pod="openshift-console/console-85946dbcb-7trx5" Sep 16 07:34:05.752924 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:34:05.752862 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6cd88575-e813-4844-80b2-c24c3e55acb7-console-serving-cert\") pod \"console-85946dbcb-7trx5\" (UID: \"6cd88575-e813-4844-80b2-c24c3e55acb7\") " pod="openshift-console/console-85946dbcb-7trx5" Sep 16 07:34:05.752924 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:34:05.752890 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6cd88575-e813-4844-80b2-c24c3e55acb7-service-ca\") pod \"console-85946dbcb-7trx5\" (UID: \"6cd88575-e813-4844-80b2-c24c3e55acb7\") " pod="openshift-console/console-85946dbcb-7trx5" Sep 16 07:34:05.753024 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:34:05.752928 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npvhw\" (UniqueName: \"kubernetes.io/projected/6cd88575-e813-4844-80b2-c24c3e55acb7-kube-api-access-npvhw\") pod \"console-85946dbcb-7trx5\" (UID: \"6cd88575-e813-4844-80b2-c24c3e55acb7\") " pod="openshift-console/console-85946dbcb-7trx5" Sep 16 07:34:05.753567 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:34:05.753539 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6cd88575-e813-4844-80b2-c24c3e55acb7-console-config\") pod \"console-85946dbcb-7trx5\" (UID: \"6cd88575-e813-4844-80b2-c24c3e55acb7\") " pod="openshift-console/console-85946dbcb-7trx5" Sep 16 07:34:05.753685 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:34:05.753539 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6cd88575-e813-4844-80b2-c24c3e55acb7-oauth-serving-cert\") pod \"console-85946dbcb-7trx5\" (UID: \"6cd88575-e813-4844-80b2-c24c3e55acb7\") " pod="openshift-console/console-85946dbcb-7trx5" Sep 16 07:34:05.753685 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:34:05.753582 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6cd88575-e813-4844-80b2-c24c3e55acb7-service-ca\") pod \"console-85946dbcb-7trx5\" (UID: \"6cd88575-e813-4844-80b2-c24c3e55acb7\") " pod="openshift-console/console-85946dbcb-7trx5" Sep 16 07:34:05.753884 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:34:05.753861 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6cd88575-e813-4844-80b2-c24c3e55acb7-trusted-ca-bundle\") pod \"console-85946dbcb-7trx5\" (UID: \"6cd88575-e813-4844-80b2-c24c3e55acb7\") " pod="openshift-console/console-85946dbcb-7trx5" Sep 16 07:34:05.755131 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:34:05.755108 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6cd88575-e813-4844-80b2-c24c3e55acb7-console-serving-cert\") pod \"console-85946dbcb-7trx5\" (UID: \"6cd88575-e813-4844-80b2-c24c3e55acb7\") " pod="openshift-console/console-85946dbcb-7trx5" Sep 16 07:34:05.755214 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:34:05.755179 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6cd88575-e813-4844-80b2-c24c3e55acb7-console-oauth-config\") pod \"console-85946dbcb-7trx5\" (UID: \"6cd88575-e813-4844-80b2-c24c3e55acb7\") " pod="openshift-console/console-85946dbcb-7trx5" Sep 16 07:34:05.763348 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:34:05.763325 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-npvhw\" (UniqueName: \"kubernetes.io/projected/6cd88575-e813-4844-80b2-c24c3e55acb7-kube-api-access-npvhw\") pod \"console-85946dbcb-7trx5\" (UID: \"6cd88575-e813-4844-80b2-c24c3e55acb7\") " pod="openshift-console/console-85946dbcb-7trx5" Sep 16 07:34:05.808326 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:34:05.808292 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85946dbcb-7trx5" Sep 16 07:34:06.075730 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:34:06.075584 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-85946dbcb-7trx5"] Sep 16 07:34:06.078224 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:34:06.078197 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cd88575_e813_4844_80b2_c24c3e55acb7.slice/crio-af29beceda7e375f7f8aae5878188893c71dcd264cef572b1e1ffb8c3bfd1456 WatchSource:0}: Error finding container af29beceda7e375f7f8aae5878188893c71dcd264cef572b1e1ffb8c3bfd1456: Status 404 returned error can't find the container with id af29beceda7e375f7f8aae5878188893c71dcd264cef572b1e1ffb8c3bfd1456 Sep 16 07:34:06.737199 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:34:06.737158 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85946dbcb-7trx5" event={"ID":"6cd88575-e813-4844-80b2-c24c3e55acb7","Type":"ContainerStarted","Data":"ba8068d48b95685830eaeafb61d79b21074097715a75ab6e310d37998c3bb4c0"} Sep 16 07:34:06.737710 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:34:06.737226 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85946dbcb-7trx5" event={"ID":"6cd88575-e813-4844-80b2-c24c3e55acb7","Type":"ContainerStarted","Data":"af29beceda7e375f7f8aae5878188893c71dcd264cef572b1e1ffb8c3bfd1456"} Sep 16 07:34:06.756481 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:34:06.756420 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-85946dbcb-7trx5" podStartSLOduration=1.7564001710000001 podStartE2EDuration="1.756400171s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:34:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:34:06.754726754 +0000 UTC m=+379.781627808" watchObservedRunningTime="2026-09-16 07:34:06.756400171 +0000 UTC m=+379.783301183" Sep 16 07:34:15.809088 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:34:15.809056 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-85946dbcb-7trx5" Sep 16 07:34:15.809463 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:34:15.809107 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-85946dbcb-7trx5" Sep 16 07:34:15.813648 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:34:15.813626 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-85946dbcb-7trx5" Sep 16 07:34:16.775608 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:34:16.775582 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-85946dbcb-7trx5" Sep 16 07:35:16.752387 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:35:16.752353 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-586bf58f78-lqcpf"] Sep 16 07:35:16.755487 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:35:16.755468 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-586bf58f78-lqcpf" Sep 16 07:35:16.772566 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:35:16.772537 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-586bf58f78-lqcpf"] Sep 16 07:35:16.838298 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:35:16.838254 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c5298967-f0f7-4ea1-b44b-40376439cc2c-trusted-ca-bundle\") pod \"console-586bf58f78-lqcpf\" (UID: \"c5298967-f0f7-4ea1-b44b-40376439cc2c\") " pod="openshift-console/console-586bf58f78-lqcpf" Sep 16 07:35:16.838454 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:35:16.838309 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c5298967-f0f7-4ea1-b44b-40376439cc2c-console-oauth-config\") pod \"console-586bf58f78-lqcpf\" (UID: \"c5298967-f0f7-4ea1-b44b-40376439cc2c\") " pod="openshift-console/console-586bf58f78-lqcpf" Sep 16 07:35:16.838454 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:35:16.838329 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c5298967-f0f7-4ea1-b44b-40376439cc2c-oauth-serving-cert\") pod \"console-586bf58f78-lqcpf\" (UID: \"c5298967-f0f7-4ea1-b44b-40376439cc2c\") " pod="openshift-console/console-586bf58f78-lqcpf" Sep 16 07:35:16.838454 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:35:16.838359 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c5298967-f0f7-4ea1-b44b-40376439cc2c-service-ca\") pod \"console-586bf58f78-lqcpf\" (UID: \"c5298967-f0f7-4ea1-b44b-40376439cc2c\") " pod="openshift-console/console-586bf58f78-lqcpf" Sep 16 07:35:16.838454 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:35:16.838386 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvqck\" (UniqueName: \"kubernetes.io/projected/c5298967-f0f7-4ea1-b44b-40376439cc2c-kube-api-access-qvqck\") pod \"console-586bf58f78-lqcpf\" (UID: \"c5298967-f0f7-4ea1-b44b-40376439cc2c\") " pod="openshift-console/console-586bf58f78-lqcpf" Sep 16 07:35:16.838454 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:35:16.838409 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c5298967-f0f7-4ea1-b44b-40376439cc2c-console-config\") pod \"console-586bf58f78-lqcpf\" (UID: \"c5298967-f0f7-4ea1-b44b-40376439cc2c\") " pod="openshift-console/console-586bf58f78-lqcpf" Sep 16 07:35:16.838454 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:35:16.838426 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c5298967-f0f7-4ea1-b44b-40376439cc2c-console-serving-cert\") pod \"console-586bf58f78-lqcpf\" (UID: \"c5298967-f0f7-4ea1-b44b-40376439cc2c\") " pod="openshift-console/console-586bf58f78-lqcpf" Sep 16 07:35:16.939747 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:35:16.939700 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c5298967-f0f7-4ea1-b44b-40376439cc2c-service-ca\") pod \"console-586bf58f78-lqcpf\" (UID: \"c5298967-f0f7-4ea1-b44b-40376439cc2c\") " pod="openshift-console/console-586bf58f78-lqcpf" Sep 16 07:35:16.939747 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:35:16.939754 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvqck\" (UniqueName: \"kubernetes.io/projected/c5298967-f0f7-4ea1-b44b-40376439cc2c-kube-api-access-qvqck\") pod \"console-586bf58f78-lqcpf\" (UID: \"c5298967-f0f7-4ea1-b44b-40376439cc2c\") " pod="openshift-console/console-586bf58f78-lqcpf" Sep 16 07:35:16.939992 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:35:16.939774 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c5298967-f0f7-4ea1-b44b-40376439cc2c-console-config\") pod \"console-586bf58f78-lqcpf\" (UID: \"c5298967-f0f7-4ea1-b44b-40376439cc2c\") " pod="openshift-console/console-586bf58f78-lqcpf" Sep 16 07:35:16.939992 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:35:16.939791 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c5298967-f0f7-4ea1-b44b-40376439cc2c-console-serving-cert\") pod \"console-586bf58f78-lqcpf\" (UID: \"c5298967-f0f7-4ea1-b44b-40376439cc2c\") " pod="openshift-console/console-586bf58f78-lqcpf" Sep 16 07:35:16.939992 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:35:16.939813 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c5298967-f0f7-4ea1-b44b-40376439cc2c-trusted-ca-bundle\") pod \"console-586bf58f78-lqcpf\" (UID: \"c5298967-f0f7-4ea1-b44b-40376439cc2c\") " pod="openshift-console/console-586bf58f78-lqcpf" Sep 16 07:35:16.939992 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:35:16.939840 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c5298967-f0f7-4ea1-b44b-40376439cc2c-console-oauth-config\") pod \"console-586bf58f78-lqcpf\" (UID: \"c5298967-f0f7-4ea1-b44b-40376439cc2c\") " pod="openshift-console/console-586bf58f78-lqcpf" Sep 16 07:35:16.939992 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:35:16.939976 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c5298967-f0f7-4ea1-b44b-40376439cc2c-oauth-serving-cert\") pod \"console-586bf58f78-lqcpf\" (UID: \"c5298967-f0f7-4ea1-b44b-40376439cc2c\") " pod="openshift-console/console-586bf58f78-lqcpf" Sep 16 07:35:16.940588 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:35:16.940558 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c5298967-f0f7-4ea1-b44b-40376439cc2c-service-ca\") pod \"console-586bf58f78-lqcpf\" (UID: \"c5298967-f0f7-4ea1-b44b-40376439cc2c\") " pod="openshift-console/console-586bf58f78-lqcpf" Sep 16 07:35:16.940713 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:35:16.940666 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c5298967-f0f7-4ea1-b44b-40376439cc2c-oauth-serving-cert\") pod \"console-586bf58f78-lqcpf\" (UID: \"c5298967-f0f7-4ea1-b44b-40376439cc2c\") " pod="openshift-console/console-586bf58f78-lqcpf" Sep 16 07:35:16.940775 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:35:16.940727 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c5298967-f0f7-4ea1-b44b-40376439cc2c-console-config\") pod \"console-586bf58f78-lqcpf\" (UID: \"c5298967-f0f7-4ea1-b44b-40376439cc2c\") " pod="openshift-console/console-586bf58f78-lqcpf" Sep 16 07:35:16.940775 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:35:16.940752 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c5298967-f0f7-4ea1-b44b-40376439cc2c-trusted-ca-bundle\") pod \"console-586bf58f78-lqcpf\" (UID: \"c5298967-f0f7-4ea1-b44b-40376439cc2c\") " pod="openshift-console/console-586bf58f78-lqcpf" Sep 16 07:35:16.942142 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:35:16.942115 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c5298967-f0f7-4ea1-b44b-40376439cc2c-console-oauth-config\") pod \"console-586bf58f78-lqcpf\" (UID: \"c5298967-f0f7-4ea1-b44b-40376439cc2c\") " pod="openshift-console/console-586bf58f78-lqcpf" Sep 16 07:35:16.942369 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:35:16.942350 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c5298967-f0f7-4ea1-b44b-40376439cc2c-console-serving-cert\") pod \"console-586bf58f78-lqcpf\" (UID: \"c5298967-f0f7-4ea1-b44b-40376439cc2c\") " pod="openshift-console/console-586bf58f78-lqcpf" Sep 16 07:35:16.947984 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:35:16.947967 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvqck\" (UniqueName: \"kubernetes.io/projected/c5298967-f0f7-4ea1-b44b-40376439cc2c-kube-api-access-qvqck\") pod \"console-586bf58f78-lqcpf\" (UID: \"c5298967-f0f7-4ea1-b44b-40376439cc2c\") " pod="openshift-console/console-586bf58f78-lqcpf" Sep 16 07:35:17.064453 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:35:17.064361 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-586bf58f78-lqcpf" Sep 16 07:35:17.333677 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:35:17.333610 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-586bf58f78-lqcpf"] Sep 16 07:35:17.336656 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:35:17.336616 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5298967_f0f7_4ea1_b44b_40376439cc2c.slice/crio-770f16c946eed12babc6f392480561fe69231bd7785c002e5025ff68643765b8 WatchSource:0}: Error finding container 770f16c946eed12babc6f392480561fe69231bd7785c002e5025ff68643765b8: Status 404 returned error can't find the container with id 770f16c946eed12babc6f392480561fe69231bd7785c002e5025ff68643765b8 Sep 16 07:35:17.980933 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:35:17.980891 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-586bf58f78-lqcpf" event={"ID":"c5298967-f0f7-4ea1-b44b-40376439cc2c","Type":"ContainerStarted","Data":"840c4003906900a3faa0d02c19319dc754c897780c77d1cbb7fb209cac4d6e5d"} Sep 16 07:35:17.981277 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:35:17.980948 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-586bf58f78-lqcpf" event={"ID":"c5298967-f0f7-4ea1-b44b-40376439cc2c","Type":"ContainerStarted","Data":"770f16c946eed12babc6f392480561fe69231bd7785c002e5025ff68643765b8"} Sep 16 07:35:17.997928 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:35:17.997880 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-586bf58f78-lqcpf" podStartSLOduration=1.9978643090000001 podStartE2EDuration="1.997864309s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:35:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:35:17.996811862 +0000 UTC m=+451.023712908" watchObservedRunningTime="2026-09-16 07:35:17.997864309 +0000 UTC m=+451.024765321" Sep 16 07:35:27.064988 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:35:27.064945 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-586bf58f78-lqcpf" Sep 16 07:35:27.065382 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:35:27.065034 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-586bf58f78-lqcpf" Sep 16 07:35:27.069752 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:35:27.069733 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-586bf58f78-lqcpf" Sep 16 07:35:28.021815 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:35:28.021787 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-586bf58f78-lqcpf" Sep 16 07:35:28.069650 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:35:28.069622 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-85946dbcb-7trx5"] Sep 16 07:35:53.088287 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:35:53.088213 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-85946dbcb-7trx5" podUID="6cd88575-e813-4844-80b2-c24c3e55acb7" containerName="console" containerID="cri-o://ba8068d48b95685830eaeafb61d79b21074097715a75ab6e310d37998c3bb4c0" gracePeriod=15 Sep 16 07:35:53.320945 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:35:53.320923 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85946dbcb-7trx5" Sep 16 07:35:53.414637 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:35:53.414606 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6cd88575-e813-4844-80b2-c24c3e55acb7-console-config\") pod \"6cd88575-e813-4844-80b2-c24c3e55acb7\" (UID: \"6cd88575-e813-4844-80b2-c24c3e55acb7\") " Sep 16 07:35:53.414637 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:35:53.414638 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6cd88575-e813-4844-80b2-c24c3e55acb7-trusted-ca-bundle\") pod \"6cd88575-e813-4844-80b2-c24c3e55acb7\" (UID: \"6cd88575-e813-4844-80b2-c24c3e55acb7\") " Sep 16 07:35:53.414858 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:35:53.414658 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6cd88575-e813-4844-80b2-c24c3e55acb7-service-ca\") pod \"6cd88575-e813-4844-80b2-c24c3e55acb7\" (UID: \"6cd88575-e813-4844-80b2-c24c3e55acb7\") " Sep 16 07:35:53.414858 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:35:53.414678 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6cd88575-e813-4844-80b2-c24c3e55acb7-console-serving-cert\") pod \"6cd88575-e813-4844-80b2-c24c3e55acb7\" (UID: \"6cd88575-e813-4844-80b2-c24c3e55acb7\") " Sep 16 07:35:53.414858 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:35:53.414694 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-npvhw\" (UniqueName: \"kubernetes.io/projected/6cd88575-e813-4844-80b2-c24c3e55acb7-kube-api-access-npvhw\") pod \"6cd88575-e813-4844-80b2-c24c3e55acb7\" (UID: \"6cd88575-e813-4844-80b2-c24c3e55acb7\") " Sep 16 07:35:53.414858 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:35:53.414733 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6cd88575-e813-4844-80b2-c24c3e55acb7-console-oauth-config\") pod \"6cd88575-e813-4844-80b2-c24c3e55acb7\" (UID: \"6cd88575-e813-4844-80b2-c24c3e55acb7\") " Sep 16 07:35:53.414858 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:35:53.414784 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6cd88575-e813-4844-80b2-c24c3e55acb7-oauth-serving-cert\") pod \"6cd88575-e813-4844-80b2-c24c3e55acb7\" (UID: \"6cd88575-e813-4844-80b2-c24c3e55acb7\") " Sep 16 07:35:53.415192 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:35:53.415058 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cd88575-e813-4844-80b2-c24c3e55acb7-service-ca" (OuterVolumeSpecName: "service-ca") pod "6cd88575-e813-4844-80b2-c24c3e55acb7" (UID: "6cd88575-e813-4844-80b2-c24c3e55acb7"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 07:35:53.415306 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:35:53.415167 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cd88575-e813-4844-80b2-c24c3e55acb7-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "6cd88575-e813-4844-80b2-c24c3e55acb7" (UID: "6cd88575-e813-4844-80b2-c24c3e55acb7"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 07:35:53.415306 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:35:53.415263 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cd88575-e813-4844-80b2-c24c3e55acb7-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "6cd88575-e813-4844-80b2-c24c3e55acb7" (UID: "6cd88575-e813-4844-80b2-c24c3e55acb7"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 07:35:53.415436 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:35:53.415299 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cd88575-e813-4844-80b2-c24c3e55acb7-console-config" (OuterVolumeSpecName: "console-config") pod "6cd88575-e813-4844-80b2-c24c3e55acb7" (UID: "6cd88575-e813-4844-80b2-c24c3e55acb7"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 07:35:53.416944 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:35:53.416922 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6cd88575-e813-4844-80b2-c24c3e55acb7-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "6cd88575-e813-4844-80b2-c24c3e55acb7" (UID: "6cd88575-e813-4844-80b2-c24c3e55acb7"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 07:35:53.417288 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:35:53.417264 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cd88575-e813-4844-80b2-c24c3e55acb7-kube-api-access-npvhw" (OuterVolumeSpecName: "kube-api-access-npvhw") pod "6cd88575-e813-4844-80b2-c24c3e55acb7" (UID: "6cd88575-e813-4844-80b2-c24c3e55acb7"). InnerVolumeSpecName "kube-api-access-npvhw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 07:35:53.417288 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:35:53.417264 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6cd88575-e813-4844-80b2-c24c3e55acb7-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "6cd88575-e813-4844-80b2-c24c3e55acb7" (UID: "6cd88575-e813-4844-80b2-c24c3e55acb7"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 07:35:53.515777 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:35:53.515743 2557 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6cd88575-e813-4844-80b2-c24c3e55acb7-oauth-serving-cert\") on node \"ip-10-0-141-221.ec2.internal\" DevicePath \"\"" Sep 16 07:35:53.515777 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:35:53.515773 2557 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6cd88575-e813-4844-80b2-c24c3e55acb7-console-config\") on node \"ip-10-0-141-221.ec2.internal\" DevicePath \"\"" Sep 16 07:35:53.515957 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:35:53.515786 2557 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6cd88575-e813-4844-80b2-c24c3e55acb7-trusted-ca-bundle\") on node \"ip-10-0-141-221.ec2.internal\" DevicePath \"\"" Sep 16 07:35:53.515957 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:35:53.515798 2557 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6cd88575-e813-4844-80b2-c24c3e55acb7-service-ca\") on node \"ip-10-0-141-221.ec2.internal\" DevicePath \"\"" Sep 16 07:35:53.515957 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:35:53.515809 2557 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6cd88575-e813-4844-80b2-c24c3e55acb7-console-serving-cert\") on node \"ip-10-0-141-221.ec2.internal\" DevicePath \"\"" Sep 16 07:35:53.515957 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:35:53.515819 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-npvhw\" (UniqueName: \"kubernetes.io/projected/6cd88575-e813-4844-80b2-c24c3e55acb7-kube-api-access-npvhw\") on node \"ip-10-0-141-221.ec2.internal\" DevicePath \"\"" Sep 16 07:35:53.515957 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:35:53.515831 2557 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6cd88575-e813-4844-80b2-c24c3e55acb7-console-oauth-config\") on node \"ip-10-0-141-221.ec2.internal\" DevicePath \"\"" Sep 16 07:35:54.108247 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:35:54.108210 2557 generic.go:356] "Generic (PLEG): container finished" podID="6cd88575-e813-4844-80b2-c24c3e55acb7" containerID="ba8068d48b95685830eaeafb61d79b21074097715a75ab6e310d37998c3bb4c0" exitCode=0 Sep 16 07:35:54.108632 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:35:54.108277 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85946dbcb-7trx5" Sep 16 07:35:54.108632 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:35:54.108302 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85946dbcb-7trx5" event={"ID":"6cd88575-e813-4844-80b2-c24c3e55acb7","Type":"ContainerDied","Data":"ba8068d48b95685830eaeafb61d79b21074097715a75ab6e310d37998c3bb4c0"} Sep 16 07:35:54.108632 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:35:54.108355 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85946dbcb-7trx5" event={"ID":"6cd88575-e813-4844-80b2-c24c3e55acb7","Type":"ContainerDied","Data":"af29beceda7e375f7f8aae5878188893c71dcd264cef572b1e1ffb8c3bfd1456"} Sep 16 07:35:54.108632 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:35:54.108377 2557 scope.go:117] "RemoveContainer" containerID="ba8068d48b95685830eaeafb61d79b21074097715a75ab6e310d37998c3bb4c0" Sep 16 07:35:54.116129 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:35:54.116114 2557 scope.go:117] "RemoveContainer" containerID="ba8068d48b95685830eaeafb61d79b21074097715a75ab6e310d37998c3bb4c0" Sep 16 07:35:54.116413 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:35:54.116394 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba8068d48b95685830eaeafb61d79b21074097715a75ab6e310d37998c3bb4c0\": container with ID starting with ba8068d48b95685830eaeafb61d79b21074097715a75ab6e310d37998c3bb4c0 not found: ID does not exist" containerID="ba8068d48b95685830eaeafb61d79b21074097715a75ab6e310d37998c3bb4c0" Sep 16 07:35:54.116478 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:35:54.116422 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba8068d48b95685830eaeafb61d79b21074097715a75ab6e310d37998c3bb4c0"} err="failed to get container status \"ba8068d48b95685830eaeafb61d79b21074097715a75ab6e310d37998c3bb4c0\": rpc error: code = NotFound desc = could not find container \"ba8068d48b95685830eaeafb61d79b21074097715a75ab6e310d37998c3bb4c0\": container with ID starting with ba8068d48b95685830eaeafb61d79b21074097715a75ab6e310d37998c3bb4c0 not found: ID does not exist" Sep 16 07:35:54.122035 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:35:54.122012 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-85946dbcb-7trx5"] Sep 16 07:35:54.123919 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:35:54.123898 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-85946dbcb-7trx5"] Sep 16 07:35:55.522976 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:35:55.522944 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6cd88575-e813-4844-80b2-c24c3e55acb7" path="/var/lib/kubelet/pods/6cd88575-e813-4844-80b2-c24c3e55acb7/volumes" Sep 16 07:37:34.693672 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:37:34.693638 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-747795c55-lfcd2"] Sep 16 07:37:34.694290 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:37:34.693999 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6cd88575-e813-4844-80b2-c24c3e55acb7" containerName="console" Sep 16 07:37:34.694290 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:37:34.694020 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="6cd88575-e813-4844-80b2-c24c3e55acb7" containerName="console" Sep 16 07:37:34.694290 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:37:34.694106 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="6cd88575-e813-4844-80b2-c24c3e55acb7" containerName="console" Sep 16 07:37:34.697027 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:37:34.697007 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-747795c55-lfcd2" Sep 16 07:37:34.730293 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:37:34.730262 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-747795c55-lfcd2"] Sep 16 07:37:34.824295 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:37:34.824260 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a120f5cd-d5b4-4c52-8067-0bb771c34f6c-oauth-serving-cert\") pod \"console-747795c55-lfcd2\" (UID: \"a120f5cd-d5b4-4c52-8067-0bb771c34f6c\") " pod="openshift-console/console-747795c55-lfcd2" Sep 16 07:37:34.824478 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:37:34.824301 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5c2c\" (UniqueName: \"kubernetes.io/projected/a120f5cd-d5b4-4c52-8067-0bb771c34f6c-kube-api-access-d5c2c\") pod \"console-747795c55-lfcd2\" (UID: \"a120f5cd-d5b4-4c52-8067-0bb771c34f6c\") " pod="openshift-console/console-747795c55-lfcd2" Sep 16 07:37:34.824478 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:37:34.824334 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a120f5cd-d5b4-4c52-8067-0bb771c34f6c-console-serving-cert\") pod \"console-747795c55-lfcd2\" (UID: \"a120f5cd-d5b4-4c52-8067-0bb771c34f6c\") " pod="openshift-console/console-747795c55-lfcd2" Sep 16 07:37:34.824478 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:37:34.824401 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a120f5cd-d5b4-4c52-8067-0bb771c34f6c-trusted-ca-bundle\") pod \"console-747795c55-lfcd2\" (UID: \"a120f5cd-d5b4-4c52-8067-0bb771c34f6c\") " pod="openshift-console/console-747795c55-lfcd2" Sep 16 07:37:34.824478 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:37:34.824450 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a120f5cd-d5b4-4c52-8067-0bb771c34f6c-console-oauth-config\") pod \"console-747795c55-lfcd2\" (UID: \"a120f5cd-d5b4-4c52-8067-0bb771c34f6c\") " pod="openshift-console/console-747795c55-lfcd2" Sep 16 07:37:34.824662 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:37:34.824480 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a120f5cd-d5b4-4c52-8067-0bb771c34f6c-console-config\") pod \"console-747795c55-lfcd2\" (UID: \"a120f5cd-d5b4-4c52-8067-0bb771c34f6c\") " pod="openshift-console/console-747795c55-lfcd2" Sep 16 07:37:34.824662 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:37:34.824528 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a120f5cd-d5b4-4c52-8067-0bb771c34f6c-service-ca\") pod \"console-747795c55-lfcd2\" (UID: \"a120f5cd-d5b4-4c52-8067-0bb771c34f6c\") " pod="openshift-console/console-747795c55-lfcd2" Sep 16 07:37:34.925323 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:37:34.925288 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a120f5cd-d5b4-4c52-8067-0bb771c34f6c-oauth-serving-cert\") pod \"console-747795c55-lfcd2\" (UID: \"a120f5cd-d5b4-4c52-8067-0bb771c34f6c\") " pod="openshift-console/console-747795c55-lfcd2" Sep 16 07:37:34.925323 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:37:34.925329 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5c2c\" (UniqueName: \"kubernetes.io/projected/a120f5cd-d5b4-4c52-8067-0bb771c34f6c-kube-api-access-d5c2c\") pod \"console-747795c55-lfcd2\" (UID: \"a120f5cd-d5b4-4c52-8067-0bb771c34f6c\") " pod="openshift-console/console-747795c55-lfcd2" Sep 16 07:37:34.925527 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:37:34.925349 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a120f5cd-d5b4-4c52-8067-0bb771c34f6c-console-serving-cert\") pod \"console-747795c55-lfcd2\" (UID: \"a120f5cd-d5b4-4c52-8067-0bb771c34f6c\") " pod="openshift-console/console-747795c55-lfcd2" Sep 16 07:37:34.925527 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:37:34.925374 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a120f5cd-d5b4-4c52-8067-0bb771c34f6c-trusted-ca-bundle\") pod \"console-747795c55-lfcd2\" (UID: \"a120f5cd-d5b4-4c52-8067-0bb771c34f6c\") " pod="openshift-console/console-747795c55-lfcd2" Sep 16 07:37:34.925527 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:37:34.925402 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a120f5cd-d5b4-4c52-8067-0bb771c34f6c-console-oauth-config\") pod \"console-747795c55-lfcd2\" (UID: \"a120f5cd-d5b4-4c52-8067-0bb771c34f6c\") " pod="openshift-console/console-747795c55-lfcd2" Sep 16 07:37:34.925527 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:37:34.925419 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a120f5cd-d5b4-4c52-8067-0bb771c34f6c-console-config\") pod \"console-747795c55-lfcd2\" (UID: \"a120f5cd-d5b4-4c52-8067-0bb771c34f6c\") " pod="openshift-console/console-747795c55-lfcd2" Sep 16 07:37:34.925527 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:37:34.925454 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a120f5cd-d5b4-4c52-8067-0bb771c34f6c-service-ca\") pod \"console-747795c55-lfcd2\" (UID: \"a120f5cd-d5b4-4c52-8067-0bb771c34f6c\") " pod="openshift-console/console-747795c55-lfcd2" Sep 16 07:37:34.926150 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:37:34.926119 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a120f5cd-d5b4-4c52-8067-0bb771c34f6c-service-ca\") pod \"console-747795c55-lfcd2\" (UID: \"a120f5cd-d5b4-4c52-8067-0bb771c34f6c\") " pod="openshift-console/console-747795c55-lfcd2" Sep 16 07:37:34.926287 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:37:34.926150 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a120f5cd-d5b4-4c52-8067-0bb771c34f6c-oauth-serving-cert\") pod \"console-747795c55-lfcd2\" (UID: \"a120f5cd-d5b4-4c52-8067-0bb771c34f6c\") " pod="openshift-console/console-747795c55-lfcd2" Sep 16 07:37:34.926287 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:37:34.926210 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a120f5cd-d5b4-4c52-8067-0bb771c34f6c-console-config\") pod \"console-747795c55-lfcd2\" (UID: \"a120f5cd-d5b4-4c52-8067-0bb771c34f6c\") " pod="openshift-console/console-747795c55-lfcd2" Sep 16 07:37:34.926387 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:37:34.926308 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a120f5cd-d5b4-4c52-8067-0bb771c34f6c-trusted-ca-bundle\") pod \"console-747795c55-lfcd2\" (UID: \"a120f5cd-d5b4-4c52-8067-0bb771c34f6c\") " pod="openshift-console/console-747795c55-lfcd2" Sep 16 07:37:34.928267 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:37:34.928251 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a120f5cd-d5b4-4c52-8067-0bb771c34f6c-console-oauth-config\") pod \"console-747795c55-lfcd2\" (UID: \"a120f5cd-d5b4-4c52-8067-0bb771c34f6c\") " pod="openshift-console/console-747795c55-lfcd2" Sep 16 07:37:34.928486 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:37:34.928459 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a120f5cd-d5b4-4c52-8067-0bb771c34f6c-console-serving-cert\") pod \"console-747795c55-lfcd2\" (UID: \"a120f5cd-d5b4-4c52-8067-0bb771c34f6c\") " pod="openshift-console/console-747795c55-lfcd2" Sep 16 07:37:34.933064 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:37:34.933043 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5c2c\" (UniqueName: \"kubernetes.io/projected/a120f5cd-d5b4-4c52-8067-0bb771c34f6c-kube-api-access-d5c2c\") pod \"console-747795c55-lfcd2\" (UID: \"a120f5cd-d5b4-4c52-8067-0bb771c34f6c\") " pod="openshift-console/console-747795c55-lfcd2" Sep 16 07:37:35.005960 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:37:35.005892 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-747795c55-lfcd2" Sep 16 07:37:35.277808 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:37:35.277783 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-747795c55-lfcd2"] Sep 16 07:37:35.279979 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:37:35.279948 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda120f5cd_d5b4_4c52_8067_0bb771c34f6c.slice/crio-964bc485c24d708487f00d55aa7fe07f3da5bfb10179353dfe87bf950a7c4f8d WatchSource:0}: Error finding container 964bc485c24d708487f00d55aa7fe07f3da5bfb10179353dfe87bf950a7c4f8d: Status 404 returned error can't find the container with id 964bc485c24d708487f00d55aa7fe07f3da5bfb10179353dfe87bf950a7c4f8d Sep 16 07:37:35.457462 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:37:35.457427 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-747795c55-lfcd2" event={"ID":"a120f5cd-d5b4-4c52-8067-0bb771c34f6c","Type":"ContainerStarted","Data":"c05a72284c8ee03b6878f10200021bd4ad242645db903fb95d5dc50ab9f307b4"} Sep 16 07:37:35.457609 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:37:35.457485 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-747795c55-lfcd2" event={"ID":"a120f5cd-d5b4-4c52-8067-0bb771c34f6c","Type":"ContainerStarted","Data":"964bc485c24d708487f00d55aa7fe07f3da5bfb10179353dfe87bf950a7c4f8d"} Sep 16 07:37:35.474912 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:37:35.474867 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-747795c55-lfcd2" podStartSLOduration=1.474853134 podStartE2EDuration="1.474853134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:37:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:37:35.473834264 +0000 UTC m=+588.500735278" watchObservedRunningTime="2026-09-16 07:37:35.474853134 +0000 UTC m=+588.501754146" Sep 16 07:37:45.006749 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:37:45.006678 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-747795c55-lfcd2" Sep 16 07:37:45.006749 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:37:45.006730 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-747795c55-lfcd2" Sep 16 07:37:45.011330 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:37:45.011310 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-747795c55-lfcd2" Sep 16 07:37:45.495512 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:37:45.495487 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-747795c55-lfcd2" Sep 16 07:37:45.538355 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:37:45.538323 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-586bf58f78-lqcpf"] Sep 16 07:38:10.558052 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:38:10.557970 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-586bf58f78-lqcpf" podUID="c5298967-f0f7-4ea1-b44b-40376439cc2c" containerName="console" containerID="cri-o://840c4003906900a3faa0d02c19319dc754c897780c77d1cbb7fb209cac4d6e5d" gracePeriod=15 Sep 16 07:38:10.793107 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:38:10.793084 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-586bf58f78-lqcpf" Sep 16 07:38:10.869286 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:38:10.869189 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c5298967-f0f7-4ea1-b44b-40376439cc2c-service-ca\") pod \"c5298967-f0f7-4ea1-b44b-40376439cc2c\" (UID: \"c5298967-f0f7-4ea1-b44b-40376439cc2c\") " Sep 16 07:38:10.869286 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:38:10.869224 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c5298967-f0f7-4ea1-b44b-40376439cc2c-console-oauth-config\") pod \"c5298967-f0f7-4ea1-b44b-40376439cc2c\" (UID: \"c5298967-f0f7-4ea1-b44b-40376439cc2c\") " Sep 16 07:38:10.869286 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:38:10.869276 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c5298967-f0f7-4ea1-b44b-40376439cc2c-console-config\") pod \"c5298967-f0f7-4ea1-b44b-40376439cc2c\" (UID: \"c5298967-f0f7-4ea1-b44b-40376439cc2c\") " Sep 16 07:38:10.869541 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:38:10.869381 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c5298967-f0f7-4ea1-b44b-40376439cc2c-trusted-ca-bundle\") pod \"c5298967-f0f7-4ea1-b44b-40376439cc2c\" (UID: \"c5298967-f0f7-4ea1-b44b-40376439cc2c\") " Sep 16 07:38:10.869541 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:38:10.869419 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c5298967-f0f7-4ea1-b44b-40376439cc2c-oauth-serving-cert\") pod \"c5298967-f0f7-4ea1-b44b-40376439cc2c\" (UID: \"c5298967-f0f7-4ea1-b44b-40376439cc2c\") " Sep 16 07:38:10.869541 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:38:10.869474 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qvqck\" (UniqueName: \"kubernetes.io/projected/c5298967-f0f7-4ea1-b44b-40376439cc2c-kube-api-access-qvqck\") pod \"c5298967-f0f7-4ea1-b44b-40376439cc2c\" (UID: \"c5298967-f0f7-4ea1-b44b-40376439cc2c\") " Sep 16 07:38:10.869541 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:38:10.869518 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c5298967-f0f7-4ea1-b44b-40376439cc2c-console-serving-cert\") pod \"c5298967-f0f7-4ea1-b44b-40376439cc2c\" (UID: \"c5298967-f0f7-4ea1-b44b-40376439cc2c\") " Sep 16 07:38:10.869737 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:38:10.869545 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5298967-f0f7-4ea1-b44b-40376439cc2c-service-ca" (OuterVolumeSpecName: "service-ca") pod "c5298967-f0f7-4ea1-b44b-40376439cc2c" (UID: "c5298967-f0f7-4ea1-b44b-40376439cc2c"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 07:38:10.869737 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:38:10.869717 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5298967-f0f7-4ea1-b44b-40376439cc2c-console-config" (OuterVolumeSpecName: "console-config") pod "c5298967-f0f7-4ea1-b44b-40376439cc2c" (UID: "c5298967-f0f7-4ea1-b44b-40376439cc2c"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 07:38:10.869843 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:38:10.869726 2557 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c5298967-f0f7-4ea1-b44b-40376439cc2c-service-ca\") on node \"ip-10-0-141-221.ec2.internal\" DevicePath \"\"" Sep 16 07:38:10.869843 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:38:10.869825 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5298967-f0f7-4ea1-b44b-40376439cc2c-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "c5298967-f0f7-4ea1-b44b-40376439cc2c" (UID: "c5298967-f0f7-4ea1-b44b-40376439cc2c"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 07:38:10.869931 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:38:10.869865 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5298967-f0f7-4ea1-b44b-40376439cc2c-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "c5298967-f0f7-4ea1-b44b-40376439cc2c" (UID: "c5298967-f0f7-4ea1-b44b-40376439cc2c"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 07:38:10.871441 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:38:10.871421 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5298967-f0f7-4ea1-b44b-40376439cc2c-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "c5298967-f0f7-4ea1-b44b-40376439cc2c" (UID: "c5298967-f0f7-4ea1-b44b-40376439cc2c"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 07:38:10.871565 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:38:10.871542 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5298967-f0f7-4ea1-b44b-40376439cc2c-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "c5298967-f0f7-4ea1-b44b-40376439cc2c" (UID: "c5298967-f0f7-4ea1-b44b-40376439cc2c"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 07:38:10.871638 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:38:10.871616 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5298967-f0f7-4ea1-b44b-40376439cc2c-kube-api-access-qvqck" (OuterVolumeSpecName: "kube-api-access-qvqck") pod "c5298967-f0f7-4ea1-b44b-40376439cc2c" (UID: "c5298967-f0f7-4ea1-b44b-40376439cc2c"). InnerVolumeSpecName "kube-api-access-qvqck". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 07:38:10.970814 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:38:10.970776 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvqck\" (UniqueName: \"kubernetes.io/projected/c5298967-f0f7-4ea1-b44b-40376439cc2c-kube-api-access-qvqck\") on node \"ip-10-0-141-221.ec2.internal\" DevicePath \"\"" Sep 16 07:38:10.970814 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:38:10.970804 2557 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c5298967-f0f7-4ea1-b44b-40376439cc2c-console-serving-cert\") on node \"ip-10-0-141-221.ec2.internal\" DevicePath \"\"" Sep 16 07:38:10.970814 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:38:10.970813 2557 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c5298967-f0f7-4ea1-b44b-40376439cc2c-console-oauth-config\") on node \"ip-10-0-141-221.ec2.internal\" DevicePath \"\"" Sep 16 07:38:10.970814 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:38:10.970822 2557 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c5298967-f0f7-4ea1-b44b-40376439cc2c-console-config\") on node \"ip-10-0-141-221.ec2.internal\" DevicePath \"\"" Sep 16 07:38:10.971066 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:38:10.970830 2557 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c5298967-f0f7-4ea1-b44b-40376439cc2c-trusted-ca-bundle\") on node \"ip-10-0-141-221.ec2.internal\" DevicePath \"\"" Sep 16 07:38:10.971066 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:38:10.970838 2557 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c5298967-f0f7-4ea1-b44b-40376439cc2c-oauth-serving-cert\") on node \"ip-10-0-141-221.ec2.internal\" DevicePath \"\"" Sep 16 07:38:11.588338 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:38:11.588308 2557 generic.go:356] "Generic (PLEG): container finished" podID="c5298967-f0f7-4ea1-b44b-40376439cc2c" containerID="840c4003906900a3faa0d02c19319dc754c897780c77d1cbb7fb209cac4d6e5d" exitCode=0 Sep 16 07:38:11.588662 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:38:11.588359 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-586bf58f78-lqcpf" Sep 16 07:38:11.588662 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:38:11.588363 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-586bf58f78-lqcpf" event={"ID":"c5298967-f0f7-4ea1-b44b-40376439cc2c","Type":"ContainerDied","Data":"840c4003906900a3faa0d02c19319dc754c897780c77d1cbb7fb209cac4d6e5d"} Sep 16 07:38:11.588662 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:38:11.588453 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-586bf58f78-lqcpf" event={"ID":"c5298967-f0f7-4ea1-b44b-40376439cc2c","Type":"ContainerDied","Data":"770f16c946eed12babc6f392480561fe69231bd7785c002e5025ff68643765b8"} Sep 16 07:38:11.588662 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:38:11.588471 2557 scope.go:117] "RemoveContainer" containerID="840c4003906900a3faa0d02c19319dc754c897780c77d1cbb7fb209cac4d6e5d" Sep 16 07:38:11.596038 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:38:11.596022 2557 scope.go:117] "RemoveContainer" containerID="840c4003906900a3faa0d02c19319dc754c897780c77d1cbb7fb209cac4d6e5d" Sep 16 07:38:11.596349 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:38:11.596327 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"840c4003906900a3faa0d02c19319dc754c897780c77d1cbb7fb209cac4d6e5d\": container with ID starting with 840c4003906900a3faa0d02c19319dc754c897780c77d1cbb7fb209cac4d6e5d not found: ID does not exist" containerID="840c4003906900a3faa0d02c19319dc754c897780c77d1cbb7fb209cac4d6e5d" Sep 16 07:38:11.596400 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:38:11.596358 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"840c4003906900a3faa0d02c19319dc754c897780c77d1cbb7fb209cac4d6e5d"} err="failed to get container status \"840c4003906900a3faa0d02c19319dc754c897780c77d1cbb7fb209cac4d6e5d\": rpc error: code = NotFound desc = could not find container \"840c4003906900a3faa0d02c19319dc754c897780c77d1cbb7fb209cac4d6e5d\": container with ID starting with 840c4003906900a3faa0d02c19319dc754c897780c77d1cbb7fb209cac4d6e5d not found: ID does not exist" Sep 16 07:38:11.602990 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:38:11.602971 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-586bf58f78-lqcpf"] Sep 16 07:38:11.605977 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:38:11.605956 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-586bf58f78-lqcpf"] Sep 16 07:38:13.522905 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:38:13.522872 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c5298967-f0f7-4ea1-b44b-40376439cc2c" path="/var/lib/kubelet/pods/c5298967-f0f7-4ea1-b44b-40376439cc2c/volumes" Sep 16 07:45:38.732431 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:45:38.732400 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-test-apply-zqr5r"] Sep 16 07:45:38.732883 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:45:38.732645 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5298967-f0f7-4ea1-b44b-40376439cc2c" containerName="console" Sep 16 07:45:38.732883 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:45:38.732655 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5298967-f0f7-4ea1-b44b-40376439cc2c" containerName="console" Sep 16 07:45:38.732883 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:45:38.732710 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5298967-f0f7-4ea1-b44b-40376439cc2c" containerName="console" Sep 16 07:45:38.735543 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:45:38.735527 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-test-apply-zqr5r" Sep 16 07:45:38.737982 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:45:38.737959 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-intra-comm\"" Sep 16 07:45:38.738885 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:45:38.738865 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-cronjob-dockercfg-p5sfz\"" Sep 16 07:45:38.738885 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:45:38.738863 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\"" Sep 16 07:45:38.739043 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:45:38.738892 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\"" Sep 16 07:45:38.743493 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:45:38.743471 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-test-apply-zqr5r"] Sep 16 07:45:38.866131 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:45:38.866103 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fn6g2\" (UniqueName: \"kubernetes.io/projected/89b4687b-2206-41e1-b36d-1da028473ce4-kube-api-access-fn6g2\") pod \"feast-test-apply-zqr5r\" (UID: \"89b4687b-2206-41e1-b36d-1da028473ce4\") " pod="test-ns-feast/feast-test-apply-zqr5r" Sep 16 07:45:38.966940 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:45:38.966908 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fn6g2\" (UniqueName: \"kubernetes.io/projected/89b4687b-2206-41e1-b36d-1da028473ce4-kube-api-access-fn6g2\") pod \"feast-test-apply-zqr5r\" (UID: \"89b4687b-2206-41e1-b36d-1da028473ce4\") " pod="test-ns-feast/feast-test-apply-zqr5r" Sep 16 07:45:38.976135 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:45:38.976108 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fn6g2\" (UniqueName: \"kubernetes.io/projected/89b4687b-2206-41e1-b36d-1da028473ce4-kube-api-access-fn6g2\") pod \"feast-test-apply-zqr5r\" (UID: \"89b4687b-2206-41e1-b36d-1da028473ce4\") " pod="test-ns-feast/feast-test-apply-zqr5r" Sep 16 07:45:39.045022 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:45:39.044971 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-test-apply-zqr5r" Sep 16 07:45:39.315445 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:45:39.315382 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-test-apply-zqr5r"] Sep 16 07:45:39.318285 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:45:39.318255 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod89b4687b_2206_41e1_b36d_1da028473ce4.slice/crio-2db035658128407d570367b56c4e0ff42028dd93b5acc3da510421b643d585f8 WatchSource:0}: Error finding container 2db035658128407d570367b56c4e0ff42028dd93b5acc3da510421b643d585f8: Status 404 returned error can't find the container with id 2db035658128407d570367b56c4e0ff42028dd93b5acc3da510421b643d585f8 Sep 16 07:45:39.320138 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:45:39.320120 2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 16 07:45:40.181523 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:45:40.181447 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-test-apply-zqr5r" event={"ID":"89b4687b-2206-41e1-b36d-1da028473ce4","Type":"ContainerStarted","Data":"2db035658128407d570367b56c4e0ff42028dd93b5acc3da510421b643d585f8"} Sep 16 07:45:45.204666 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:45:45.204629 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-test-apply-zqr5r" event={"ID":"89b4687b-2206-41e1-b36d-1da028473ce4","Type":"ContainerStarted","Data":"28dfb15d50ae45e6b6f59d40bbd2e213912e84b22c5410747297df9ddd6dc595"} Sep 16 07:45:51.229016 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:45:51.228984 2557 generic.go:356] "Generic (PLEG): container finished" podID="89b4687b-2206-41e1-b36d-1da028473ce4" containerID="28dfb15d50ae45e6b6f59d40bbd2e213912e84b22c5410747297df9ddd6dc595" exitCode=0 Sep 16 07:45:51.229394 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:45:51.229024 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-test-apply-zqr5r" event={"ID":"89b4687b-2206-41e1-b36d-1da028473ce4","Type":"ContainerDied","Data":"28dfb15d50ae45e6b6f59d40bbd2e213912e84b22c5410747297df9ddd6dc595"} Sep 16 07:45:52.235428 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:45:52.235385 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-test-apply-zqr5r" event={"ID":"89b4687b-2206-41e1-b36d-1da028473ce4","Type":"ContainerStarted","Data":"499d0863965782dc8c373052dbb2d492ae518beed8478b9145508ae0e98a6e36"} Sep 16 07:45:52.257415 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:45:52.257357 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-test-apply-zqr5r" podStartSLOduration=2.842258401 podStartE2EDuration="14.25734149s" totalImagesPullingTime="5.008818676s" totalInitContainerRuntime="6.406264413s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:45:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:45:52.256388246 +0000 UTC m=+1085.283289259" watchObservedRunningTime="2026-09-16 07:45:52.25734149 +0000 UTC m=+1085.284242553" Sep 16 07:46:01.269035 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:46:01.269002 2557 generic.go:356] "Generic (PLEG): container finished" podID="89b4687b-2206-41e1-b36d-1da028473ce4" containerID="499d0863965782dc8c373052dbb2d492ae518beed8478b9145508ae0e98a6e36" exitCode=0 Sep 16 07:46:01.269487 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:46:01.269075 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-test-apply-zqr5r" event={"ID":"89b4687b-2206-41e1-b36d-1da028473ce4","Type":"ContainerDied","Data":"499d0863965782dc8c373052dbb2d492ae518beed8478b9145508ae0e98a6e36"} Sep 16 07:46:02.391061 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:46:02.391039 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-test-apply-zqr5r" Sep 16 07:46:02.542255 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:46:02.542180 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fn6g2\" (UniqueName: \"kubernetes.io/projected/89b4687b-2206-41e1-b36d-1da028473ce4-kube-api-access-fn6g2\") pod \"89b4687b-2206-41e1-b36d-1da028473ce4\" (UID: \"89b4687b-2206-41e1-b36d-1da028473ce4\") " Sep 16 07:46:02.544186 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:46:02.544163 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89b4687b-2206-41e1-b36d-1da028473ce4-kube-api-access-fn6g2" (OuterVolumeSpecName: "kube-api-access-fn6g2") pod "89b4687b-2206-41e1-b36d-1da028473ce4" (UID: "89b4687b-2206-41e1-b36d-1da028473ce4"). InnerVolumeSpecName "kube-api-access-fn6g2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 07:46:02.642794 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:46:02.642762 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fn6g2\" (UniqueName: \"kubernetes.io/projected/89b4687b-2206-41e1-b36d-1da028473ce4-kube-api-access-fn6g2\") on node \"ip-10-0-141-221.ec2.internal\" DevicePath \"\"" Sep 16 07:46:03.277820 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:46:03.277791 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-test-apply-zqr5r" event={"ID":"89b4687b-2206-41e1-b36d-1da028473ce4","Type":"ContainerDied","Data":"2db035658128407d570367b56c4e0ff42028dd93b5acc3da510421b643d585f8"} Sep 16 07:46:03.277820 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:46:03.277825 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-test-apply-zqr5r" Sep 16 07:46:03.278029 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:46:03.277829 2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2db035658128407d570367b56c4e0ff42028dd93b5acc3da510421b643d585f8" Sep 16 07:46:03.451514 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:46:03.451486 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/test-ns-feast_feast-test-apply-zqr5r_89b4687b-2206-41e1-b36d-1da028473ce4/feast-0/0.log" Sep 16 07:46:03.457035 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:46:03.457012 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/test-ns-feast_feast-test-apply-zqr5r_89b4687b-2206-41e1-b36d-1da028473ce4/feast-1/0.log" Sep 16 07:46:40.320982 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:46:40.320948 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-test-apply-zqr5r"] Sep 16 07:46:40.327188 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:46:40.327164 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-test-apply-zqr5r"] Sep 16 07:46:41.523182 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:46:41.523150 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="89b4687b-2206-41e1-b36d-1da028473ce4" path="/var/lib/kubelet/pods/89b4687b-2206-41e1-b36d-1da028473ce4/volumes" Sep 16 07:51:47.507931 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:51:47.507751 2557 scope.go:117] "RemoveContainer" containerID="28dfb15d50ae45e6b6f59d40bbd2e213912e84b22c5410747297df9ddd6dc595" Sep 16 07:51:58.863955 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:51:58.863919 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["feast-operator-system/feast-operator-controller-manager-689b877b4b-f4d2x"] Sep 16 07:51:58.864351 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:51:58.864197 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89b4687b-2206-41e1-b36d-1da028473ce4" containerName="feast-0" Sep 16 07:51:58.864351 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:51:58.864210 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="89b4687b-2206-41e1-b36d-1da028473ce4" containerName="feast-0" Sep 16 07:51:58.864351 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:51:58.864226 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89b4687b-2206-41e1-b36d-1da028473ce4" containerName="feast-1" Sep 16 07:51:58.864351 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:51:58.864253 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="89b4687b-2206-41e1-b36d-1da028473ce4" containerName="feast-1" Sep 16 07:51:58.864351 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:51:58.864302 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="89b4687b-2206-41e1-b36d-1da028473ce4" containerName="feast-1" Sep 16 07:51:58.867141 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:51:58.867126 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-689b877b4b-f4d2x" Sep 16 07:51:58.869368 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:51:58.869345 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"feast-operator-system\"/\"kube-root-ca.crt\"" Sep 16 07:51:58.870038 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:51:58.870021 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"feast-operator-system\"/\"feast-operator-controller-manager-dockercfg-85vbd\"" Sep 16 07:51:58.870132 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:51:58.870053 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"feast-operator-system\"/\"openshift-service-ca.crt\"" Sep 16 07:51:58.876001 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:51:58.875976 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-689b877b4b-f4d2x"] Sep 16 07:51:59.032603 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:51:59.032569 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgjhz\" (UniqueName: \"kubernetes.io/projected/07feeadd-44f9-4237-aebd-0cef7ce23acc-kube-api-access-hgjhz\") pod \"feast-operator-controller-manager-689b877b4b-f4d2x\" (UID: \"07feeadd-44f9-4237-aebd-0cef7ce23acc\") " pod="feast-operator-system/feast-operator-controller-manager-689b877b4b-f4d2x" Sep 16 07:51:59.133889 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:51:59.133823 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgjhz\" (UniqueName: \"kubernetes.io/projected/07feeadd-44f9-4237-aebd-0cef7ce23acc-kube-api-access-hgjhz\") pod \"feast-operator-controller-manager-689b877b4b-f4d2x\" (UID: \"07feeadd-44f9-4237-aebd-0cef7ce23acc\") " pod="feast-operator-system/feast-operator-controller-manager-689b877b4b-f4d2x" Sep 16 07:51:59.143642 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:51:59.143616 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgjhz\" (UniqueName: \"kubernetes.io/projected/07feeadd-44f9-4237-aebd-0cef7ce23acc-kube-api-access-hgjhz\") pod \"feast-operator-controller-manager-689b877b4b-f4d2x\" (UID: \"07feeadd-44f9-4237-aebd-0cef7ce23acc\") " pod="feast-operator-system/feast-operator-controller-manager-689b877b4b-f4d2x" Sep 16 07:51:59.177537 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:51:59.177511 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-689b877b4b-f4d2x" Sep 16 07:51:59.446092 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:51:59.446065 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-689b877b4b-f4d2x"] Sep 16 07:51:59.448795 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:51:59.448770 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07feeadd_44f9_4237_aebd_0cef7ce23acc.slice/crio-88432d7c5ce838f687fe5f1a1d1f222af6443b8d9871a6dd8d8bf65df92ae153 WatchSource:0}: Error finding container 88432d7c5ce838f687fe5f1a1d1f222af6443b8d9871a6dd8d8bf65df92ae153: Status 404 returned error can't find the container with id 88432d7c5ce838f687fe5f1a1d1f222af6443b8d9871a6dd8d8bf65df92ae153 Sep 16 07:51:59.450686 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:51:59.450673 2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 16 07:51:59.513545 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:51:59.513513 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-689b877b4b-f4d2x" event={"ID":"07feeadd-44f9-4237-aebd-0cef7ce23acc","Type":"ContainerStarted","Data":"88432d7c5ce838f687fe5f1a1d1f222af6443b8d9871a6dd8d8bf65df92ae153"} Sep 16 07:52:01.524776 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:52:01.524678 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-689b877b4b-f4d2x" event={"ID":"07feeadd-44f9-4237-aebd-0cef7ce23acc","Type":"ContainerStarted","Data":"f1117cb73955ce35387d2bf734353df2f63156d6262b34bb98ccd79223d6f775"} Sep 16 07:52:01.525162 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:52:01.524808 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="feast-operator-system/feast-operator-controller-manager-689b877b4b-f4d2x" Sep 16 07:52:01.541407 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:52:01.541364 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="feast-operator-system/feast-operator-controller-manager-689b877b4b-f4d2x" podStartSLOduration=1.7463111279999999 podStartE2EDuration="3.541350906s" totalImagesPullingTime="1.795039778s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:51:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:52:01.540720199 +0000 UTC m=+1454.567621238" watchObservedRunningTime="2026-09-16 07:52:01.541350906 +0000 UTC m=+1454.568251917" Sep 16 07:52:12.530563 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:52:12.530530 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="feast-operator-system/feast-operator-controller-manager-689b877b4b-f4d2x" Sep 16 07:52:13.879545 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:52:13.879487 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-74rfn"] Sep 16 07:52:13.883105 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:52:13.883084 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74rfn" Sep 16 07:52:13.886201 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:52:13.886177 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\"" Sep 16 07:52:13.886458 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:52:13.886440 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-ui-tls\"" Sep 16 07:52:13.886665 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:52:13.886648 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-online-tls\"" Sep 16 07:52:13.886880 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:52:13.886867 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-offline-tls\"" Sep 16 07:52:13.887163 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:52:13.887148 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-registry-tls\"" Sep 16 07:52:13.887448 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:52:13.887432 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\"" Sep 16 07:52:13.888274 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:52:13.888230 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-tbf99\"" Sep 16 07:52:13.897283 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:52:13.897260 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-74rfn"] Sep 16 07:52:13.941567 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:52:13.941532 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgtwj\" (UniqueName: \"kubernetes.io/projected/7ff1ce99-7bf6-4f01-bcb7-fdc458e62507-kube-api-access-cgtwj\") pod \"feast-simple-feast-setup-57576b8fcf-74rfn\" (UID: \"7ff1ce99-7bf6-4f01-bcb7-fdc458e62507\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74rfn" Sep 16 07:52:13.941567 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:52:13.941568 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/7ff1ce99-7bf6-4f01-bcb7-fdc458e62507-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-74rfn\" (UID: \"7ff1ce99-7bf6-4f01-bcb7-fdc458e62507\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74rfn" Sep 16 07:52:13.941738 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:52:13.941596 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/7ff1ce99-7bf6-4f01-bcb7-fdc458e62507-feast-data\") pod \"feast-simple-feast-setup-57576b8fcf-74rfn\" (UID: \"7ff1ce99-7bf6-4f01-bcb7-fdc458e62507\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74rfn" Sep 16 07:52:13.941738 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:52:13.941647 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/7ff1ce99-7bf6-4f01-bcb7-fdc458e62507-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-74rfn\" (UID: \"7ff1ce99-7bf6-4f01-bcb7-fdc458e62507\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74rfn" Sep 16 07:52:13.941738 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:52:13.941681 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/7ff1ce99-7bf6-4f01-bcb7-fdc458e62507-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-74rfn\" (UID: \"7ff1ce99-7bf6-4f01-bcb7-fdc458e62507\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74rfn" Sep 16 07:52:13.941738 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:52:13.941702 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/7ff1ce99-7bf6-4f01-bcb7-fdc458e62507-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-74rfn\" (UID: \"7ff1ce99-7bf6-4f01-bcb7-fdc458e62507\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74rfn" Sep 16 07:52:14.042384 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:52:14.042350 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/7ff1ce99-7bf6-4f01-bcb7-fdc458e62507-feast-data\") pod \"feast-simple-feast-setup-57576b8fcf-74rfn\" (UID: \"7ff1ce99-7bf6-4f01-bcb7-fdc458e62507\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74rfn" Sep 16 07:52:14.042384 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:52:14.042386 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/7ff1ce99-7bf6-4f01-bcb7-fdc458e62507-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-74rfn\" (UID: \"7ff1ce99-7bf6-4f01-bcb7-fdc458e62507\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74rfn" Sep 16 07:52:14.042607 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:52:14.042425 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/7ff1ce99-7bf6-4f01-bcb7-fdc458e62507-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-74rfn\" (UID: \"7ff1ce99-7bf6-4f01-bcb7-fdc458e62507\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74rfn" Sep 16 07:52:14.042607 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:52:14.042450 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/7ff1ce99-7bf6-4f01-bcb7-fdc458e62507-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-74rfn\" (UID: \"7ff1ce99-7bf6-4f01-bcb7-fdc458e62507\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74rfn" Sep 16 07:52:14.042607 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:52:14.042498 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgtwj\" (UniqueName: \"kubernetes.io/projected/7ff1ce99-7bf6-4f01-bcb7-fdc458e62507-kube-api-access-cgtwj\") pod \"feast-simple-feast-setup-57576b8fcf-74rfn\" (UID: \"7ff1ce99-7bf6-4f01-bcb7-fdc458e62507\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74rfn" Sep 16 07:52:14.042607 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:52:14.042577 2557 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-offline-tls: secret "feast-simple-feast-setup-offline-tls" not found Sep 16 07:52:14.042607 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:52:14.042602 2557 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-registry-tls: secret "feast-simple-feast-setup-registry-tls" not found Sep 16 07:52:14.042826 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:52:14.042640 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ff1ce99-7bf6-4f01-bcb7-fdc458e62507-offline-tls podName:7ff1ce99-7bf6-4f01-bcb7-fdc458e62507 nodeName:}" failed. No retries permitted until 2026-09-16 07:52:14.542625778 +0000 UTC m=+1467.569526772 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/7ff1ce99-7bf6-4f01-bcb7-fdc458e62507-offline-tls") pod "feast-simple-feast-setup-57576b8fcf-74rfn" (UID: "7ff1ce99-7bf6-4f01-bcb7-fdc458e62507") : secret "feast-simple-feast-setup-offline-tls" not found Sep 16 07:52:14.042826 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:52:14.042639 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/7ff1ce99-7bf6-4f01-bcb7-fdc458e62507-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-74rfn\" (UID: \"7ff1ce99-7bf6-4f01-bcb7-fdc458e62507\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74rfn" Sep 16 07:52:14.042826 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:52:14.042577 2557 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-ui-tls: secret "feast-simple-feast-setup-ui-tls" not found Sep 16 07:52:14.042826 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:52:14.042666 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ff1ce99-7bf6-4f01-bcb7-fdc458e62507-registry-tls podName:7ff1ce99-7bf6-4f01-bcb7-fdc458e62507 nodeName:}" failed. No retries permitted until 2026-09-16 07:52:14.542647776 +0000 UTC m=+1467.569548786 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/7ff1ce99-7bf6-4f01-bcb7-fdc458e62507-registry-tls") pod "feast-simple-feast-setup-57576b8fcf-74rfn" (UID: "7ff1ce99-7bf6-4f01-bcb7-fdc458e62507") : secret "feast-simple-feast-setup-registry-tls" not found Sep 16 07:52:14.042826 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:52:14.042722 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ff1ce99-7bf6-4f01-bcb7-fdc458e62507-ui-tls podName:7ff1ce99-7bf6-4f01-bcb7-fdc458e62507 nodeName:}" failed. No retries permitted until 2026-09-16 07:52:14.542703612 +0000 UTC m=+1467.569604620 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "ui-tls" (UniqueName: "kubernetes.io/secret/7ff1ce99-7bf6-4f01-bcb7-fdc458e62507-ui-tls") pod "feast-simple-feast-setup-57576b8fcf-74rfn" (UID: "7ff1ce99-7bf6-4f01-bcb7-fdc458e62507") : secret "feast-simple-feast-setup-ui-tls" not found Sep 16 07:52:14.042826 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:52:14.042773 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/7ff1ce99-7bf6-4f01-bcb7-fdc458e62507-feast-data\") pod \"feast-simple-feast-setup-57576b8fcf-74rfn\" (UID: \"7ff1ce99-7bf6-4f01-bcb7-fdc458e62507\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74rfn" Sep 16 07:52:14.044954 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:52:14.044936 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/7ff1ce99-7bf6-4f01-bcb7-fdc458e62507-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-74rfn\" (UID: \"7ff1ce99-7bf6-4f01-bcb7-fdc458e62507\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74rfn" Sep 16 07:52:14.051594 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:52:14.051570 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgtwj\" (UniqueName: \"kubernetes.io/projected/7ff1ce99-7bf6-4f01-bcb7-fdc458e62507-kube-api-access-cgtwj\") pod \"feast-simple-feast-setup-57576b8fcf-74rfn\" (UID: \"7ff1ce99-7bf6-4f01-bcb7-fdc458e62507\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74rfn" Sep 16 07:52:14.546789 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:52:14.546756 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/7ff1ce99-7bf6-4f01-bcb7-fdc458e62507-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-74rfn\" (UID: \"7ff1ce99-7bf6-4f01-bcb7-fdc458e62507\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74rfn" Sep 16 07:52:14.546789 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:52:14.546799 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/7ff1ce99-7bf6-4f01-bcb7-fdc458e62507-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-74rfn\" (UID: \"7ff1ce99-7bf6-4f01-bcb7-fdc458e62507\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74rfn" Sep 16 07:52:14.546995 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:52:14.546816 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/7ff1ce99-7bf6-4f01-bcb7-fdc458e62507-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-74rfn\" (UID: \"7ff1ce99-7bf6-4f01-bcb7-fdc458e62507\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74rfn" Sep 16 07:52:14.549316 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:52:14.549291 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/7ff1ce99-7bf6-4f01-bcb7-fdc458e62507-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-74rfn\" (UID: \"7ff1ce99-7bf6-4f01-bcb7-fdc458e62507\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74rfn" Sep 16 07:52:14.549446 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:52:14.549325 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/7ff1ce99-7bf6-4f01-bcb7-fdc458e62507-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-74rfn\" (UID: \"7ff1ce99-7bf6-4f01-bcb7-fdc458e62507\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74rfn" Sep 16 07:52:14.549446 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:52:14.549397 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/7ff1ce99-7bf6-4f01-bcb7-fdc458e62507-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-74rfn\" (UID: \"7ff1ce99-7bf6-4f01-bcb7-fdc458e62507\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74rfn" Sep 16 07:52:14.792904 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:52:14.792864 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74rfn" Sep 16 07:52:15.070678 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:52:15.070655 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-74rfn"] Sep 16 07:52:15.072326 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:52:15.072299 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7ff1ce99_7bf6_4f01_bcb7_fdc458e62507.slice/crio-2bef12481f94827d2e85493c21b7074a59e04635c48f3f9d5e2e1d3edd17add7 WatchSource:0}: Error finding container 2bef12481f94827d2e85493c21b7074a59e04635c48f3f9d5e2e1d3edd17add7: Status 404 returned error can't find the container with id 2bef12481f94827d2e85493c21b7074a59e04635c48f3f9d5e2e1d3edd17add7 Sep 16 07:52:15.573819 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:52:15.573785 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74rfn" event={"ID":"7ff1ce99-7bf6-4f01-bcb7-fdc458e62507","Type":"ContainerStarted","Data":"2bef12481f94827d2e85493c21b7074a59e04635c48f3f9d5e2e1d3edd17add7"} Sep 16 07:52:28.633047 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:52:28.633011 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74rfn" event={"ID":"7ff1ce99-7bf6-4f01-bcb7-fdc458e62507","Type":"ContainerStarted","Data":"13c8a50e58512249d43156c6a9e4afb9405bcb69924a3a7f642e9bae235652f2"} Sep 16 07:52:33.651629 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:52:33.651593 2557 generic.go:356] "Generic (PLEG): container finished" podID="7ff1ce99-7bf6-4f01-bcb7-fdc458e62507" containerID="13c8a50e58512249d43156c6a9e4afb9405bcb69924a3a7f642e9bae235652f2" exitCode=0 Sep 16 07:52:33.652126 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:52:33.651659 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74rfn" event={"ID":"7ff1ce99-7bf6-4f01-bcb7-fdc458e62507","Type":"ContainerDied","Data":"13c8a50e58512249d43156c6a9e4afb9405bcb69924a3a7f642e9bae235652f2"} Sep 16 07:52:34.665052 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:52:34.665014 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74rfn" event={"ID":"7ff1ce99-7bf6-4f01-bcb7-fdc458e62507","Type":"ContainerStarted","Data":"2e690b3fed899dcb60dfc8c85aac167ca54ccacd6cd5229bb5ca2470f3d3cbd2"} Sep 16 07:52:39.686155 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:52:39.686119 2557 generic.go:356] "Generic (PLEG): container finished" podID="7ff1ce99-7bf6-4f01-bcb7-fdc458e62507" containerID="2e690b3fed899dcb60dfc8c85aac167ca54ccacd6cd5229bb5ca2470f3d3cbd2" exitCode=0 Sep 16 07:52:39.686518 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:52:39.686166 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74rfn" event={"ID":"7ff1ce99-7bf6-4f01-bcb7-fdc458e62507","Type":"ContainerDied","Data":"2e690b3fed899dcb60dfc8c85aac167ca54ccacd6cd5229bb5ca2470f3d3cbd2"} Sep 16 07:52:40.696534 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:52:40.696488 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74rfn" event={"ID":"7ff1ce99-7bf6-4f01-bcb7-fdc458e62507","Type":"ContainerStarted","Data":"1c2bf1e32b4d05b3e80d954a19bd6a1e7346d168ed64271a7f82c71ef644814f"} Sep 16 07:52:40.697015 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:52:40.696565 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74rfn" event={"ID":"7ff1ce99-7bf6-4f01-bcb7-fdc458e62507","Type":"ContainerStarted","Data":"a6de63a5bd229f7cbe78bb3d3b846f9a77def48e817f565f1989ee681403cc1a"} Sep 16 07:52:40.697015 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:52:40.696586 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74rfn" event={"ID":"7ff1ce99-7bf6-4f01-bcb7-fdc458e62507","Type":"ContainerStarted","Data":"2b53c1ff5934908e0ec478d44bce1868412bedc652150c28c2f45536c7f456ef"} Sep 16 07:52:40.697015 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:52:40.696604 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74rfn" event={"ID":"7ff1ce99-7bf6-4f01-bcb7-fdc458e62507","Type":"ContainerStarted","Data":"3a3d96ff3de23f9a6c143f6940e0fe112c15e82e52048f01da9f4f49712ead26"} Sep 16 07:52:40.723534 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:52:40.723471 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74rfn" podStartSLOduration=15.115318411 podStartE2EDuration="27.723451301s" totalImagesPullingTime="12.60813289s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:52:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:52:40.722309011 +0000 UTC m=+1493.749210018" watchObservedRunningTime="2026-09-16 07:52:40.723451301 +0000 UTC m=+1493.750352314" Sep 16 07:52:41.793336 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:52:41.793210 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74rfn" Sep 16 07:52:41.793336 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:52:41.793281 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74rfn" Sep 16 07:52:41.793336 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:52:41.793305 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74rfn" Sep 16 07:52:41.793336 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:52:41.793322 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74rfn" Sep 16 07:52:41.795380 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:52:41.795229 2557 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74rfn" podUID="7ff1ce99-7bf6-4f01-bcb7-fdc458e62507" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.20:8443: connect: connection refused" Sep 16 07:52:41.795380 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:52:41.795272 2557 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74rfn" podUID="7ff1ce99-7bf6-4f01-bcb7-fdc458e62507" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.20:6571: connect: connection refused" Sep 16 07:52:41.795380 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:52:41.795277 2557 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74rfn" podUID="7ff1ce99-7bf6-4f01-bcb7-fdc458e62507" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.20:8816: connect: connection refused" Sep 16 07:52:41.795380 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:52:41.795324 2557 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74rfn" podUID="7ff1ce99-7bf6-4f01-bcb7-fdc458e62507" containerName="online" probeResult="failure" output="Get \"https://10.133.0.20:6567/health\": dial tcp 10.133.0.20:6567: connect: connection refused" Sep 16 07:52:44.794119 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:52:44.793750 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74rfn" Sep 16 07:52:44.794119 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:52:44.793848 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74rfn" Sep 16 07:52:44.794119 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:52:44.793927 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74rfn" Sep 16 07:52:44.794119 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:52:44.794051 2557 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74rfn" podUID="7ff1ce99-7bf6-4f01-bcb7-fdc458e62507" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.20:6571: connect: connection refused" Sep 16 07:52:44.794119 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:52:44.794059 2557 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74rfn" podUID="7ff1ce99-7bf6-4f01-bcb7-fdc458e62507" containerName="online" probeResult="failure" output="Get \"https://10.133.0.20:6567/health\": dial tcp 10.133.0.20:6567: connect: connection refused" Sep 16 07:52:44.794827 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:52:44.794314 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74rfn" Sep 16 07:52:44.794827 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:52:44.794678 2557 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74rfn" podUID="7ff1ce99-7bf6-4f01-bcb7-fdc458e62507" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.20:8816: connect: connection refused" Sep 16 07:52:44.794827 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:52:44.794680 2557 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74rfn" podUID="7ff1ce99-7bf6-4f01-bcb7-fdc458e62507" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.20:8443: connect: connection refused" Sep 16 07:52:47.524724 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:52:47.524692 2557 scope.go:117] "RemoveContainer" containerID="499d0863965782dc8c373052dbb2d492ae518beed8478b9145508ae0e98a6e36" Sep 16 07:52:47.794099 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:52:47.794009 2557 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74rfn" podUID="7ff1ce99-7bf6-4f01-bcb7-fdc458e62507" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.20:8443: connect: connection refused" Sep 16 07:52:47.794948 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:52:47.794923 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74rfn" Sep 16 07:52:47.795135 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:52:47.795116 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74rfn" Sep 16 07:52:47.795854 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:52:47.795832 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74rfn" Sep 16 07:52:47.795987 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:52:47.795969 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74rfn" Sep 16 07:52:47.800403 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:52:47.800383 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74rfn" Sep 16 07:52:48.740358 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:52:48.740330 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74rfn" Sep 16 07:52:50.794504 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:52:50.794468 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74rfn" Sep 16 07:52:51.749037 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:52:51.749013 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74rfn" Sep 16 07:53:54.309084 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:53:54.309049 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-74rfn"] Sep 16 07:53:54.309720 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:53:54.309373 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74rfn" podUID="7ff1ce99-7bf6-4f01-bcb7-fdc458e62507" containerName="registry" containerID="cri-o://3a3d96ff3de23f9a6c143f6940e0fe112c15e82e52048f01da9f4f49712ead26" gracePeriod=30 Sep 16 07:53:54.309720 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:53:54.309400 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74rfn" podUID="7ff1ce99-7bf6-4f01-bcb7-fdc458e62507" containerName="ui" containerID="cri-o://1c2bf1e32b4d05b3e80d954a19bd6a1e7346d168ed64271a7f82c71ef644814f" gracePeriod=30 Sep 16 07:53:54.309720 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:53:54.309406 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74rfn" podUID="7ff1ce99-7bf6-4f01-bcb7-fdc458e62507" containerName="online" containerID="cri-o://2b53c1ff5934908e0ec478d44bce1868412bedc652150c28c2f45536c7f456ef" gracePeriod=30 Sep 16 07:53:54.309720 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:53:54.309411 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74rfn" podUID="7ff1ce99-7bf6-4f01-bcb7-fdc458e62507" containerName="offline" containerID="cri-o://a6de63a5bd229f7cbe78bb3d3b846f9a77def48e817f565f1989ee681403cc1a" gracePeriod=30 Sep 16 07:53:54.692797 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:53:54.692756 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-cvn72"] Sep 16 07:53:54.697174 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:53:54.697150 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-cvn72" Sep 16 07:53:54.699413 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:53:54.699394 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-8hcn9\"" Sep 16 07:53:54.706130 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:53:54.706103 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-cvn72"] Sep 16 07:53:54.816753 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:53:54.816720 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-cvn72\" (UID: \"a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-cvn72" Sep 16 07:53:54.816921 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:53:54.816772 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e-feast-data\") pod \"feast-simple-feast-setup-57576b8fcf-cvn72\" (UID: \"a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-cvn72" Sep 16 07:53:54.816921 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:53:54.816879 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n66rn\" (UniqueName: \"kubernetes.io/projected/a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e-kube-api-access-n66rn\") pod \"feast-simple-feast-setup-57576b8fcf-cvn72\" (UID: \"a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-cvn72" Sep 16 07:53:54.817024 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:53:54.817005 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-cvn72\" (UID: \"a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-cvn72" Sep 16 07:53:54.817067 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:53:54.817033 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-cvn72\" (UID: \"a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-cvn72" Sep 16 07:53:54.817067 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:53:54.817055 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-cvn72\" (UID: \"a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-cvn72" Sep 16 07:53:54.917737 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:53:54.917702 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-cvn72\" (UID: \"a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-cvn72" Sep 16 07:53:54.917737 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:53:54.917738 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-cvn72\" (UID: \"a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-cvn72" Sep 16 07:53:54.917936 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:53:54.917764 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-cvn72\" (UID: \"a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-cvn72" Sep 16 07:53:54.917936 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:53:54.917788 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-cvn72\" (UID: \"a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-cvn72" Sep 16 07:53:54.917936 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:53:54.917821 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e-feast-data\") pod \"feast-simple-feast-setup-57576b8fcf-cvn72\" (UID: \"a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-cvn72" Sep 16 07:53:54.917936 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:53:54.917855 2557 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-offline-tls: secret "feast-simple-feast-setup-offline-tls" not found Sep 16 07:53:54.917936 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:53:54.917866 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n66rn\" (UniqueName: \"kubernetes.io/projected/a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e-kube-api-access-n66rn\") pod \"feast-simple-feast-setup-57576b8fcf-cvn72\" (UID: \"a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-cvn72" Sep 16 07:53:54.917936 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:53:54.917902 2557 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-online-tls: secret "feast-simple-feast-setup-online-tls" not found Sep 16 07:53:54.917936 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:53:54.917919 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e-offline-tls podName:a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e nodeName:}" failed. No retries permitted until 2026-09-16 07:53:55.417898872 +0000 UTC m=+1568.444799865 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e-offline-tls") pod "feast-simple-feast-setup-57576b8fcf-cvn72" (UID: "a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e") : secret "feast-simple-feast-setup-offline-tls" not found Sep 16 07:53:54.918302 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:53:54.917971 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e-online-tls podName:a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e nodeName:}" failed. No retries permitted until 2026-09-16 07:53:55.417952958 +0000 UTC m=+1568.444853961 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e-online-tls") pod "feast-simple-feast-setup-57576b8fcf-cvn72" (UID: "a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e") : secret "feast-simple-feast-setup-online-tls" not found Sep 16 07:53:54.918544 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:53:54.918504 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e-feast-data\") pod \"feast-simple-feast-setup-57576b8fcf-cvn72\" (UID: \"a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-cvn72" Sep 16 07:53:54.920437 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:53:54.920405 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-cvn72\" (UID: \"a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-cvn72" Sep 16 07:53:54.920559 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:53:54.920523 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-cvn72\" (UID: \"a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-cvn72" Sep 16 07:53:54.928361 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:53:54.928339 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n66rn\" (UniqueName: \"kubernetes.io/projected/a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e-kube-api-access-n66rn\") pod \"feast-simple-feast-setup-57576b8fcf-cvn72\" (UID: \"a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-cvn72" Sep 16 07:53:54.976170 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:53:54.976081 2557 generic.go:356] "Generic (PLEG): container finished" podID="7ff1ce99-7bf6-4f01-bcb7-fdc458e62507" containerID="1c2bf1e32b4d05b3e80d954a19bd6a1e7346d168ed64271a7f82c71ef644814f" exitCode=0 Sep 16 07:53:54.976170 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:53:54.976125 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74rfn" event={"ID":"7ff1ce99-7bf6-4f01-bcb7-fdc458e62507","Type":"ContainerDied","Data":"1c2bf1e32b4d05b3e80d954a19bd6a1e7346d168ed64271a7f82c71ef644814f"} Sep 16 07:53:55.423666 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:53:55.423629 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-cvn72\" (UID: \"a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-cvn72" Sep 16 07:53:55.423666 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:53:55.423666 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-cvn72\" (UID: \"a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-cvn72" Sep 16 07:53:55.426076 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:53:55.426051 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-cvn72\" (UID: \"a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-cvn72" Sep 16 07:53:55.426145 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:53:55.426064 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-cvn72\" (UID: \"a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-cvn72" Sep 16 07:53:55.606846 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:53:55.606752 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-cvn72" Sep 16 07:53:55.878439 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:53:55.878415 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-cvn72"] Sep 16 07:53:55.879982 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:53:55.879960 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda173f8ce_a5ea_4567_9dbc_8f3fc03c7b3e.slice/crio-0013ce7b3c4207d2694fd27731d152ed811ffc22e0ab2a1c1ec649c357c28575 WatchSource:0}: Error finding container 0013ce7b3c4207d2694fd27731d152ed811ffc22e0ab2a1c1ec649c357c28575: Status 404 returned error can't find the container with id 0013ce7b3c4207d2694fd27731d152ed811ffc22e0ab2a1c1ec649c357c28575 Sep 16 07:53:55.981691 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:53:55.981658 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-cvn72" event={"ID":"a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e","Type":"ContainerStarted","Data":"dcecc26cd76a210b8a1609d3002c0b27d29de4c1a935d7dfd1b9dbf0dd169f4f"} Sep 16 07:53:55.981866 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:53:55.981707 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-cvn72" event={"ID":"a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e","Type":"ContainerStarted","Data":"0013ce7b3c4207d2694fd27731d152ed811ffc22e0ab2a1c1ec649c357c28575"} Sep 16 07:53:55.987609 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:53:55.987583 2557 generic.go:356] "Generic (PLEG): container finished" podID="7ff1ce99-7bf6-4f01-bcb7-fdc458e62507" containerID="2b53c1ff5934908e0ec478d44bce1868412bedc652150c28c2f45536c7f456ef" exitCode=0 Sep 16 07:53:55.987749 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:53:55.987642 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74rfn" event={"ID":"7ff1ce99-7bf6-4f01-bcb7-fdc458e62507","Type":"ContainerDied","Data":"2b53c1ff5934908e0ec478d44bce1868412bedc652150c28c2f45536c7f456ef"} Sep 16 07:53:58.738161 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:53:58.738124 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74rfn" podUID="7ff1ce99-7bf6-4f01-bcb7-fdc458e62507" containerName="online" probeResult="failure" output="Get \"https://10.133.0.20:6567/health\": dial tcp 10.133.0.20:6567: connect: connection refused" Sep 16 07:54:01.008113 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:54:01.008077 2557 generic.go:356] "Generic (PLEG): container finished" podID="a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e" containerID="dcecc26cd76a210b8a1609d3002c0b27d29de4c1a935d7dfd1b9dbf0dd169f4f" exitCode=0 Sep 16 07:54:01.008509 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:54:01.008148 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-cvn72" event={"ID":"a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e","Type":"ContainerDied","Data":"dcecc26cd76a210b8a1609d3002c0b27d29de4c1a935d7dfd1b9dbf0dd169f4f"} Sep 16 07:54:01.749647 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:54:01.749601 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74rfn" podUID="7ff1ce99-7bf6-4f01-bcb7-fdc458e62507" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.20:8443: connect: connection refused" Sep 16 07:54:02.014419 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:54:02.014340 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-cvn72" event={"ID":"a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e","Type":"ContainerStarted","Data":"72c9b0ed5138685abee26955d261b4fbc53e4207a4301b405014f105a20cec1d"} Sep 16 07:54:07.035605 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:54:07.035573 2557 generic.go:356] "Generic (PLEG): container finished" podID="a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e" containerID="72c9b0ed5138685abee26955d261b4fbc53e4207a4301b405014f105a20cec1d" exitCode=0 Sep 16 07:54:07.036063 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:54:07.035656 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-cvn72" event={"ID":"a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e","Type":"ContainerDied","Data":"72c9b0ed5138685abee26955d261b4fbc53e4207a4301b405014f105a20cec1d"} Sep 16 07:54:08.045825 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:54:08.045780 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-cvn72" event={"ID":"a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e","Type":"ContainerStarted","Data":"078175717de5a7d4e5903efba2fd731e6f3d7cfb3acebbacd1946fc2a0483757"} Sep 16 07:54:08.046331 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:54:08.045854 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-cvn72" event={"ID":"a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e","Type":"ContainerStarted","Data":"e6dfa2954fc67484d28878e1a9f1905e7aa7449d8a28f40de1cf43b5112344bb"} Sep 16 07:54:08.046331 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:54:08.045878 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-cvn72" event={"ID":"a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e","Type":"ContainerStarted","Data":"07c05265b4f6a09eae295f09615aa631d5ce9ecf421117dc89bfdf905175d715"} Sep 16 07:54:08.046331 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:54:08.045897 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-cvn72" event={"ID":"a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e","Type":"ContainerStarted","Data":"cc9e9a6cd9164e255c86d7c77cb267adc2cfb4b54b43ac4fefec8974b92382a8"} Sep 16 07:54:08.108180 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:54:08.108116 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-cvn72" podStartSLOduration=14.108095984 podStartE2EDuration="14.108095984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:53:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:54:08.107462653 +0000 UTC m=+1581.134363691" watchObservedRunningTime="2026-09-16 07:54:08.108095984 +0000 UTC m=+1581.134997002" Sep 16 07:54:08.738087 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:54:08.738037 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74rfn" podUID="7ff1ce99-7bf6-4f01-bcb7-fdc458e62507" containerName="online" probeResult="failure" output="Get \"https://10.133.0.20:6567/health\": dial tcp 10.133.0.20:6567: connect: connection refused" Sep 16 07:54:10.607747 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:54:10.607710 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-cvn72" Sep 16 07:54:10.608326 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:54:10.608301 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-cvn72" Sep 16 07:54:10.608326 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:54:10.608333 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-cvn72" Sep 16 07:54:10.608546 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:54:10.608350 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-cvn72" Sep 16 07:54:10.609915 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:54:10.609872 2557 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-cvn72" podUID="a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.21:8816: connect: connection refused" Sep 16 07:54:10.609915 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:54:10.609893 2557 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-cvn72" podUID="a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.21:8443: connect: connection refused" Sep 16 07:54:10.610086 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:54:10.609942 2557 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-cvn72" podUID="a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e" containerName="online" probeResult="failure" output="Get \"https://10.133.0.21:6567/health\": dial tcp 10.133.0.21:6567: connect: connection refused" Sep 16 07:54:10.610086 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:54:10.610038 2557 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-cvn72" podUID="a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.21:6571: connect: connection refused" Sep 16 07:54:11.749652 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:54:11.749591 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74rfn" podUID="7ff1ce99-7bf6-4f01-bcb7-fdc458e62507" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.20:8443: connect: connection refused" Sep 16 07:54:13.608171 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:54:13.608119 2557 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-cvn72" podUID="a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.21:8443: connect: connection refused" Sep 16 07:54:13.608171 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:54:13.608152 2557 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-cvn72" podUID="a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.21:6571: connect: connection refused" Sep 16 07:54:13.608738 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:54:13.608170 2557 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-cvn72" podUID="a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.21:8816: connect: connection refused" Sep 16 07:54:13.608851 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:54:13.608158 2557 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-cvn72" podUID="a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e" containerName="online" probeResult="failure" output="Get \"https://10.133.0.21:6567/health\": dial tcp 10.133.0.21:6567: connect: connection refused" Sep 16 07:54:15.607686 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:54:15.607650 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-cvn72" Sep 16 07:54:15.608087 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:54:15.607706 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-cvn72" Sep 16 07:54:15.608087 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:54:15.607719 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-cvn72" Sep 16 07:54:15.608087 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:54:15.607729 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-cvn72" Sep 16 07:54:16.608308 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:54:16.608280 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-cvn72" Sep 16 07:54:16.608750 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:54:16.608380 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-cvn72" Sep 16 07:54:16.608750 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:54:16.608667 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-cvn72" Sep 16 07:54:16.609395 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:54:16.609370 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-cvn72" Sep 16 07:54:16.609516 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:54:16.609462 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-cvn72" Sep 16 07:54:16.609516 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:54:16.609507 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-cvn72" Sep 16 07:54:16.613274 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:54:16.613256 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-cvn72" Sep 16 07:54:17.094780 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:54:17.094755 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-cvn72" Sep 16 07:54:18.738165 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:54:18.738119 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74rfn" podUID="7ff1ce99-7bf6-4f01-bcb7-fdc458e62507" containerName="online" probeResult="failure" output="Get \"https://10.133.0.20:6567/health\": dial tcp 10.133.0.20:6567: connect: connection refused" Sep 16 07:54:18.738601 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:54:18.738279 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74rfn" Sep 16 07:54:21.748906 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:54:21.748855 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74rfn" podUID="7ff1ce99-7bf6-4f01-bcb7-fdc458e62507" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.20:8443: connect: connection refused" Sep 16 07:54:21.749344 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:54:21.748996 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74rfn" Sep 16 07:54:24.952511 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:54:24.952481 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74rfn" Sep 16 07:54:25.079219 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:54:25.079139 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/7ff1ce99-7bf6-4f01-bcb7-fdc458e62507-ui-tls\") pod \"7ff1ce99-7bf6-4f01-bcb7-fdc458e62507\" (UID: \"7ff1ce99-7bf6-4f01-bcb7-fdc458e62507\") " Sep 16 07:54:25.079219 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:54:25.079190 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/7ff1ce99-7bf6-4f01-bcb7-fdc458e62507-online-tls\") pod \"7ff1ce99-7bf6-4f01-bcb7-fdc458e62507\" (UID: \"7ff1ce99-7bf6-4f01-bcb7-fdc458e62507\") " Sep 16 07:54:25.079219 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:54:25.079206 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/7ff1ce99-7bf6-4f01-bcb7-fdc458e62507-registry-tls\") pod \"7ff1ce99-7bf6-4f01-bcb7-fdc458e62507\" (UID: \"7ff1ce99-7bf6-4f01-bcb7-fdc458e62507\") " Sep 16 07:54:25.079486 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:54:25.079250 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cgtwj\" (UniqueName: \"kubernetes.io/projected/7ff1ce99-7bf6-4f01-bcb7-fdc458e62507-kube-api-access-cgtwj\") pod \"7ff1ce99-7bf6-4f01-bcb7-fdc458e62507\" (UID: \"7ff1ce99-7bf6-4f01-bcb7-fdc458e62507\") " Sep 16 07:54:25.079486 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:54:25.079270 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/7ff1ce99-7bf6-4f01-bcb7-fdc458e62507-feast-data\") pod \"7ff1ce99-7bf6-4f01-bcb7-fdc458e62507\" (UID: \"7ff1ce99-7bf6-4f01-bcb7-fdc458e62507\") " Sep 16 07:54:25.079486 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:54:25.079315 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/7ff1ce99-7bf6-4f01-bcb7-fdc458e62507-offline-tls\") pod \"7ff1ce99-7bf6-4f01-bcb7-fdc458e62507\" (UID: \"7ff1ce99-7bf6-4f01-bcb7-fdc458e62507\") " Sep 16 07:54:25.079871 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:54:25.079831 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ff1ce99-7bf6-4f01-bcb7-fdc458e62507-feast-data" (OuterVolumeSpecName: "feast-data") pod "7ff1ce99-7bf6-4f01-bcb7-fdc458e62507" (UID: "7ff1ce99-7bf6-4f01-bcb7-fdc458e62507"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 07:54:25.081607 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:54:25.081573 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ff1ce99-7bf6-4f01-bcb7-fdc458e62507-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "7ff1ce99-7bf6-4f01-bcb7-fdc458e62507" (UID: "7ff1ce99-7bf6-4f01-bcb7-fdc458e62507"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 07:54:25.081786 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:54:25.081770 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ff1ce99-7bf6-4f01-bcb7-fdc458e62507-online-tls" (OuterVolumeSpecName: "online-tls") pod "7ff1ce99-7bf6-4f01-bcb7-fdc458e62507" (UID: "7ff1ce99-7bf6-4f01-bcb7-fdc458e62507"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 07:54:25.081904 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:54:25.081884 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ff1ce99-7bf6-4f01-bcb7-fdc458e62507-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "7ff1ce99-7bf6-4f01-bcb7-fdc458e62507" (UID: "7ff1ce99-7bf6-4f01-bcb7-fdc458e62507"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 07:54:25.081904 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:54:25.081897 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ff1ce99-7bf6-4f01-bcb7-fdc458e62507-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "7ff1ce99-7bf6-4f01-bcb7-fdc458e62507" (UID: "7ff1ce99-7bf6-4f01-bcb7-fdc458e62507"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 07:54:25.081977 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:54:25.081919 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ff1ce99-7bf6-4f01-bcb7-fdc458e62507-kube-api-access-cgtwj" (OuterVolumeSpecName: "kube-api-access-cgtwj") pod "7ff1ce99-7bf6-4f01-bcb7-fdc458e62507" (UID: "7ff1ce99-7bf6-4f01-bcb7-fdc458e62507"). InnerVolumeSpecName "kube-api-access-cgtwj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 07:54:25.124525 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:54:25.124482 2557 generic.go:356] "Generic (PLEG): container finished" podID="7ff1ce99-7bf6-4f01-bcb7-fdc458e62507" containerID="a6de63a5bd229f7cbe78bb3d3b846f9a77def48e817f565f1989ee681403cc1a" exitCode=137 Sep 16 07:54:25.124525 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:54:25.124520 2557 generic.go:356] "Generic (PLEG): container finished" podID="7ff1ce99-7bf6-4f01-bcb7-fdc458e62507" containerID="3a3d96ff3de23f9a6c143f6940e0fe112c15e82e52048f01da9f4f49712ead26" exitCode=137 Sep 16 07:54:25.124711 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:54:25.124565 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74rfn" Sep 16 07:54:25.124711 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:54:25.124569 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74rfn" event={"ID":"7ff1ce99-7bf6-4f01-bcb7-fdc458e62507","Type":"ContainerDied","Data":"a6de63a5bd229f7cbe78bb3d3b846f9a77def48e817f565f1989ee681403cc1a"} Sep 16 07:54:25.124711 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:54:25.124614 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74rfn" event={"ID":"7ff1ce99-7bf6-4f01-bcb7-fdc458e62507","Type":"ContainerDied","Data":"3a3d96ff3de23f9a6c143f6940e0fe112c15e82e52048f01da9f4f49712ead26"} Sep 16 07:54:25.124711 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:54:25.124628 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74rfn" event={"ID":"7ff1ce99-7bf6-4f01-bcb7-fdc458e62507","Type":"ContainerDied","Data":"2bef12481f94827d2e85493c21b7074a59e04635c48f3f9d5e2e1d3edd17add7"} Sep 16 07:54:25.124711 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:54:25.124648 2557 scope.go:117] "RemoveContainer" containerID="1c2bf1e32b4d05b3e80d954a19bd6a1e7346d168ed64271a7f82c71ef644814f" Sep 16 07:54:25.132406 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:54:25.132389 2557 scope.go:117] "RemoveContainer" containerID="a6de63a5bd229f7cbe78bb3d3b846f9a77def48e817f565f1989ee681403cc1a" Sep 16 07:54:25.140051 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:54:25.140033 2557 scope.go:117] "RemoveContainer" containerID="2b53c1ff5934908e0ec478d44bce1868412bedc652150c28c2f45536c7f456ef" Sep 16 07:54:25.151110 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:54:25.146066 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-74rfn"] Sep 16 07:54:25.151649 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:54:25.151627 2557 scope.go:117] "RemoveContainer" containerID="3a3d96ff3de23f9a6c143f6940e0fe112c15e82e52048f01da9f4f49712ead26" Sep 16 07:54:25.153421 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:54:25.153395 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-74rfn"] Sep 16 07:54:25.159433 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:54:25.159411 2557 scope.go:117] "RemoveContainer" containerID="2e690b3fed899dcb60dfc8c85aac167ca54ccacd6cd5229bb5ca2470f3d3cbd2" Sep 16 07:54:25.166487 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:54:25.166472 2557 scope.go:117] "RemoveContainer" containerID="13c8a50e58512249d43156c6a9e4afb9405bcb69924a3a7f642e9bae235652f2" Sep 16 07:54:25.173501 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:54:25.173469 2557 scope.go:117] "RemoveContainer" containerID="1c2bf1e32b4d05b3e80d954a19bd6a1e7346d168ed64271a7f82c71ef644814f" Sep 16 07:54:25.173770 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:54:25.173742 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c2bf1e32b4d05b3e80d954a19bd6a1e7346d168ed64271a7f82c71ef644814f\": container with ID starting with 1c2bf1e32b4d05b3e80d954a19bd6a1e7346d168ed64271a7f82c71ef644814f not found: ID does not exist" containerID="1c2bf1e32b4d05b3e80d954a19bd6a1e7346d168ed64271a7f82c71ef644814f" Sep 16 07:54:25.173835 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:54:25.173779 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c2bf1e32b4d05b3e80d954a19bd6a1e7346d168ed64271a7f82c71ef644814f"} err="failed to get container status \"1c2bf1e32b4d05b3e80d954a19bd6a1e7346d168ed64271a7f82c71ef644814f\": rpc error: code = NotFound desc = could not find container \"1c2bf1e32b4d05b3e80d954a19bd6a1e7346d168ed64271a7f82c71ef644814f\": container with ID starting with 1c2bf1e32b4d05b3e80d954a19bd6a1e7346d168ed64271a7f82c71ef644814f not found: ID does not exist" Sep 16 07:54:25.173835 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:54:25.173799 2557 scope.go:117] "RemoveContainer" containerID="a6de63a5bd229f7cbe78bb3d3b846f9a77def48e817f565f1989ee681403cc1a" Sep 16 07:54:25.174018 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:54:25.174000 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6de63a5bd229f7cbe78bb3d3b846f9a77def48e817f565f1989ee681403cc1a\": container with ID starting with a6de63a5bd229f7cbe78bb3d3b846f9a77def48e817f565f1989ee681403cc1a not found: ID does not exist" containerID="a6de63a5bd229f7cbe78bb3d3b846f9a77def48e817f565f1989ee681403cc1a" Sep 16 07:54:25.174055 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:54:25.174024 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6de63a5bd229f7cbe78bb3d3b846f9a77def48e817f565f1989ee681403cc1a"} err="failed to get container status \"a6de63a5bd229f7cbe78bb3d3b846f9a77def48e817f565f1989ee681403cc1a\": rpc error: code = NotFound desc = could not find container \"a6de63a5bd229f7cbe78bb3d3b846f9a77def48e817f565f1989ee681403cc1a\": container with ID starting with a6de63a5bd229f7cbe78bb3d3b846f9a77def48e817f565f1989ee681403cc1a not found: ID does not exist" Sep 16 07:54:25.174055 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:54:25.174039 2557 scope.go:117] "RemoveContainer" containerID="2b53c1ff5934908e0ec478d44bce1868412bedc652150c28c2f45536c7f456ef" Sep 16 07:54:25.174305 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:54:25.174290 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b53c1ff5934908e0ec478d44bce1868412bedc652150c28c2f45536c7f456ef\": container with ID starting with 2b53c1ff5934908e0ec478d44bce1868412bedc652150c28c2f45536c7f456ef not found: ID does not exist" containerID="2b53c1ff5934908e0ec478d44bce1868412bedc652150c28c2f45536c7f456ef" Sep 16 07:54:25.174363 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:54:25.174308 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b53c1ff5934908e0ec478d44bce1868412bedc652150c28c2f45536c7f456ef"} err="failed to get container status \"2b53c1ff5934908e0ec478d44bce1868412bedc652150c28c2f45536c7f456ef\": rpc error: code = NotFound desc = could not find container \"2b53c1ff5934908e0ec478d44bce1868412bedc652150c28c2f45536c7f456ef\": container with ID starting with 2b53c1ff5934908e0ec478d44bce1868412bedc652150c28c2f45536c7f456ef not found: ID does not exist" Sep 16 07:54:25.174363 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:54:25.174320 2557 scope.go:117] "RemoveContainer" containerID="3a3d96ff3de23f9a6c143f6940e0fe112c15e82e52048f01da9f4f49712ead26" Sep 16 07:54:25.174495 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:54:25.174481 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a3d96ff3de23f9a6c143f6940e0fe112c15e82e52048f01da9f4f49712ead26\": container with ID starting with 3a3d96ff3de23f9a6c143f6940e0fe112c15e82e52048f01da9f4f49712ead26 not found: ID does not exist" containerID="3a3d96ff3de23f9a6c143f6940e0fe112c15e82e52048f01da9f4f49712ead26" Sep 16 07:54:25.174534 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:54:25.174497 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a3d96ff3de23f9a6c143f6940e0fe112c15e82e52048f01da9f4f49712ead26"} err="failed to get container status \"3a3d96ff3de23f9a6c143f6940e0fe112c15e82e52048f01da9f4f49712ead26\": rpc error: code = NotFound desc = could not find container \"3a3d96ff3de23f9a6c143f6940e0fe112c15e82e52048f01da9f4f49712ead26\": container with ID starting with 3a3d96ff3de23f9a6c143f6940e0fe112c15e82e52048f01da9f4f49712ead26 not found: ID does not exist" Sep 16 07:54:25.174534 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:54:25.174507 2557 scope.go:117] "RemoveContainer" containerID="2e690b3fed899dcb60dfc8c85aac167ca54ccacd6cd5229bb5ca2470f3d3cbd2" Sep 16 07:54:25.174740 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:54:25.174723 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e690b3fed899dcb60dfc8c85aac167ca54ccacd6cd5229bb5ca2470f3d3cbd2\": container with ID starting with 2e690b3fed899dcb60dfc8c85aac167ca54ccacd6cd5229bb5ca2470f3d3cbd2 not found: ID does not exist" containerID="2e690b3fed899dcb60dfc8c85aac167ca54ccacd6cd5229bb5ca2470f3d3cbd2" Sep 16 07:54:25.174776 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:54:25.174746 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e690b3fed899dcb60dfc8c85aac167ca54ccacd6cd5229bb5ca2470f3d3cbd2"} err="failed to get container status \"2e690b3fed899dcb60dfc8c85aac167ca54ccacd6cd5229bb5ca2470f3d3cbd2\": rpc error: code = NotFound desc = could not find container \"2e690b3fed899dcb60dfc8c85aac167ca54ccacd6cd5229bb5ca2470f3d3cbd2\": container with ID starting with 2e690b3fed899dcb60dfc8c85aac167ca54ccacd6cd5229bb5ca2470f3d3cbd2 not found: ID does not exist" Sep 16 07:54:25.174776 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:54:25.174765 2557 scope.go:117] "RemoveContainer" containerID="13c8a50e58512249d43156c6a9e4afb9405bcb69924a3a7f642e9bae235652f2" Sep 16 07:54:25.175010 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:54:25.174996 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13c8a50e58512249d43156c6a9e4afb9405bcb69924a3a7f642e9bae235652f2\": container with ID starting with 13c8a50e58512249d43156c6a9e4afb9405bcb69924a3a7f642e9bae235652f2 not found: ID does not exist" containerID="13c8a50e58512249d43156c6a9e4afb9405bcb69924a3a7f642e9bae235652f2" Sep 16 07:54:25.175045 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:54:25.175013 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13c8a50e58512249d43156c6a9e4afb9405bcb69924a3a7f642e9bae235652f2"} err="failed to get container status \"13c8a50e58512249d43156c6a9e4afb9405bcb69924a3a7f642e9bae235652f2\": rpc error: code = NotFound desc = could not find container \"13c8a50e58512249d43156c6a9e4afb9405bcb69924a3a7f642e9bae235652f2\": container with ID starting with 13c8a50e58512249d43156c6a9e4afb9405bcb69924a3a7f642e9bae235652f2 not found: ID does not exist" Sep 16 07:54:25.175045 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:54:25.175025 2557 scope.go:117] "RemoveContainer" containerID="1c2bf1e32b4d05b3e80d954a19bd6a1e7346d168ed64271a7f82c71ef644814f" Sep 16 07:54:25.175250 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:54:25.175221 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c2bf1e32b4d05b3e80d954a19bd6a1e7346d168ed64271a7f82c71ef644814f"} err="failed to get container status \"1c2bf1e32b4d05b3e80d954a19bd6a1e7346d168ed64271a7f82c71ef644814f\": rpc error: code = NotFound desc = could not find container \"1c2bf1e32b4d05b3e80d954a19bd6a1e7346d168ed64271a7f82c71ef644814f\": container with ID starting with 1c2bf1e32b4d05b3e80d954a19bd6a1e7346d168ed64271a7f82c71ef644814f not found: ID does not exist" Sep 16 07:54:25.175305 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:54:25.175251 2557 scope.go:117] "RemoveContainer" containerID="a6de63a5bd229f7cbe78bb3d3b846f9a77def48e817f565f1989ee681403cc1a" Sep 16 07:54:25.175492 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:54:25.175474 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6de63a5bd229f7cbe78bb3d3b846f9a77def48e817f565f1989ee681403cc1a"} err="failed to get container status \"a6de63a5bd229f7cbe78bb3d3b846f9a77def48e817f565f1989ee681403cc1a\": rpc error: code = NotFound desc = could not find container \"a6de63a5bd229f7cbe78bb3d3b846f9a77def48e817f565f1989ee681403cc1a\": container with ID starting with a6de63a5bd229f7cbe78bb3d3b846f9a77def48e817f565f1989ee681403cc1a not found: ID does not exist" Sep 16 07:54:25.175529 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:54:25.175492 2557 scope.go:117] "RemoveContainer" containerID="2b53c1ff5934908e0ec478d44bce1868412bedc652150c28c2f45536c7f456ef" Sep 16 07:54:25.175711 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:54:25.175689 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b53c1ff5934908e0ec478d44bce1868412bedc652150c28c2f45536c7f456ef"} err="failed to get container status \"2b53c1ff5934908e0ec478d44bce1868412bedc652150c28c2f45536c7f456ef\": rpc error: code = NotFound desc = could not find container \"2b53c1ff5934908e0ec478d44bce1868412bedc652150c28c2f45536c7f456ef\": container with ID starting with 2b53c1ff5934908e0ec478d44bce1868412bedc652150c28c2f45536c7f456ef not found: ID does not exist" Sep 16 07:54:25.175748 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:54:25.175712 2557 scope.go:117] "RemoveContainer" containerID="3a3d96ff3de23f9a6c143f6940e0fe112c15e82e52048f01da9f4f49712ead26" Sep 16 07:54:25.175907 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:54:25.175892 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a3d96ff3de23f9a6c143f6940e0fe112c15e82e52048f01da9f4f49712ead26"} err="failed to get container status \"3a3d96ff3de23f9a6c143f6940e0fe112c15e82e52048f01da9f4f49712ead26\": rpc error: code = NotFound desc = could not find container \"3a3d96ff3de23f9a6c143f6940e0fe112c15e82e52048f01da9f4f49712ead26\": container with ID starting with 3a3d96ff3de23f9a6c143f6940e0fe112c15e82e52048f01da9f4f49712ead26 not found: ID does not exist" Sep 16 07:54:25.175943 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:54:25.175907 2557 scope.go:117] "RemoveContainer" containerID="2e690b3fed899dcb60dfc8c85aac167ca54ccacd6cd5229bb5ca2470f3d3cbd2" Sep 16 07:54:25.176126 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:54:25.176108 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e690b3fed899dcb60dfc8c85aac167ca54ccacd6cd5229bb5ca2470f3d3cbd2"} err="failed to get container status \"2e690b3fed899dcb60dfc8c85aac167ca54ccacd6cd5229bb5ca2470f3d3cbd2\": rpc error: code = NotFound desc = could not find container \"2e690b3fed899dcb60dfc8c85aac167ca54ccacd6cd5229bb5ca2470f3d3cbd2\": container with ID starting with 2e690b3fed899dcb60dfc8c85aac167ca54ccacd6cd5229bb5ca2470f3d3cbd2 not found: ID does not exist" Sep 16 07:54:25.176126 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:54:25.176126 2557 scope.go:117] "RemoveContainer" containerID="13c8a50e58512249d43156c6a9e4afb9405bcb69924a3a7f642e9bae235652f2" Sep 16 07:54:25.176364 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:54:25.176348 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13c8a50e58512249d43156c6a9e4afb9405bcb69924a3a7f642e9bae235652f2"} err="failed to get container status \"13c8a50e58512249d43156c6a9e4afb9405bcb69924a3a7f642e9bae235652f2\": rpc error: code = NotFound desc = could not find container \"13c8a50e58512249d43156c6a9e4afb9405bcb69924a3a7f642e9bae235652f2\": container with ID starting with 13c8a50e58512249d43156c6a9e4afb9405bcb69924a3a7f642e9bae235652f2 not found: ID does not exist" Sep 16 07:54:25.180765 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:54:25.180746 2557 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/7ff1ce99-7bf6-4f01-bcb7-fdc458e62507-offline-tls\") on node \"ip-10-0-141-221.ec2.internal\" DevicePath \"\"" Sep 16 07:54:25.180830 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:54:25.180767 2557 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/7ff1ce99-7bf6-4f01-bcb7-fdc458e62507-ui-tls\") on node \"ip-10-0-141-221.ec2.internal\" DevicePath \"\"" Sep 16 07:54:25.180830 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:54:25.180799 2557 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/7ff1ce99-7bf6-4f01-bcb7-fdc458e62507-online-tls\") on node \"ip-10-0-141-221.ec2.internal\" DevicePath \"\"" Sep 16 07:54:25.180830 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:54:25.180807 2557 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/7ff1ce99-7bf6-4f01-bcb7-fdc458e62507-registry-tls\") on node \"ip-10-0-141-221.ec2.internal\" DevicePath \"\"" Sep 16 07:54:25.180830 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:54:25.180816 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgtwj\" (UniqueName: \"kubernetes.io/projected/7ff1ce99-7bf6-4f01-bcb7-fdc458e62507-kube-api-access-cgtwj\") on node \"ip-10-0-141-221.ec2.internal\" DevicePath \"\"" Sep 16 07:54:25.180830 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:54:25.180825 2557 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/7ff1ce99-7bf6-4f01-bcb7-fdc458e62507-feast-data\") on node \"ip-10-0-141-221.ec2.internal\" DevicePath \"\"" Sep 16 07:54:25.523842 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:54:25.523811 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ff1ce99-7bf6-4f01-bcb7-fdc458e62507" path="/var/lib/kubelet/pods/7ff1ce99-7bf6-4f01-bcb7-fdc458e62507/volumes" Sep 16 07:55:19.746974 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:55:19.746939 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5bjgf"] Sep 16 07:55:19.747403 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:55:19.747190 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ff1ce99-7bf6-4f01-bcb7-fdc458e62507" containerName="registry" Sep 16 07:55:19.747403 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:55:19.747200 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ff1ce99-7bf6-4f01-bcb7-fdc458e62507" containerName="registry" Sep 16 07:55:19.747403 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:55:19.747211 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ff1ce99-7bf6-4f01-bcb7-fdc458e62507" containerName="offline" Sep 16 07:55:19.747403 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:55:19.747216 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ff1ce99-7bf6-4f01-bcb7-fdc458e62507" containerName="offline" Sep 16 07:55:19.747403 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:55:19.747223 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ff1ce99-7bf6-4f01-bcb7-fdc458e62507" containerName="ui" Sep 16 07:55:19.747403 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:55:19.747229 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ff1ce99-7bf6-4f01-bcb7-fdc458e62507" containerName="ui" Sep 16 07:55:19.747403 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:55:19.747258 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ff1ce99-7bf6-4f01-bcb7-fdc458e62507" containerName="online" Sep 16 07:55:19.747403 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:55:19.747264 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ff1ce99-7bf6-4f01-bcb7-fdc458e62507" containerName="online" Sep 16 07:55:19.747403 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:55:19.747270 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ff1ce99-7bf6-4f01-bcb7-fdc458e62507" containerName="feast-apply" Sep 16 07:55:19.747403 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:55:19.747275 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ff1ce99-7bf6-4f01-bcb7-fdc458e62507" containerName="feast-apply" Sep 16 07:55:19.747403 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:55:19.747281 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ff1ce99-7bf6-4f01-bcb7-fdc458e62507" containerName="feast-init" Sep 16 07:55:19.747403 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:55:19.747285 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ff1ce99-7bf6-4f01-bcb7-fdc458e62507" containerName="feast-init" Sep 16 07:55:19.747403 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:55:19.747330 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ff1ce99-7bf6-4f01-bcb7-fdc458e62507" containerName="ui" Sep 16 07:55:19.747403 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:55:19.747337 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ff1ce99-7bf6-4f01-bcb7-fdc458e62507" containerName="registry" Sep 16 07:55:19.747403 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:55:19.747343 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ff1ce99-7bf6-4f01-bcb7-fdc458e62507" containerName="online" Sep 16 07:55:19.747403 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:55:19.747349 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ff1ce99-7bf6-4f01-bcb7-fdc458e62507" containerName="offline" Sep 16 07:55:19.750392 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:55:19.750376 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5bjgf" Sep 16 07:55:19.752701 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:55:19.752673 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-online-tls\"" Sep 16 07:55:19.752844 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:55:19.752720 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-ui-tls\"" Sep 16 07:55:19.752844 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:55:19.752675 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"openshift-service-ca.crt\"" Sep 16 07:55:19.752844 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:55:19.752831 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-dockercfg-5bsjg\"" Sep 16 07:55:19.753018 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:55:19.752864 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-offline-tls\"" Sep 16 07:55:19.753018 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:55:19.752979 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-client-ca\"" Sep 16 07:55:19.753098 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:55:19.753072 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"kube-root-ca.crt\"" Sep 16 07:55:19.767301 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:55:19.767277 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5bjgf"] Sep 16 07:55:19.777657 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:55:19.777635 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/31920438-c2e3-41c2-a9a2-f67afab9dfe1-online-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-5bjgf\" (UID: \"31920438-c2e3-41c2-a9a2-f67afab9dfe1\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5bjgf" Sep 16 07:55:19.777787 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:55:19.777681 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/31920438-c2e3-41c2-a9a2-f67afab9dfe1-registry-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-5bjgf\" (UID: \"31920438-c2e3-41c2-a9a2-f67afab9dfe1\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5bjgf" Sep 16 07:55:19.777787 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:55:19.777707 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k624t\" (UniqueName: \"kubernetes.io/projected/31920438-c2e3-41c2-a9a2-f67afab9dfe1-kube-api-access-k624t\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-5bjgf\" (UID: \"31920438-c2e3-41c2-a9a2-f67afab9dfe1\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5bjgf" Sep 16 07:55:19.777787 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:55:19.777725 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/31920438-c2e3-41c2-a9a2-f67afab9dfe1-ui-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-5bjgf\" (UID: \"31920438-c2e3-41c2-a9a2-f67afab9dfe1\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5bjgf" Sep 16 07:55:19.777886 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:55:19.777796 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/31920438-c2e3-41c2-a9a2-f67afab9dfe1-offline-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-5bjgf\" (UID: \"31920438-c2e3-41c2-a9a2-f67afab9dfe1\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5bjgf" Sep 16 07:55:19.777886 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:55:19.777835 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/31920438-c2e3-41c2-a9a2-f67afab9dfe1-feast-data\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-5bjgf\" (UID: \"31920438-c2e3-41c2-a9a2-f67afab9dfe1\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5bjgf" Sep 16 07:55:19.878635 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:55:19.878599 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/31920438-c2e3-41c2-a9a2-f67afab9dfe1-feast-data\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-5bjgf\" (UID: \"31920438-c2e3-41c2-a9a2-f67afab9dfe1\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5bjgf" Sep 16 07:55:19.878822 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:55:19.878651 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/31920438-c2e3-41c2-a9a2-f67afab9dfe1-online-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-5bjgf\" (UID: \"31920438-c2e3-41c2-a9a2-f67afab9dfe1\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5bjgf" Sep 16 07:55:19.878822 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:55:19.878706 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/31920438-c2e3-41c2-a9a2-f67afab9dfe1-registry-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-5bjgf\" (UID: \"31920438-c2e3-41c2-a9a2-f67afab9dfe1\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5bjgf" Sep 16 07:55:19.878822 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:55:19.878738 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k624t\" (UniqueName: \"kubernetes.io/projected/31920438-c2e3-41c2-a9a2-f67afab9dfe1-kube-api-access-k624t\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-5bjgf\" (UID: \"31920438-c2e3-41c2-a9a2-f67afab9dfe1\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5bjgf" Sep 16 07:55:19.878822 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:55:19.878765 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/31920438-c2e3-41c2-a9a2-f67afab9dfe1-ui-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-5bjgf\" (UID: \"31920438-c2e3-41c2-a9a2-f67afab9dfe1\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5bjgf" Sep 16 07:55:19.878822 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:55:19.878803 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/31920438-c2e3-41c2-a9a2-f67afab9dfe1-offline-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-5bjgf\" (UID: \"31920438-c2e3-41c2-a9a2-f67afab9dfe1\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5bjgf" Sep 16 07:55:19.879065 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:55:19.878821 2557 secret.go:189] Couldn't get secret test-ns-remote-registry/feast-simple-feast-remote-setup-online-tls: secret "feast-simple-feast-remote-setup-online-tls" not found Sep 16 07:55:19.879065 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:55:19.878915 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31920438-c2e3-41c2-a9a2-f67afab9dfe1-online-tls podName:31920438-c2e3-41c2-a9a2-f67afab9dfe1 nodeName:}" failed. No retries permitted until 2026-09-16 07:55:20.378894813 +0000 UTC m=+1653.405795819 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/31920438-c2e3-41c2-a9a2-f67afab9dfe1-online-tls") pod "feast-simple-feast-remote-setup-c5d49d8c6-5bjgf" (UID: "31920438-c2e3-41c2-a9a2-f67afab9dfe1") : secret "feast-simple-feast-remote-setup-online-tls" not found Sep 16 07:55:19.879065 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:55:19.878916 2557 secret.go:189] Couldn't get secret test-ns-remote-registry/feast-simple-feast-remote-setup-offline-tls: secret "feast-simple-feast-remote-setup-offline-tls" not found Sep 16 07:55:19.879065 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:55:19.878938 2557 secret.go:189] Couldn't get secret test-ns-remote-registry/feast-simple-feast-remote-setup-ui-tls: secret "feast-simple-feast-remote-setup-ui-tls" not found Sep 16 07:55:19.879065 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:55:19.878956 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31920438-c2e3-41c2-a9a2-f67afab9dfe1-offline-tls podName:31920438-c2e3-41c2-a9a2-f67afab9dfe1 nodeName:}" failed. No retries permitted until 2026-09-16 07:55:20.378946604 +0000 UTC m=+1653.405847594 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/31920438-c2e3-41c2-a9a2-f67afab9dfe1-offline-tls") pod "feast-simple-feast-remote-setup-c5d49d8c6-5bjgf" (UID: "31920438-c2e3-41c2-a9a2-f67afab9dfe1") : secret "feast-simple-feast-remote-setup-offline-tls" not found Sep 16 07:55:19.879065 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:55:19.879005 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31920438-c2e3-41c2-a9a2-f67afab9dfe1-ui-tls podName:31920438-c2e3-41c2-a9a2-f67afab9dfe1 nodeName:}" failed. No retries permitted until 2026-09-16 07:55:20.378985567 +0000 UTC m=+1653.405886560 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "ui-tls" (UniqueName: "kubernetes.io/secret/31920438-c2e3-41c2-a9a2-f67afab9dfe1-ui-tls") pod "feast-simple-feast-remote-setup-c5d49d8c6-5bjgf" (UID: "31920438-c2e3-41c2-a9a2-f67afab9dfe1") : secret "feast-simple-feast-remote-setup-ui-tls" not found Sep 16 07:55:19.879065 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:55:19.879022 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/31920438-c2e3-41c2-a9a2-f67afab9dfe1-feast-data\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-5bjgf\" (UID: \"31920438-c2e3-41c2-a9a2-f67afab9dfe1\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5bjgf" Sep 16 07:55:19.879383 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:55:19.879366 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/31920438-c2e3-41c2-a9a2-f67afab9dfe1-registry-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-5bjgf\" (UID: \"31920438-c2e3-41c2-a9a2-f67afab9dfe1\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5bjgf" Sep 16 07:55:19.888684 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:55:19.888653 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k624t\" (UniqueName: \"kubernetes.io/projected/31920438-c2e3-41c2-a9a2-f67afab9dfe1-kube-api-access-k624t\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-5bjgf\" (UID: \"31920438-c2e3-41c2-a9a2-f67afab9dfe1\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5bjgf" Sep 16 07:55:20.383283 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:55:20.383218 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/31920438-c2e3-41c2-a9a2-f67afab9dfe1-ui-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-5bjgf\" (UID: \"31920438-c2e3-41c2-a9a2-f67afab9dfe1\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5bjgf" Sep 16 07:55:20.383283 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:55:20.383289 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/31920438-c2e3-41c2-a9a2-f67afab9dfe1-offline-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-5bjgf\" (UID: \"31920438-c2e3-41c2-a9a2-f67afab9dfe1\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5bjgf" Sep 16 07:55:20.383495 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:55:20.383323 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/31920438-c2e3-41c2-a9a2-f67afab9dfe1-online-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-5bjgf\" (UID: \"31920438-c2e3-41c2-a9a2-f67afab9dfe1\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5bjgf" Sep 16 07:55:20.383495 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:55:20.383416 2557 secret.go:189] Couldn't get secret test-ns-remote-registry/feast-simple-feast-remote-setup-offline-tls: secret "feast-simple-feast-remote-setup-offline-tls" not found Sep 16 07:55:20.383495 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:55:20.383492 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31920438-c2e3-41c2-a9a2-f67afab9dfe1-offline-tls podName:31920438-c2e3-41c2-a9a2-f67afab9dfe1 nodeName:}" failed. No retries permitted until 2026-09-16 07:55:21.38347228 +0000 UTC m=+1654.410373276 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/31920438-c2e3-41c2-a9a2-f67afab9dfe1-offline-tls") pod "feast-simple-feast-remote-setup-c5d49d8c6-5bjgf" (UID: "31920438-c2e3-41c2-a9a2-f67afab9dfe1") : secret "feast-simple-feast-remote-setup-offline-tls" not found Sep 16 07:55:20.385827 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:55:20.385793 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/31920438-c2e3-41c2-a9a2-f67afab9dfe1-ui-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-5bjgf\" (UID: \"31920438-c2e3-41c2-a9a2-f67afab9dfe1\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5bjgf" Sep 16 07:55:20.385938 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:55:20.385848 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/31920438-c2e3-41c2-a9a2-f67afab9dfe1-online-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-5bjgf\" (UID: \"31920438-c2e3-41c2-a9a2-f67afab9dfe1\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5bjgf" Sep 16 07:55:21.391691 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:55:21.391662 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/31920438-c2e3-41c2-a9a2-f67afab9dfe1-offline-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-5bjgf\" (UID: \"31920438-c2e3-41c2-a9a2-f67afab9dfe1\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5bjgf" Sep 16 07:55:21.394099 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:55:21.394075 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/31920438-c2e3-41c2-a9a2-f67afab9dfe1-offline-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-5bjgf\" (UID: \"31920438-c2e3-41c2-a9a2-f67afab9dfe1\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5bjgf" Sep 16 07:55:21.559925 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:55:21.559888 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5bjgf" Sep 16 07:55:21.832312 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:55:21.832285 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5bjgf"] Sep 16 07:55:21.834101 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:55:21.834075 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod31920438_c2e3_41c2_a9a2_f67afab9dfe1.slice/crio-c45cd1503746f4f710364825c98ee1e6f3b6a0a0fb2a41c6a74cf58f3257f946 WatchSource:0}: Error finding container c45cd1503746f4f710364825c98ee1e6f3b6a0a0fb2a41c6a74cf58f3257f946: Status 404 returned error can't find the container with id c45cd1503746f4f710364825c98ee1e6f3b6a0a0fb2a41c6a74cf58f3257f946 Sep 16 07:55:22.345843 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:55:22.345803 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5bjgf" event={"ID":"31920438-c2e3-41c2-a9a2-f67afab9dfe1","Type":"ContainerStarted","Data":"a50eeef5376405a8e817c767b623b46131edb4100d01b4931d3100d6fc322971"} Sep 16 07:55:22.346062 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:55:22.345863 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5bjgf" event={"ID":"31920438-c2e3-41c2-a9a2-f67afab9dfe1","Type":"ContainerStarted","Data":"c45cd1503746f4f710364825c98ee1e6f3b6a0a0fb2a41c6a74cf58f3257f946"} Sep 16 07:55:27.368392 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:55:27.368309 2557 generic.go:356] "Generic (PLEG): container finished" podID="31920438-c2e3-41c2-a9a2-f67afab9dfe1" containerID="a50eeef5376405a8e817c767b623b46131edb4100d01b4931d3100d6fc322971" exitCode=0 Sep 16 07:55:27.368392 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:55:27.368375 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5bjgf" event={"ID":"31920438-c2e3-41c2-a9a2-f67afab9dfe1","Type":"ContainerDied","Data":"a50eeef5376405a8e817c767b623b46131edb4100d01b4931d3100d6fc322971"} Sep 16 07:55:28.374249 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:55:28.374190 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5bjgf" event={"ID":"31920438-c2e3-41c2-a9a2-f67afab9dfe1","Type":"ContainerStarted","Data":"d7f1deb379b4bf1e9b7d8935afb72f140fb4d713cf26d28920b6388ad3a44a4a"} Sep 16 07:55:49.460970 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:55:49.460933 2557 generic.go:356] "Generic (PLEG): container finished" podID="31920438-c2e3-41c2-a9a2-f67afab9dfe1" containerID="d7f1deb379b4bf1e9b7d8935afb72f140fb4d713cf26d28920b6388ad3a44a4a" exitCode=0 Sep 16 07:55:49.461390 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:55:49.461007 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5bjgf" event={"ID":"31920438-c2e3-41c2-a9a2-f67afab9dfe1","Type":"ContainerDied","Data":"d7f1deb379b4bf1e9b7d8935afb72f140fb4d713cf26d28920b6388ad3a44a4a"} Sep 16 07:55:50.469517 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:55:50.469477 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5bjgf" event={"ID":"31920438-c2e3-41c2-a9a2-f67afab9dfe1","Type":"ContainerStarted","Data":"bad939a5394b89ce0235762e615e34ea7fc0ff43542fed262594c1bde44a0916"} Sep 16 07:55:50.470010 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:55:50.469529 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5bjgf" event={"ID":"31920438-c2e3-41c2-a9a2-f67afab9dfe1","Type":"ContainerStarted","Data":"042e2d6908eb72dae7b91a362508c5044d24eeaef25fbf4ef796d3476ba9a597"} Sep 16 07:55:50.470010 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:55:50.469544 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5bjgf" event={"ID":"31920438-c2e3-41c2-a9a2-f67afab9dfe1","Type":"ContainerStarted","Data":"fdf0080e0c53db9025430f059f427d01cb38da500561eff7b891644db91202cb"} Sep 16 07:55:50.494495 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:55:50.494416 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5bjgf" podStartSLOduration=31.49439171 podStartE2EDuration="31.49439171s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:55:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:55:50.492493941 +0000 UTC m=+1683.519394965" watchObservedRunningTime="2026-09-16 07:55:50.49439171 +0000 UTC m=+1683.521292739" Sep 16 07:55:51.560824 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:55:51.560778 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5bjgf" Sep 16 07:55:51.561408 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:55:51.561377 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5bjgf" Sep 16 07:55:51.561558 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:55:51.561428 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5bjgf" Sep 16 07:55:51.561558 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:55:51.561446 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5bjgf" Sep 16 07:55:51.561558 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:55:51.561464 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5bjgf" Sep 16 07:55:51.561558 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:55:51.561479 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5bjgf" Sep 16 07:55:51.562735 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:55:51.562702 2557 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5bjgf" podUID="31920438-c2e3-41c2-a9a2-f67afab9dfe1" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.22:8816: connect: connection refused" Sep 16 07:55:51.562870 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:55:51.562736 2557 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5bjgf" podUID="31920438-c2e3-41c2-a9a2-f67afab9dfe1" containerName="online" probeResult="failure" output="Get \"https://10.133.0.22:6567/health\": dial tcp 10.133.0.22:6567: connect: connection refused" Sep 16 07:55:51.562870 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:55:51.562713 2557 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5bjgf" podUID="31920438-c2e3-41c2-a9a2-f67afab9dfe1" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.22:8443: connect: connection refused" Sep 16 07:55:54.560701 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:55:54.560649 2557 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5bjgf" podUID="31920438-c2e3-41c2-a9a2-f67afab9dfe1" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.22:8816: connect: connection refused" Sep 16 07:55:54.561183 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:55:54.560717 2557 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5bjgf" podUID="31920438-c2e3-41c2-a9a2-f67afab9dfe1" containerName="online" probeResult="failure" output="Get \"https://10.133.0.22:6567/health\": dial tcp 10.133.0.22:6567: connect: connection refused" Sep 16 07:55:54.561183 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:55:54.560649 2557 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5bjgf" podUID="31920438-c2e3-41c2-a9a2-f67afab9dfe1" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.22:8443: connect: connection refused" Sep 16 07:55:57.560656 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:55:57.560608 2557 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5bjgf" podUID="31920438-c2e3-41c2-a9a2-f67afab9dfe1" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.22:8443: connect: connection refused" Sep 16 07:55:57.560656 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:55:57.560624 2557 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5bjgf" podUID="31920438-c2e3-41c2-a9a2-f67afab9dfe1" containerName="online" probeResult="failure" output="Get \"https://10.133.0.22:6567/health\": dial tcp 10.133.0.22:6567: connect: connection refused" Sep 16 07:55:57.561499 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:55:57.561465 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5bjgf" Sep 16 07:55:57.562346 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:55:57.562326 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5bjgf" Sep 16 07:56:00.560915 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:56:00.560887 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5bjgf" Sep 16 07:56:00.561678 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:56:00.561660 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5bjgf" Sep 16 07:56:01.204305 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:56:01.204269 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5bjgf" Sep 16 07:56:02.192279 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:56:02.192225 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5bjgf" Sep 16 07:56:02.565450 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:56:02.565356 2557 prober.go:120] "Probe failed" probeType="Liveness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5bjgf" podUID="31920438-c2e3-41c2-a9a2-f67afab9dfe1" containerName="online" probeResult="failure" output="Get \"https://10.133.0.22:6567/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Sep 16 07:56:22.564384 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:56:22.564334 2557 prober.go:120] "Probe failed" probeType="Liveness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5bjgf" podUID="31920438-c2e3-41c2-a9a2-f67afab9dfe1" containerName="online" probeResult="failure" output="Get \"https://10.133.0.22:6567/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 16 07:56:42.565422 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:56:42.565364 2557 prober.go:120] "Probe failed" probeType="Liveness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5bjgf" podUID="31920438-c2e3-41c2-a9a2-f67afab9dfe1" containerName="online" probeResult="failure" output="Get \"https://10.133.0.22:6567/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 16 07:57:02.564400 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:02.564354 2557 prober.go:120] "Probe failed" probeType="Liveness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5bjgf" podUID="31920438-c2e3-41c2-a9a2-f67afab9dfe1" containerName="online" probeResult="failure" output="Get \"https://10.133.0.22:6567/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Sep 16 07:57:04.225788 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:04.225749 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5bjgf"] Sep 16 07:57:04.226215 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:04.225972 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5bjgf" podUID="31920438-c2e3-41c2-a9a2-f67afab9dfe1" containerName="online" containerID="cri-o://fdf0080e0c53db9025430f059f427d01cb38da500561eff7b891644db91202cb" gracePeriod=30 Sep 16 07:57:04.226215 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:04.226023 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5bjgf" podUID="31920438-c2e3-41c2-a9a2-f67afab9dfe1" containerName="offline" containerID="cri-o://042e2d6908eb72dae7b91a362508c5044d24eeaef25fbf4ef796d3476ba9a597" gracePeriod=30 Sep 16 07:57:04.226403 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:04.226360 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5bjgf" podUID="31920438-c2e3-41c2-a9a2-f67afab9dfe1" containerName="ui" containerID="cri-o://bad939a5394b89ce0235762e615e34ea7fc0ff43542fed262594c1bde44a0916" gracePeriod=30 Sep 16 07:57:04.369693 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:04.369654 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-cvn72"] Sep 16 07:57:04.369913 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:04.369886 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-cvn72" podUID="a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e" containerName="registry" containerID="cri-o://cc9e9a6cd9164e255c86d7c77cb267adc2cfb4b54b43ac4fefec8974b92382a8" gracePeriod=30 Sep 16 07:57:04.370050 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:04.369994 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-cvn72" podUID="a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e" containerName="ui" containerID="cri-o://078175717de5a7d4e5903efba2fd731e6f3d7cfb3acebbacd1946fc2a0483757" gracePeriod=30 Sep 16 07:57:04.370050 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:04.369994 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-cvn72" podUID="a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e" containerName="offline" containerID="cri-o://e6dfa2954fc67484d28878e1a9f1905e7aa7449d8a28f40de1cf43b5112344bb" gracePeriod=30 Sep 16 07:57:04.370163 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:04.370101 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-cvn72" podUID="a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e" containerName="online" containerID="cri-o://07c05265b4f6a09eae295f09615aa631d5ce9ecf421117dc89bfdf905175d715" gracePeriod=30 Sep 16 07:57:05.777034 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:05.777002 2557 generic.go:356] "Generic (PLEG): container finished" podID="a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e" containerID="078175717de5a7d4e5903efba2fd731e6f3d7cfb3acebbacd1946fc2a0483757" exitCode=0 Sep 16 07:57:05.777034 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:05.777025 2557 generic.go:356] "Generic (PLEG): container finished" podID="a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e" containerID="07c05265b4f6a09eae295f09615aa631d5ce9ecf421117dc89bfdf905175d715" exitCode=0 Sep 16 07:57:05.777488 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:05.777072 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-cvn72" event={"ID":"a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e","Type":"ContainerDied","Data":"078175717de5a7d4e5903efba2fd731e6f3d7cfb3acebbacd1946fc2a0483757"} Sep 16 07:57:05.777488 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:05.777119 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-cvn72" event={"ID":"a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e","Type":"ContainerDied","Data":"07c05265b4f6a09eae295f09615aa631d5ce9ecf421117dc89bfdf905175d715"} Sep 16 07:57:05.779745 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:05.779723 2557 generic.go:356] "Generic (PLEG): container finished" podID="31920438-c2e3-41c2-a9a2-f67afab9dfe1" containerID="bad939a5394b89ce0235762e615e34ea7fc0ff43542fed262594c1bde44a0916" exitCode=0 Sep 16 07:57:05.779745 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:05.779743 2557 generic.go:356] "Generic (PLEG): container finished" podID="31920438-c2e3-41c2-a9a2-f67afab9dfe1" containerID="fdf0080e0c53db9025430f059f427d01cb38da500561eff7b891644db91202cb" exitCode=0 Sep 16 07:57:05.779888 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:05.779790 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5bjgf" event={"ID":"31920438-c2e3-41c2-a9a2-f67afab9dfe1","Type":"ContainerDied","Data":"bad939a5394b89ce0235762e615e34ea7fc0ff43542fed262594c1bde44a0916"} Sep 16 07:57:05.779888 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:05.779816 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5bjgf" event={"ID":"31920438-c2e3-41c2-a9a2-f67afab9dfe1","Type":"ContainerDied","Data":"fdf0080e0c53db9025430f059f427d01cb38da500561eff7b891644db91202cb"} Sep 16 07:57:06.609361 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:06.609312 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-cvn72" podUID="a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.21:8443: connect: connection refused" Sep 16 07:57:07.092353 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:07.092314 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-cvn72" podUID="a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e" containerName="online" probeResult="failure" output="Get \"https://10.133.0.21:6567/health\": dial tcp 10.133.0.21:6567: connect: connection refused" Sep 16 07:57:10.562280 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:10.562211 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5bjgf" podUID="31920438-c2e3-41c2-a9a2-f67afab9dfe1" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.22:8443: connect: connection refused" Sep 16 07:57:11.529476 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:11.529437 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5bjgf" podUID="31920438-c2e3-41c2-a9a2-f67afab9dfe1" containerName="online" probeResult="failure" output="Get \"https://10.133.0.22:6567/health\": dial tcp 10.133.0.22:6567: connect: connection refused" Sep 16 07:57:16.610040 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:16.609994 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-cvn72" podUID="a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.21:8443: connect: connection refused" Sep 16 07:57:17.092732 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:17.092686 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-cvn72" podUID="a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e" containerName="online" probeResult="failure" output="Get \"https://10.133.0.21:6567/health\": dial tcp 10.133.0.21:6567: connect: connection refused" Sep 16 07:57:20.562033 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:20.561992 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5bjgf" podUID="31920438-c2e3-41c2-a9a2-f67afab9dfe1" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.22:8443: connect: connection refused" Sep 16 07:57:21.529250 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:21.529202 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5bjgf" podUID="31920438-c2e3-41c2-a9a2-f67afab9dfe1" containerName="online" probeResult="failure" output="Get \"https://10.133.0.22:6567/health\": dial tcp 10.133.0.22:6567: connect: connection refused" Sep 16 07:57:26.609676 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:26.609623 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-cvn72" podUID="a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.21:8443: connect: connection refused" Sep 16 07:57:26.610104 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:26.609776 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-cvn72" Sep 16 07:57:27.092052 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:27.092014 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-cvn72" podUID="a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e" containerName="online" probeResult="failure" output="Get \"https://10.133.0.21:6567/health\": dial tcp 10.133.0.21:6567: connect: connection refused" Sep 16 07:57:27.092216 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:27.092120 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-cvn72" Sep 16 07:57:30.562008 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:30.561967 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5bjgf" podUID="31920438-c2e3-41c2-a9a2-f67afab9dfe1" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.22:8443: connect: connection refused" Sep 16 07:57:30.562408 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:30.562097 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5bjgf" Sep 16 07:57:31.529839 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:31.529797 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5bjgf" podUID="31920438-c2e3-41c2-a9a2-f67afab9dfe1" containerName="online" probeResult="failure" output="Get \"https://10.133.0.22:6567/health\": dial tcp 10.133.0.22:6567: connect: connection refused" Sep 16 07:57:31.530002 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:31.529910 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5bjgf" Sep 16 07:57:34.873793 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:34.873767 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5bjgf" Sep 16 07:57:34.893452 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:34.893426 2557 generic.go:356] "Generic (PLEG): container finished" podID="a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e" containerID="e6dfa2954fc67484d28878e1a9f1905e7aa7449d8a28f40de1cf43b5112344bb" exitCode=137 Sep 16 07:57:34.893452 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:34.893448 2557 generic.go:356] "Generic (PLEG): container finished" podID="a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e" containerID="cc9e9a6cd9164e255c86d7c77cb267adc2cfb4b54b43ac4fefec8974b92382a8" exitCode=137 Sep 16 07:57:34.893622 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:34.893498 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-cvn72" event={"ID":"a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e","Type":"ContainerDied","Data":"e6dfa2954fc67484d28878e1a9f1905e7aa7449d8a28f40de1cf43b5112344bb"} Sep 16 07:57:34.893622 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:34.893555 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-cvn72" event={"ID":"a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e","Type":"ContainerDied","Data":"cc9e9a6cd9164e255c86d7c77cb267adc2cfb4b54b43ac4fefec8974b92382a8"} Sep 16 07:57:34.897319 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:34.897287 2557 generic.go:356] "Generic (PLEG): container finished" podID="31920438-c2e3-41c2-a9a2-f67afab9dfe1" containerID="042e2d6908eb72dae7b91a362508c5044d24eeaef25fbf4ef796d3476ba9a597" exitCode=137 Sep 16 07:57:34.897475 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:34.897336 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5bjgf" Sep 16 07:57:34.897475 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:34.897355 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5bjgf" event={"ID":"31920438-c2e3-41c2-a9a2-f67afab9dfe1","Type":"ContainerDied","Data":"042e2d6908eb72dae7b91a362508c5044d24eeaef25fbf4ef796d3476ba9a597"} Sep 16 07:57:34.897475 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:34.897414 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5bjgf" event={"ID":"31920438-c2e3-41c2-a9a2-f67afab9dfe1","Type":"ContainerDied","Data":"c45cd1503746f4f710364825c98ee1e6f3b6a0a0fb2a41c6a74cf58f3257f946"} Sep 16 07:57:34.897475 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:34.897441 2557 scope.go:117] "RemoveContainer" containerID="bad939a5394b89ce0235762e615e34ea7fc0ff43542fed262594c1bde44a0916" Sep 16 07:57:34.905847 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:34.905811 2557 scope.go:117] "RemoveContainer" containerID="042e2d6908eb72dae7b91a362508c5044d24eeaef25fbf4ef796d3476ba9a597" Sep 16 07:57:34.913767 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:34.913743 2557 scope.go:117] "RemoveContainer" containerID="fdf0080e0c53db9025430f059f427d01cb38da500561eff7b891644db91202cb" Sep 16 07:57:34.922548 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:34.922528 2557 scope.go:117] "RemoveContainer" containerID="d7f1deb379b4bf1e9b7d8935afb72f140fb4d713cf26d28920b6388ad3a44a4a" Sep 16 07:57:34.931198 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:34.931175 2557 scope.go:117] "RemoveContainer" containerID="a50eeef5376405a8e817c767b623b46131edb4100d01b4931d3100d6fc322971" Sep 16 07:57:34.938378 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:34.938357 2557 scope.go:117] "RemoveContainer" containerID="bad939a5394b89ce0235762e615e34ea7fc0ff43542fed262594c1bde44a0916" Sep 16 07:57:34.938693 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:57:34.938668 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bad939a5394b89ce0235762e615e34ea7fc0ff43542fed262594c1bde44a0916\": container with ID starting with bad939a5394b89ce0235762e615e34ea7fc0ff43542fed262594c1bde44a0916 not found: ID does not exist" containerID="bad939a5394b89ce0235762e615e34ea7fc0ff43542fed262594c1bde44a0916" Sep 16 07:57:34.938755 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:34.938708 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bad939a5394b89ce0235762e615e34ea7fc0ff43542fed262594c1bde44a0916"} err="failed to get container status \"bad939a5394b89ce0235762e615e34ea7fc0ff43542fed262594c1bde44a0916\": rpc error: code = NotFound desc = could not find container \"bad939a5394b89ce0235762e615e34ea7fc0ff43542fed262594c1bde44a0916\": container with ID starting with bad939a5394b89ce0235762e615e34ea7fc0ff43542fed262594c1bde44a0916 not found: ID does not exist" Sep 16 07:57:34.938755 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:34.938732 2557 scope.go:117] "RemoveContainer" containerID="042e2d6908eb72dae7b91a362508c5044d24eeaef25fbf4ef796d3476ba9a597" Sep 16 07:57:34.939027 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:57:34.939010 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"042e2d6908eb72dae7b91a362508c5044d24eeaef25fbf4ef796d3476ba9a597\": container with ID starting with 042e2d6908eb72dae7b91a362508c5044d24eeaef25fbf4ef796d3476ba9a597 not found: ID does not exist" containerID="042e2d6908eb72dae7b91a362508c5044d24eeaef25fbf4ef796d3476ba9a597" Sep 16 07:57:34.939077 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:34.939030 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"042e2d6908eb72dae7b91a362508c5044d24eeaef25fbf4ef796d3476ba9a597"} err="failed to get container status \"042e2d6908eb72dae7b91a362508c5044d24eeaef25fbf4ef796d3476ba9a597\": rpc error: code = NotFound desc = could not find container \"042e2d6908eb72dae7b91a362508c5044d24eeaef25fbf4ef796d3476ba9a597\": container with ID starting with 042e2d6908eb72dae7b91a362508c5044d24eeaef25fbf4ef796d3476ba9a597 not found: ID does not exist" Sep 16 07:57:34.939077 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:34.939044 2557 scope.go:117] "RemoveContainer" containerID="fdf0080e0c53db9025430f059f427d01cb38da500561eff7b891644db91202cb" Sep 16 07:57:34.939306 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:57:34.939282 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdf0080e0c53db9025430f059f427d01cb38da500561eff7b891644db91202cb\": container with ID starting with fdf0080e0c53db9025430f059f427d01cb38da500561eff7b891644db91202cb not found: ID does not exist" containerID="fdf0080e0c53db9025430f059f427d01cb38da500561eff7b891644db91202cb" Sep 16 07:57:34.939374 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:34.939313 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdf0080e0c53db9025430f059f427d01cb38da500561eff7b891644db91202cb"} err="failed to get container status \"fdf0080e0c53db9025430f059f427d01cb38da500561eff7b891644db91202cb\": rpc error: code = NotFound desc = could not find container \"fdf0080e0c53db9025430f059f427d01cb38da500561eff7b891644db91202cb\": container with ID starting with fdf0080e0c53db9025430f059f427d01cb38da500561eff7b891644db91202cb not found: ID does not exist" Sep 16 07:57:34.939374 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:34.939330 2557 scope.go:117] "RemoveContainer" containerID="d7f1deb379b4bf1e9b7d8935afb72f140fb4d713cf26d28920b6388ad3a44a4a" Sep 16 07:57:34.939578 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:57:34.939556 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7f1deb379b4bf1e9b7d8935afb72f140fb4d713cf26d28920b6388ad3a44a4a\": container with ID starting with d7f1deb379b4bf1e9b7d8935afb72f140fb4d713cf26d28920b6388ad3a44a4a not found: ID does not exist" containerID="d7f1deb379b4bf1e9b7d8935afb72f140fb4d713cf26d28920b6388ad3a44a4a" Sep 16 07:57:34.939618 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:34.939586 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7f1deb379b4bf1e9b7d8935afb72f140fb4d713cf26d28920b6388ad3a44a4a"} err="failed to get container status \"d7f1deb379b4bf1e9b7d8935afb72f140fb4d713cf26d28920b6388ad3a44a4a\": rpc error: code = NotFound desc = could not find container \"d7f1deb379b4bf1e9b7d8935afb72f140fb4d713cf26d28920b6388ad3a44a4a\": container with ID starting with d7f1deb379b4bf1e9b7d8935afb72f140fb4d713cf26d28920b6388ad3a44a4a not found: ID does not exist" Sep 16 07:57:34.939618 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:34.939604 2557 scope.go:117] "RemoveContainer" containerID="a50eeef5376405a8e817c767b623b46131edb4100d01b4931d3100d6fc322971" Sep 16 07:57:34.939877 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:57:34.939858 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a50eeef5376405a8e817c767b623b46131edb4100d01b4931d3100d6fc322971\": container with ID starting with a50eeef5376405a8e817c767b623b46131edb4100d01b4931d3100d6fc322971 not found: ID does not exist" containerID="a50eeef5376405a8e817c767b623b46131edb4100d01b4931d3100d6fc322971" Sep 16 07:57:34.939920 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:34.939882 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a50eeef5376405a8e817c767b623b46131edb4100d01b4931d3100d6fc322971"} err="failed to get container status \"a50eeef5376405a8e817c767b623b46131edb4100d01b4931d3100d6fc322971\": rpc error: code = NotFound desc = could not find container \"a50eeef5376405a8e817c767b623b46131edb4100d01b4931d3100d6fc322971\": container with ID starting with a50eeef5376405a8e817c767b623b46131edb4100d01b4931d3100d6fc322971 not found: ID does not exist" Sep 16 07:57:34.993370 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:34.993284 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/31920438-c2e3-41c2-a9a2-f67afab9dfe1-feast-data\") pod \"31920438-c2e3-41c2-a9a2-f67afab9dfe1\" (UID: \"31920438-c2e3-41c2-a9a2-f67afab9dfe1\") " Sep 16 07:57:34.993370 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:34.993346 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/31920438-c2e3-41c2-a9a2-f67afab9dfe1-registry-tls\") pod \"31920438-c2e3-41c2-a9a2-f67afab9dfe1\" (UID: \"31920438-c2e3-41c2-a9a2-f67afab9dfe1\") " Sep 16 07:57:34.993572 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:34.993388 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/31920438-c2e3-41c2-a9a2-f67afab9dfe1-online-tls\") pod \"31920438-c2e3-41c2-a9a2-f67afab9dfe1\" (UID: \"31920438-c2e3-41c2-a9a2-f67afab9dfe1\") " Sep 16 07:57:34.993572 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:34.993442 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k624t\" (UniqueName: \"kubernetes.io/projected/31920438-c2e3-41c2-a9a2-f67afab9dfe1-kube-api-access-k624t\") pod \"31920438-c2e3-41c2-a9a2-f67afab9dfe1\" (UID: \"31920438-c2e3-41c2-a9a2-f67afab9dfe1\") " Sep 16 07:57:34.993572 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:34.993502 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/31920438-c2e3-41c2-a9a2-f67afab9dfe1-offline-tls\") pod \"31920438-c2e3-41c2-a9a2-f67afab9dfe1\" (UID: \"31920438-c2e3-41c2-a9a2-f67afab9dfe1\") " Sep 16 07:57:34.993572 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:34.993545 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/31920438-c2e3-41c2-a9a2-f67afab9dfe1-ui-tls\") pod \"31920438-c2e3-41c2-a9a2-f67afab9dfe1\" (UID: \"31920438-c2e3-41c2-a9a2-f67afab9dfe1\") " Sep 16 07:57:34.993772 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:34.993745 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31920438-c2e3-41c2-a9a2-f67afab9dfe1-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "31920438-c2e3-41c2-a9a2-f67afab9dfe1" (UID: "31920438-c2e3-41c2-a9a2-f67afab9dfe1"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 07:57:34.993863 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:34.993834 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31920438-c2e3-41c2-a9a2-f67afab9dfe1-feast-data" (OuterVolumeSpecName: "feast-data") pod "31920438-c2e3-41c2-a9a2-f67afab9dfe1" (UID: "31920438-c2e3-41c2-a9a2-f67afab9dfe1"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 07:57:34.995722 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:34.995696 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31920438-c2e3-41c2-a9a2-f67afab9dfe1-kube-api-access-k624t" (OuterVolumeSpecName: "kube-api-access-k624t") pod "31920438-c2e3-41c2-a9a2-f67afab9dfe1" (UID: "31920438-c2e3-41c2-a9a2-f67afab9dfe1"). InnerVolumeSpecName "kube-api-access-k624t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 07:57:34.995822 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:34.995754 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31920438-c2e3-41c2-a9a2-f67afab9dfe1-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "31920438-c2e3-41c2-a9a2-f67afab9dfe1" (UID: "31920438-c2e3-41c2-a9a2-f67afab9dfe1"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 07:57:34.995822 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:34.995786 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31920438-c2e3-41c2-a9a2-f67afab9dfe1-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "31920438-c2e3-41c2-a9a2-f67afab9dfe1" (UID: "31920438-c2e3-41c2-a9a2-f67afab9dfe1"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 07:57:34.995973 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:34.995957 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31920438-c2e3-41c2-a9a2-f67afab9dfe1-online-tls" (OuterVolumeSpecName: "online-tls") pod "31920438-c2e3-41c2-a9a2-f67afab9dfe1" (UID: "31920438-c2e3-41c2-a9a2-f67afab9dfe1"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 07:57:35.015803 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:35.015780 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-cvn72" Sep 16 07:57:35.094309 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:35.094271 2557 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/31920438-c2e3-41c2-a9a2-f67afab9dfe1-registry-tls\") on node \"ip-10-0-141-221.ec2.internal\" DevicePath \"\"" Sep 16 07:57:35.094462 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:35.094324 2557 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/31920438-c2e3-41c2-a9a2-f67afab9dfe1-online-tls\") on node \"ip-10-0-141-221.ec2.internal\" DevicePath \"\"" Sep 16 07:57:35.094462 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:35.094334 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k624t\" (UniqueName: \"kubernetes.io/projected/31920438-c2e3-41c2-a9a2-f67afab9dfe1-kube-api-access-k624t\") on node \"ip-10-0-141-221.ec2.internal\" DevicePath \"\"" Sep 16 07:57:35.094462 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:35.094345 2557 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/31920438-c2e3-41c2-a9a2-f67afab9dfe1-offline-tls\") on node \"ip-10-0-141-221.ec2.internal\" DevicePath \"\"" Sep 16 07:57:35.094462 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:35.094358 2557 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/31920438-c2e3-41c2-a9a2-f67afab9dfe1-ui-tls\") on node \"ip-10-0-141-221.ec2.internal\" DevicePath \"\"" Sep 16 07:57:35.094462 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:35.094366 2557 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/31920438-c2e3-41c2-a9a2-f67afab9dfe1-feast-data\") on node \"ip-10-0-141-221.ec2.internal\" DevicePath \"\"" Sep 16 07:57:35.195265 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:35.195223 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e-offline-tls\") pod \"a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e\" (UID: \"a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e\") " Sep 16 07:57:35.195361 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:35.195277 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e-online-tls\") pod \"a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e\" (UID: \"a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e\") " Sep 16 07:57:35.195361 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:35.195306 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e-registry-tls\") pod \"a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e\" (UID: \"a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e\") " Sep 16 07:57:35.195361 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:35.195331 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e-ui-tls\") pod \"a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e\" (UID: \"a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e\") " Sep 16 07:57:35.195361 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:35.195350 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n66rn\" (UniqueName: \"kubernetes.io/projected/a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e-kube-api-access-n66rn\") pod \"a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e\" (UID: \"a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e\") " Sep 16 07:57:35.195503 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:35.195377 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e-feast-data\") pod \"a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e\" (UID: \"a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e\") " Sep 16 07:57:35.196049 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:35.196005 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e-feast-data" (OuterVolumeSpecName: "feast-data") pod "a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e" (UID: "a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 07:57:35.197521 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:35.197490 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e" (UID: "a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 07:57:35.197521 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:35.197508 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e" (UID: "a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 07:57:35.197644 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:35.197555 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e" (UID: "a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 07:57:35.197878 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:35.197827 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e-online-tls" (OuterVolumeSpecName: "online-tls") pod "a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e" (UID: "a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 07:57:35.197878 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:35.197850 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e-kube-api-access-n66rn" (OuterVolumeSpecName: "kube-api-access-n66rn") pod "a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e" (UID: "a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e"). InnerVolumeSpecName "kube-api-access-n66rn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 07:57:35.220306 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:35.220280 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5bjgf"] Sep 16 07:57:35.223838 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:35.223817 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5bjgf"] Sep 16 07:57:35.296998 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:35.296926 2557 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e-offline-tls\") on node \"ip-10-0-141-221.ec2.internal\" DevicePath \"\"" Sep 16 07:57:35.296998 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:35.296948 2557 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e-online-tls\") on node \"ip-10-0-141-221.ec2.internal\" DevicePath \"\"" Sep 16 07:57:35.296998 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:35.296957 2557 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e-registry-tls\") on node \"ip-10-0-141-221.ec2.internal\" DevicePath \"\"" Sep 16 07:57:35.296998 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:35.296966 2557 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e-ui-tls\") on node \"ip-10-0-141-221.ec2.internal\" DevicePath \"\"" Sep 16 07:57:35.296998 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:35.296978 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n66rn\" (UniqueName: \"kubernetes.io/projected/a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e-kube-api-access-n66rn\") on node \"ip-10-0-141-221.ec2.internal\" DevicePath \"\"" Sep 16 07:57:35.296998 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:35.296989 2557 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e-feast-data\") on node \"ip-10-0-141-221.ec2.internal\" DevicePath \"\"" Sep 16 07:57:35.523869 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:35.523835 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31920438-c2e3-41c2-a9a2-f67afab9dfe1" path="/var/lib/kubelet/pods/31920438-c2e3-41c2-a9a2-f67afab9dfe1/volumes" Sep 16 07:57:35.904312 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:35.904272 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-cvn72" event={"ID":"a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e","Type":"ContainerDied","Data":"0013ce7b3c4207d2694fd27731d152ed811ffc22e0ab2a1c1ec649c357c28575"} Sep 16 07:57:35.904312 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:35.904311 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-cvn72" Sep 16 07:57:35.904857 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:35.904336 2557 scope.go:117] "RemoveContainer" containerID="078175717de5a7d4e5903efba2fd731e6f3d7cfb3acebbacd1946fc2a0483757" Sep 16 07:57:35.912392 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:35.912370 2557 scope.go:117] "RemoveContainer" containerID="e6dfa2954fc67484d28878e1a9f1905e7aa7449d8a28f40de1cf43b5112344bb" Sep 16 07:57:35.919926 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:35.919909 2557 scope.go:117] "RemoveContainer" containerID="07c05265b4f6a09eae295f09615aa631d5ce9ecf421117dc89bfdf905175d715" Sep 16 07:57:35.927131 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:35.927111 2557 scope.go:117] "RemoveContainer" containerID="cc9e9a6cd9164e255c86d7c77cb267adc2cfb4b54b43ac4fefec8974b92382a8" Sep 16 07:57:35.931817 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:35.931795 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-cvn72"] Sep 16 07:57:35.933835 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:35.933813 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-cvn72"] Sep 16 07:57:35.936438 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:35.936419 2557 scope.go:117] "RemoveContainer" containerID="72c9b0ed5138685abee26955d261b4fbc53e4207a4301b405014f105a20cec1d" Sep 16 07:57:35.943742 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:35.943723 2557 scope.go:117] "RemoveContainer" containerID="dcecc26cd76a210b8a1609d3002c0b27d29de4c1a935d7dfd1b9dbf0dd169f4f" Sep 16 07:57:37.524256 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:37.524208 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e" path="/var/lib/kubelet/pods/a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e/volumes" Sep 16 07:57:49.575119 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:49.575089 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-689b877b4b-f4d2x"] Sep 16 07:57:49.575528 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:49.575293 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="feast-operator-system/feast-operator-controller-manager-689b877b4b-f4d2x" podUID="07feeadd-44f9-4237-aebd-0cef7ce23acc" containerName="manager" containerID="cri-o://f1117cb73955ce35387d2bf734353df2f63156d6262b34bb98ccd79223d6f775" gracePeriod=10 Sep 16 07:57:49.821498 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:49.821473 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-689b877b4b-f4d2x" Sep 16 07:57:49.960543 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:49.960509 2557 generic.go:356] "Generic (PLEG): container finished" podID="07feeadd-44f9-4237-aebd-0cef7ce23acc" containerID="f1117cb73955ce35387d2bf734353df2f63156d6262b34bb98ccd79223d6f775" exitCode=0 Sep 16 07:57:49.960722 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:49.960551 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-689b877b4b-f4d2x" Sep 16 07:57:49.960722 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:49.960586 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-689b877b4b-f4d2x" event={"ID":"07feeadd-44f9-4237-aebd-0cef7ce23acc","Type":"ContainerDied","Data":"f1117cb73955ce35387d2bf734353df2f63156d6262b34bb98ccd79223d6f775"} Sep 16 07:57:49.960722 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:49.960627 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-689b877b4b-f4d2x" event={"ID":"07feeadd-44f9-4237-aebd-0cef7ce23acc","Type":"ContainerDied","Data":"88432d7c5ce838f687fe5f1a1d1f222af6443b8d9871a6dd8d8bf65df92ae153"} Sep 16 07:57:49.960722 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:49.960647 2557 scope.go:117] "RemoveContainer" containerID="f1117cb73955ce35387d2bf734353df2f63156d6262b34bb98ccd79223d6f775" Sep 16 07:57:49.968189 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:49.968170 2557 scope.go:117] "RemoveContainer" containerID="f1117cb73955ce35387d2bf734353df2f63156d6262b34bb98ccd79223d6f775" Sep 16 07:57:49.968495 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:57:49.968476 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1117cb73955ce35387d2bf734353df2f63156d6262b34bb98ccd79223d6f775\": container with ID starting with f1117cb73955ce35387d2bf734353df2f63156d6262b34bb98ccd79223d6f775 not found: ID does not exist" containerID="f1117cb73955ce35387d2bf734353df2f63156d6262b34bb98ccd79223d6f775" Sep 16 07:57:49.968568 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:49.968508 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1117cb73955ce35387d2bf734353df2f63156d6262b34bb98ccd79223d6f775"} err="failed to get container status \"f1117cb73955ce35387d2bf734353df2f63156d6262b34bb98ccd79223d6f775\": rpc error: code = NotFound desc = could not find container \"f1117cb73955ce35387d2bf734353df2f63156d6262b34bb98ccd79223d6f775\": container with ID starting with f1117cb73955ce35387d2bf734353df2f63156d6262b34bb98ccd79223d6f775 not found: ID does not exist" Sep 16 07:57:50.004776 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:50.004748 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hgjhz\" (UniqueName: \"kubernetes.io/projected/07feeadd-44f9-4237-aebd-0cef7ce23acc-kube-api-access-hgjhz\") pod \"07feeadd-44f9-4237-aebd-0cef7ce23acc\" (UID: \"07feeadd-44f9-4237-aebd-0cef7ce23acc\") " Sep 16 07:57:50.006867 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:50.006836 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07feeadd-44f9-4237-aebd-0cef7ce23acc-kube-api-access-hgjhz" (OuterVolumeSpecName: "kube-api-access-hgjhz") pod "07feeadd-44f9-4237-aebd-0cef7ce23acc" (UID: "07feeadd-44f9-4237-aebd-0cef7ce23acc"). InnerVolumeSpecName "kube-api-access-hgjhz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 07:57:50.106075 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:50.106040 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgjhz\" (UniqueName: \"kubernetes.io/projected/07feeadd-44f9-4237-aebd-0cef7ce23acc-kube-api-access-hgjhz\") on node \"ip-10-0-141-221.ec2.internal\" DevicePath \"\"" Sep 16 07:57:50.283734 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:50.283698 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-689b877b4b-f4d2x"] Sep 16 07:57:50.289628 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:50.289595 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-689b877b4b-f4d2x"] Sep 16 07:57:51.522735 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:51.522694 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="07feeadd-44f9-4237-aebd-0cef7ce23acc" path="/var/lib/kubelet/pods/07feeadd-44f9-4237-aebd-0cef7ce23acc/volumes" Sep 16 07:57:54.495744 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:54.495710 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["feast-operator-system/feast-operator-controller-manager-689b877b4b-5m4nl"] Sep 16 07:57:54.496095 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:54.495955 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e" containerName="feast-init" Sep 16 07:57:54.496095 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:54.495966 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e" containerName="feast-init" Sep 16 07:57:54.496095 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:54.495975 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e" containerName="ui" Sep 16 07:57:54.496095 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:54.495980 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e" containerName="ui" Sep 16 07:57:54.496095 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:54.495991 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e" containerName="feast-apply" Sep 16 07:57:54.496095 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:54.495997 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e" containerName="feast-apply" Sep 16 07:57:54.496095 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:54.496008 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07feeadd-44f9-4237-aebd-0cef7ce23acc" containerName="manager" Sep 16 07:57:54.496095 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:54.496013 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="07feeadd-44f9-4237-aebd-0cef7ce23acc" containerName="manager" Sep 16 07:57:54.496095 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:54.496020 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e" containerName="registry" Sep 16 07:57:54.496095 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:54.496024 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e" containerName="registry" Sep 16 07:57:54.496095 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:54.496031 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e" containerName="online" Sep 16 07:57:54.496095 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:54.496036 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e" containerName="online" Sep 16 07:57:54.496095 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:54.496041 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e" containerName="offline" Sep 16 07:57:54.496095 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:54.496046 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e" containerName="offline" Sep 16 07:57:54.496095 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:54.496051 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31920438-c2e3-41c2-a9a2-f67afab9dfe1" containerName="offline" Sep 16 07:57:54.496095 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:54.496056 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="31920438-c2e3-41c2-a9a2-f67afab9dfe1" containerName="offline" Sep 16 07:57:54.496095 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:54.496061 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31920438-c2e3-41c2-a9a2-f67afab9dfe1" containerName="ui" Sep 16 07:57:54.496095 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:54.496066 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="31920438-c2e3-41c2-a9a2-f67afab9dfe1" containerName="ui" Sep 16 07:57:54.496095 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:54.496073 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31920438-c2e3-41c2-a9a2-f67afab9dfe1" containerName="online" Sep 16 07:57:54.496095 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:54.496078 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="31920438-c2e3-41c2-a9a2-f67afab9dfe1" containerName="online" Sep 16 07:57:54.496095 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:54.496084 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31920438-c2e3-41c2-a9a2-f67afab9dfe1" containerName="feast-init" Sep 16 07:57:54.496095 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:54.496089 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="31920438-c2e3-41c2-a9a2-f67afab9dfe1" containerName="feast-init" Sep 16 07:57:54.496095 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:54.496094 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31920438-c2e3-41c2-a9a2-f67afab9dfe1" containerName="feast-apply" Sep 16 07:57:54.496095 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:54.496098 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="31920438-c2e3-41c2-a9a2-f67afab9dfe1" containerName="feast-apply" Sep 16 07:57:54.496820 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:54.496138 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="31920438-c2e3-41c2-a9a2-f67afab9dfe1" containerName="online" Sep 16 07:57:54.496820 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:54.496145 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="07feeadd-44f9-4237-aebd-0cef7ce23acc" containerName="manager" Sep 16 07:57:54.496820 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:54.496150 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="31920438-c2e3-41c2-a9a2-f67afab9dfe1" containerName="ui" Sep 16 07:57:54.496820 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:54.496155 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="31920438-c2e3-41c2-a9a2-f67afab9dfe1" containerName="offline" Sep 16 07:57:54.496820 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:54.496161 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e" containerName="registry" Sep 16 07:57:54.496820 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:54.496168 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e" containerName="offline" Sep 16 07:57:54.496820 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:54.496172 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e" containerName="ui" Sep 16 07:57:54.496820 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:54.496178 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="a173f8ce-a5ea-4567-9dbc-8f3fc03c7b3e" containerName="online" Sep 16 07:57:54.498884 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:54.498869 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-689b877b4b-5m4nl" Sep 16 07:57:54.501248 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:54.501217 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"feast-operator-system\"/\"kube-root-ca.crt\"" Sep 16 07:57:54.501353 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:54.501247 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"feast-operator-system\"/\"feast-operator-controller-manager-dockercfg-85vbd\"" Sep 16 07:57:54.502064 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:54.502048 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"feast-operator-system\"/\"openshift-service-ca.crt\"" Sep 16 07:57:54.510726 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:54.510706 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-689b877b4b-5m4nl"] Sep 16 07:57:54.536636 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:54.536612 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bskj7\" (UniqueName: \"kubernetes.io/projected/cd144e26-7815-4198-badd-07ac51f18e39-kube-api-access-bskj7\") pod \"feast-operator-controller-manager-689b877b4b-5m4nl\" (UID: \"cd144e26-7815-4198-badd-07ac51f18e39\") " pod="feast-operator-system/feast-operator-controller-manager-689b877b4b-5m4nl" Sep 16 07:57:54.637430 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:54.637404 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bskj7\" (UniqueName: \"kubernetes.io/projected/cd144e26-7815-4198-badd-07ac51f18e39-kube-api-access-bskj7\") pod \"feast-operator-controller-manager-689b877b4b-5m4nl\" (UID: \"cd144e26-7815-4198-badd-07ac51f18e39\") " pod="feast-operator-system/feast-operator-controller-manager-689b877b4b-5m4nl" Sep 16 07:57:54.653064 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:54.653040 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bskj7\" (UniqueName: \"kubernetes.io/projected/cd144e26-7815-4198-badd-07ac51f18e39-kube-api-access-bskj7\") pod \"feast-operator-controller-manager-689b877b4b-5m4nl\" (UID: \"cd144e26-7815-4198-badd-07ac51f18e39\") " pod="feast-operator-system/feast-operator-controller-manager-689b877b4b-5m4nl" Sep 16 07:57:54.808851 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:54.808780 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-689b877b4b-5m4nl" Sep 16 07:57:55.075076 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:55.075009 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-689b877b4b-5m4nl"] Sep 16 07:57:55.079114 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:57:55.079088 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd144e26_7815_4198_badd_07ac51f18e39.slice/crio-df55dd9c2d99d19411204bc2b6d027a6faa0f38501f689f7bd0ce1985a8ae50b WatchSource:0}: Error finding container df55dd9c2d99d19411204bc2b6d027a6faa0f38501f689f7bd0ce1985a8ae50b: Status 404 returned error can't find the container with id df55dd9c2d99d19411204bc2b6d027a6faa0f38501f689f7bd0ce1985a8ae50b Sep 16 07:57:55.981710 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:55.981677 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-689b877b4b-5m4nl" event={"ID":"cd144e26-7815-4198-badd-07ac51f18e39","Type":"ContainerStarted","Data":"8301f7d9eeee40aa68e33a32fcdad415315f6d924e62d8db13db372db9365ab0"} Sep 16 07:57:55.982081 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:55.981728 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-689b877b4b-5m4nl" event={"ID":"cd144e26-7815-4198-badd-07ac51f18e39","Type":"ContainerStarted","Data":"df55dd9c2d99d19411204bc2b6d027a6faa0f38501f689f7bd0ce1985a8ae50b"} Sep 16 07:57:55.982081 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:55.981816 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="feast-operator-system/feast-operator-controller-manager-689b877b4b-5m4nl" Sep 16 07:57:55.998025 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:57:55.997979 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="feast-operator-system/feast-operator-controller-manager-689b877b4b-5m4nl" podStartSLOduration=1.99796702 podStartE2EDuration="1.99796702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:57:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:57:55.99707205 +0000 UTC m=+1809.023973083" watchObservedRunningTime="2026-09-16 07:57:55.99796702 +0000 UTC m=+1809.024868033" Sep 16 07:58:06.987583 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:58:06.987555 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="feast-operator-system/feast-operator-controller-manager-689b877b4b-5m4nl" Sep 16 07:58:14.242138 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:58:14.242096 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-87h77"] Sep 16 07:58:14.248464 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:58:14.248443 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-87h77" Sep 16 07:58:14.250910 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:58:14.250889 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-ui-tls\"" Sep 16 07:58:14.251029 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:58:14.250923 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\"" Sep 16 07:58:14.251029 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:58:14.250928 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-registry-tls\"" Sep 16 07:58:14.251029 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:58:14.250920 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-online-tls\"" Sep 16 07:58:14.251029 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:58:14.250906 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-56xcf\"" Sep 16 07:58:14.251257 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:58:14.251038 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\"" Sep 16 07:58:14.251257 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:58:14.250911 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-offline-tls\"" Sep 16 07:58:14.256033 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:58:14.256011 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-87h77"] Sep 16 07:58:14.270900 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:58:14.270880 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-87h77\" (UID: \"6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-87h77" Sep 16 07:58:14.270998 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:58:14.270910 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-87h77\" (UID: \"6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-87h77" Sep 16 07:58:14.270998 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:58:14.270934 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-87h77\" (UID: \"6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-87h77" Sep 16 07:58:14.270998 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:58:14.270976 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18-feast-data\") pod \"feast-simple-feast-setup-57576b8fcf-87h77\" (UID: \"6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-87h77" Sep 16 07:58:14.271103 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:58:14.271007 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkfr4\" (UniqueName: \"kubernetes.io/projected/6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18-kube-api-access-wkfr4\") pod \"feast-simple-feast-setup-57576b8fcf-87h77\" (UID: \"6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-87h77" Sep 16 07:58:14.271103 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:58:14.271032 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-87h77\" (UID: \"6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-87h77" Sep 16 07:58:14.372087 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:58:14.372055 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-87h77\" (UID: \"6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-87h77" Sep 16 07:58:14.372087 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:58:14.372092 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-87h77\" (UID: \"6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-87h77" Sep 16 07:58:14.372302 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:58:14.372116 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-87h77\" (UID: \"6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-87h77" Sep 16 07:58:14.372302 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:58:14.372133 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18-feast-data\") pod \"feast-simple-feast-setup-57576b8fcf-87h77\" (UID: \"6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-87h77" Sep 16 07:58:14.372302 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:58:14.372153 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkfr4\" (UniqueName: \"kubernetes.io/projected/6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18-kube-api-access-wkfr4\") pod \"feast-simple-feast-setup-57576b8fcf-87h77\" (UID: \"6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-87h77" Sep 16 07:58:14.372302 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:58:14.372185 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-87h77\" (UID: \"6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-87h77" Sep 16 07:58:14.372302 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:58:14.372206 2557 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-registry-tls: secret "feast-simple-feast-setup-registry-tls" not found Sep 16 07:58:14.372302 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:58:14.372215 2557 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-online-tls: secret "feast-simple-feast-setup-online-tls" not found Sep 16 07:58:14.372302 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:58:14.372301 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18-registry-tls podName:6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18 nodeName:}" failed. No retries permitted until 2026-09-16 07:58:14.872282782 +0000 UTC m=+1827.899183772 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18-registry-tls") pod "feast-simple-feast-setup-57576b8fcf-87h77" (UID: "6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18") : secret "feast-simple-feast-setup-registry-tls" not found Sep 16 07:58:14.372561 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:58:14.372325 2557 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-offline-tls: secret "feast-simple-feast-setup-offline-tls" not found Sep 16 07:58:14.372561 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:58:14.372207 2557 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-ui-tls: secret "feast-simple-feast-setup-ui-tls" not found Sep 16 07:58:14.372561 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:58:14.372327 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18-online-tls podName:6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18 nodeName:}" failed. No retries permitted until 2026-09-16 07:58:14.872317797 +0000 UTC m=+1827.899218794 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18-online-tls") pod "feast-simple-feast-setup-57576b8fcf-87h77" (UID: "6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18") : secret "feast-simple-feast-setup-online-tls" not found Sep 16 07:58:14.372561 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:58:14.372422 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18-offline-tls podName:6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18 nodeName:}" failed. No retries permitted until 2026-09-16 07:58:14.872403592 +0000 UTC m=+1827.899304587 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18-offline-tls") pod "feast-simple-feast-setup-57576b8fcf-87h77" (UID: "6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18") : secret "feast-simple-feast-setup-offline-tls" not found Sep 16 07:58:14.372561 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:58:14.372436 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18-ui-tls podName:6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18 nodeName:}" failed. No retries permitted until 2026-09-16 07:58:14.872428297 +0000 UTC m=+1827.899329295 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "ui-tls" (UniqueName: "kubernetes.io/secret/6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18-ui-tls") pod "feast-simple-feast-setup-57576b8fcf-87h77" (UID: "6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18") : secret "feast-simple-feast-setup-ui-tls" not found Sep 16 07:58:14.372561 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:58:14.372544 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18-feast-data\") pod \"feast-simple-feast-setup-57576b8fcf-87h77\" (UID: \"6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-87h77" Sep 16 07:58:14.382616 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:58:14.382592 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkfr4\" (UniqueName: \"kubernetes.io/projected/6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18-kube-api-access-wkfr4\") pod \"feast-simple-feast-setup-57576b8fcf-87h77\" (UID: \"6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-87h77" Sep 16 07:58:14.876329 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:58:14.876293 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-87h77\" (UID: \"6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-87h77" Sep 16 07:58:14.876495 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:58:14.876339 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-87h77\" (UID: \"6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-87h77" Sep 16 07:58:14.876495 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:58:14.876372 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-87h77\" (UID: \"6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-87h77" Sep 16 07:58:14.876495 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:58:14.876392 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-87h77\" (UID: \"6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-87h77" Sep 16 07:58:14.876495 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:58:14.876450 2557 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-online-tls: secret "feast-simple-feast-setup-online-tls" not found Sep 16 07:58:14.876633 ip-10-0-141-221 kubenswrapper[2557]: E0916 07:58:14.876524 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18-online-tls podName:6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18 nodeName:}" failed. No retries permitted until 2026-09-16 07:58:15.876505165 +0000 UTC m=+1828.903406174 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18-online-tls") pod "feast-simple-feast-setup-57576b8fcf-87h77" (UID: "6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18") : secret "feast-simple-feast-setup-online-tls" not found Sep 16 07:58:14.878847 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:58:14.878818 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-87h77\" (UID: \"6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-87h77" Sep 16 07:58:14.878939 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:58:14.878908 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-87h77\" (UID: \"6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-87h77" Sep 16 07:58:14.879012 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:58:14.878991 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-87h77\" (UID: \"6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-87h77" Sep 16 07:58:15.882831 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:58:15.882794 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-87h77\" (UID: \"6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-87h77" Sep 16 07:58:15.885154 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:58:15.885133 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-87h77\" (UID: \"6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-87h77" Sep 16 07:58:16.058420 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:58:16.058392 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-87h77" Sep 16 07:58:16.332001 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:58:16.331974 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-87h77"] Sep 16 07:58:16.333347 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:58:16.333321 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6c1047a7_8f58_4fce_b7ce_fcdc6ff34b18.slice/crio-a000a911d5d7800f46e7581f94755cbdc14bf3722c9643d001df98eed5755ee5 WatchSource:0}: Error finding container a000a911d5d7800f46e7581f94755cbdc14bf3722c9643d001df98eed5755ee5: Status 404 returned error can't find the container with id a000a911d5d7800f46e7581f94755cbdc14bf3722c9643d001df98eed5755ee5 Sep 16 07:58:17.054660 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:58:17.054625 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-87h77" event={"ID":"6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18","Type":"ContainerStarted","Data":"fd98684ba596b5141d9e48951f7877ba071c363f8b683feea58828338eb34c60"} Sep 16 07:58:17.055054 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:58:17.054673 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-87h77" event={"ID":"6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18","Type":"ContainerStarted","Data":"a000a911d5d7800f46e7581f94755cbdc14bf3722c9643d001df98eed5755ee5"} Sep 16 07:58:22.076101 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:58:22.076064 2557 generic.go:356] "Generic (PLEG): container finished" podID="6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18" containerID="fd98684ba596b5141d9e48951f7877ba071c363f8b683feea58828338eb34c60" exitCode=0 Sep 16 07:58:22.076490 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:58:22.076136 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-87h77" event={"ID":"6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18","Type":"ContainerDied","Data":"fd98684ba596b5141d9e48951f7877ba071c363f8b683feea58828338eb34c60"} Sep 16 07:58:23.082215 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:58:23.082177 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-87h77" event={"ID":"6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18","Type":"ContainerStarted","Data":"82e59c96d3d54ac0e824f1db1cb3fe0b405f9439e875c0b31a578a05dc136424"} Sep 16 07:58:28.100794 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:58:28.100708 2557 generic.go:356] "Generic (PLEG): container finished" podID="6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18" containerID="82e59c96d3d54ac0e824f1db1cb3fe0b405f9439e875c0b31a578a05dc136424" exitCode=0 Sep 16 07:58:28.100794 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:58:28.100783 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-87h77" event={"ID":"6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18","Type":"ContainerDied","Data":"82e59c96d3d54ac0e824f1db1cb3fe0b405f9439e875c0b31a578a05dc136424"} Sep 16 07:58:29.110718 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:58:29.110677 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-87h77" event={"ID":"6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18","Type":"ContainerStarted","Data":"27334b4ccd87f3c4e079e396d89ddabffec03f65a49ad610144da32a9eecea80"} Sep 16 07:58:29.111200 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:58:29.110740 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-87h77" event={"ID":"6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18","Type":"ContainerStarted","Data":"6d4678e0b594abd1c0c7886b6c224f6103723c94978fe413e7341698ca917101"} Sep 16 07:58:29.111200 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:58:29.110763 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-87h77" event={"ID":"6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18","Type":"ContainerStarted","Data":"e20f881ea6c25eb0343ea885c45eaf186823ab2c9c6d825153a8005f5f3f7323"} Sep 16 07:58:29.111200 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:58:29.110783 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-87h77" event={"ID":"6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18","Type":"ContainerStarted","Data":"1bc14d1868aa738a870a4c48faa51123d949920302b52728ee025fa0609712c7"} Sep 16 07:58:29.142957 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:58:29.142897 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-87h77" podStartSLOduration=15.142877885 podStartE2EDuration="15.142877885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:58:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:58:29.142781371 +0000 UTC m=+1842.169682409" watchObservedRunningTime="2026-09-16 07:58:29.142877885 +0000 UTC m=+1842.169778898" Sep 16 07:58:31.058637 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:58:31.058599 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-87h77" Sep 16 07:58:31.059204 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:58:31.059185 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-87h77" Sep 16 07:58:31.059724 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:58:31.059709 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-87h77" Sep 16 07:58:31.059853 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:58:31.059842 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-87h77" Sep 16 07:58:31.060586 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:58:31.060556 2557 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-87h77" podUID="6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.24:8816: connect: connection refused" Sep 16 07:58:31.060780 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:58:31.060582 2557 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-87h77" podUID="6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.24:8443: connect: connection refused" Sep 16 07:58:31.060859 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:58:31.060653 2557 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-87h77" podUID="6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18" containerName="online" probeResult="failure" output="Get \"https://10.133.0.24:6567/health\": dial tcp 10.133.0.24:6567: connect: connection refused" Sep 16 07:58:31.060859 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:58:31.060666 2557 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-87h77" podUID="6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.24:6571: connect: connection refused" Sep 16 07:58:34.059198 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:58:34.059155 2557 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-87h77" podUID="6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.24:8816: connect: connection refused" Sep 16 07:58:34.060076 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:58:34.059155 2557 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-87h77" podUID="6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.24:8443: connect: connection refused" Sep 16 07:58:34.060187 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:58:34.059155 2557 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-87h77" podUID="6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.24:6571: connect: connection refused" Sep 16 07:58:34.060187 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:58:34.059209 2557 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-87h77" podUID="6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18" containerName="online" probeResult="failure" output="Get \"https://10.133.0.24:6567/health\": dial tcp 10.133.0.24:6567: connect: connection refused" Sep 16 07:58:36.059154 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:58:36.059110 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-87h77" Sep 16 07:58:36.059551 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:58:36.059167 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-87h77" Sep 16 07:58:36.059551 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:58:36.059185 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-87h77" Sep 16 07:58:36.059551 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:58:36.059196 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-87h77" Sep 16 07:58:37.059569 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:58:37.059537 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-87h77" Sep 16 07:58:37.060057 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:58:37.059655 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-87h77" Sep 16 07:58:37.060057 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:58:37.059827 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-87h77" Sep 16 07:58:37.060404 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:58:37.060381 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-87h77" Sep 16 07:58:37.060524 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:58:37.060451 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-87h77" Sep 16 07:58:37.060698 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:58:37.060680 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-87h77" Sep 16 07:58:37.064725 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:58:37.064708 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-87h77" Sep 16 07:58:37.152773 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:58:37.152748 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-87h77" Sep 16 07:59:39.743706 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:59:39.743668 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-87h77"] Sep 16 07:59:39.744320 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:59:39.743959 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-87h77" podUID="6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18" containerName="registry" containerID="cri-o://1bc14d1868aa738a870a4c48faa51123d949920302b52728ee025fa0609712c7" gracePeriod=30 Sep 16 07:59:39.744392 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:59:39.744312 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-87h77" podUID="6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18" containerName="online" containerID="cri-o://e20f881ea6c25eb0343ea885c45eaf186823ab2c9c6d825153a8005f5f3f7323" gracePeriod=30 Sep 16 07:59:39.744392 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:59:39.744320 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-87h77" podUID="6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18" containerName="offline" containerID="cri-o://6d4678e0b594abd1c0c7886b6c224f6103723c94978fe413e7341698ca917101" gracePeriod=30 Sep 16 07:59:39.744511 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:59:39.744444 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-87h77" podUID="6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18" containerName="ui" containerID="cri-o://27334b4ccd87f3c4e079e396d89ddabffec03f65a49ad610144da32a9eecea80" gracePeriod=30 Sep 16 07:59:40.224518 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:59:40.224466 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-c4bxp"] Sep 16 07:59:40.228560 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:59:40.228534 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-c4bxp" Sep 16 07:59:40.230919 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:59:40.230883 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-c8bhn\"" Sep 16 07:59:40.239876 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:59:40.239840 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/23ed5836-df96-41f7-bd69-129fd932f13c-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-c4bxp\" (UID: \"23ed5836-df96-41f7-bd69-129fd932f13c\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-c4bxp" Sep 16 07:59:40.240035 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:59:40.239923 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/23ed5836-df96-41f7-bd69-129fd932f13c-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-c4bxp\" (UID: \"23ed5836-df96-41f7-bd69-129fd932f13c\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-c4bxp" Sep 16 07:59:40.240035 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:59:40.239995 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/23ed5836-df96-41f7-bd69-129fd932f13c-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-c4bxp\" (UID: \"23ed5836-df96-41f7-bd69-129fd932f13c\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-c4bxp" Sep 16 07:59:40.240035 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:59:40.240027 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/23ed5836-df96-41f7-bd69-129fd932f13c-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-c4bxp\" (UID: \"23ed5836-df96-41f7-bd69-129fd932f13c\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-c4bxp" Sep 16 07:59:40.240186 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:59:40.240100 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kb4nh\" (UniqueName: \"kubernetes.io/projected/23ed5836-df96-41f7-bd69-129fd932f13c-kube-api-access-kb4nh\") pod \"feast-simple-feast-setup-57576b8fcf-c4bxp\" (UID: \"23ed5836-df96-41f7-bd69-129fd932f13c\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-c4bxp" Sep 16 07:59:40.240186 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:59:40.240142 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/23ed5836-df96-41f7-bd69-129fd932f13c-feast-data\") pod \"feast-simple-feast-setup-57576b8fcf-c4bxp\" (UID: \"23ed5836-df96-41f7-bd69-129fd932f13c\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-c4bxp" Sep 16 07:59:40.241821 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:59:40.241788 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-c4bxp"] Sep 16 07:59:40.341393 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:59:40.341348 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/23ed5836-df96-41f7-bd69-129fd932f13c-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-c4bxp\" (UID: \"23ed5836-df96-41f7-bd69-129fd932f13c\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-c4bxp" Sep 16 07:59:40.341393 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:59:40.341394 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/23ed5836-df96-41f7-bd69-129fd932f13c-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-c4bxp\" (UID: \"23ed5836-df96-41f7-bd69-129fd932f13c\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-c4bxp" Sep 16 07:59:40.341675 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:59:40.341415 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kb4nh\" (UniqueName: \"kubernetes.io/projected/23ed5836-df96-41f7-bd69-129fd932f13c-kube-api-access-kb4nh\") pod \"feast-simple-feast-setup-57576b8fcf-c4bxp\" (UID: \"23ed5836-df96-41f7-bd69-129fd932f13c\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-c4bxp" Sep 16 07:59:40.341675 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:59:40.341434 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/23ed5836-df96-41f7-bd69-129fd932f13c-feast-data\") pod \"feast-simple-feast-setup-57576b8fcf-c4bxp\" (UID: \"23ed5836-df96-41f7-bd69-129fd932f13c\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-c4bxp" Sep 16 07:59:40.341675 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:59:40.341471 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/23ed5836-df96-41f7-bd69-129fd932f13c-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-c4bxp\" (UID: \"23ed5836-df96-41f7-bd69-129fd932f13c\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-c4bxp" Sep 16 07:59:40.341675 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:59:40.341516 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/23ed5836-df96-41f7-bd69-129fd932f13c-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-c4bxp\" (UID: \"23ed5836-df96-41f7-bd69-129fd932f13c\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-c4bxp" Sep 16 07:59:40.341923 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:59:40.341878 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/23ed5836-df96-41f7-bd69-129fd932f13c-feast-data\") pod \"feast-simple-feast-setup-57576b8fcf-c4bxp\" (UID: \"23ed5836-df96-41f7-bd69-129fd932f13c\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-c4bxp" Sep 16 07:59:40.344223 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:59:40.344196 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/23ed5836-df96-41f7-bd69-129fd932f13c-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-c4bxp\" (UID: \"23ed5836-df96-41f7-bd69-129fd932f13c\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-c4bxp" Sep 16 07:59:40.344552 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:59:40.344531 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/23ed5836-df96-41f7-bd69-129fd932f13c-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-c4bxp\" (UID: \"23ed5836-df96-41f7-bd69-129fd932f13c\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-c4bxp" Sep 16 07:59:40.344623 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:59:40.344602 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/23ed5836-df96-41f7-bd69-129fd932f13c-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-c4bxp\" (UID: \"23ed5836-df96-41f7-bd69-129fd932f13c\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-c4bxp" Sep 16 07:59:40.344663 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:59:40.344610 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/23ed5836-df96-41f7-bd69-129fd932f13c-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-c4bxp\" (UID: \"23ed5836-df96-41f7-bd69-129fd932f13c\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-c4bxp" Sep 16 07:59:40.349637 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:59:40.349610 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kb4nh\" (UniqueName: \"kubernetes.io/projected/23ed5836-df96-41f7-bd69-129fd932f13c-kube-api-access-kb4nh\") pod \"feast-simple-feast-setup-57576b8fcf-c4bxp\" (UID: \"23ed5836-df96-41f7-bd69-129fd932f13c\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-c4bxp" Sep 16 07:59:40.539013 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:59:40.538917 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-c4bxp" Sep 16 07:59:40.816163 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:59:40.816134 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-c4bxp"] Sep 16 07:59:40.817785 ip-10-0-141-221 kubenswrapper[2557]: W0916 07:59:40.817759 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod23ed5836_df96_41f7_bd69_129fd932f13c.slice/crio-d517b192adf7d469f6f7760f14cbd008d497c1d13dbe4aebe471c9f9893fe1ee WatchSource:0}: Error finding container d517b192adf7d469f6f7760f14cbd008d497c1d13dbe4aebe471c9f9893fe1ee: Status 404 returned error can't find the container with id d517b192adf7d469f6f7760f14cbd008d497c1d13dbe4aebe471c9f9893fe1ee Sep 16 07:59:41.389134 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:59:41.389099 2557 generic.go:356] "Generic (PLEG): container finished" podID="6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18" containerID="27334b4ccd87f3c4e079e396d89ddabffec03f65a49ad610144da32a9eecea80" exitCode=0 Sep 16 07:59:41.389134 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:59:41.389123 2557 generic.go:356] "Generic (PLEG): container finished" podID="6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18" containerID="e20f881ea6c25eb0343ea885c45eaf186823ab2c9c6d825153a8005f5f3f7323" exitCode=0 Sep 16 07:59:41.389438 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:59:41.389168 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-87h77" event={"ID":"6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18","Type":"ContainerDied","Data":"27334b4ccd87f3c4e079e396d89ddabffec03f65a49ad610144da32a9eecea80"} Sep 16 07:59:41.389438 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:59:41.389206 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-87h77" event={"ID":"6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18","Type":"ContainerDied","Data":"e20f881ea6c25eb0343ea885c45eaf186823ab2c9c6d825153a8005f5f3f7323"} Sep 16 07:59:41.390597 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:59:41.390566 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-c4bxp" event={"ID":"23ed5836-df96-41f7-bd69-129fd932f13c","Type":"ContainerStarted","Data":"25c4b319b22cbeec8576f34ab28af3441b18d884596145296aa5803cb75be3e7"} Sep 16 07:59:41.390597 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:59:41.390600 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-c4bxp" event={"ID":"23ed5836-df96-41f7-bd69-129fd932f13c","Type":"ContainerStarted","Data":"d517b192adf7d469f6f7760f14cbd008d497c1d13dbe4aebe471c9f9893fe1ee"} Sep 16 07:59:46.412445 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:59:46.412412 2557 generic.go:356] "Generic (PLEG): container finished" podID="23ed5836-df96-41f7-bd69-129fd932f13c" containerID="25c4b319b22cbeec8576f34ab28af3441b18d884596145296aa5803cb75be3e7" exitCode=0 Sep 16 07:59:46.412820 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:59:46.412465 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-c4bxp" event={"ID":"23ed5836-df96-41f7-bd69-129fd932f13c","Type":"ContainerDied","Data":"25c4b319b22cbeec8576f34ab28af3441b18d884596145296aa5803cb75be3e7"} Sep 16 07:59:47.060649 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:59:47.060601 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-87h77" podUID="6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.24:8443: connect: connection refused" Sep 16 07:59:47.150115 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:59:47.150075 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-87h77" podUID="6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18" containerName="online" probeResult="failure" output="Get \"https://10.133.0.24:6567/health\": dial tcp 10.133.0.24:6567: connect: connection refused" Sep 16 07:59:47.418304 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:59:47.418260 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-c4bxp" event={"ID":"23ed5836-df96-41f7-bd69-129fd932f13c","Type":"ContainerStarted","Data":"354a581f7e8d5d3d9cde3c5fc653038c5a84b170c7cb21693e60828bab62edb8"} Sep 16 07:59:52.441291 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:59:52.441260 2557 generic.go:356] "Generic (PLEG): container finished" podID="23ed5836-df96-41f7-bd69-129fd932f13c" containerID="354a581f7e8d5d3d9cde3c5fc653038c5a84b170c7cb21693e60828bab62edb8" exitCode=0 Sep 16 07:59:52.441653 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:59:52.441326 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-c4bxp" event={"ID":"23ed5836-df96-41f7-bd69-129fd932f13c","Type":"ContainerDied","Data":"354a581f7e8d5d3d9cde3c5fc653038c5a84b170c7cb21693e60828bab62edb8"} Sep 16 07:59:53.451314 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:59:53.451275 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-c4bxp" event={"ID":"23ed5836-df96-41f7-bd69-129fd932f13c","Type":"ContainerStarted","Data":"7a02e32b0b1fe24f47bbb850df8fffbe6aafa15a7fc7303851ff4dfbd5c679e4"} Sep 16 07:59:53.451813 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:59:53.451336 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-c4bxp" event={"ID":"23ed5836-df96-41f7-bd69-129fd932f13c","Type":"ContainerStarted","Data":"013a1b2fe4d94dd257ebfa55a3deefb39aaf6dd5ebdbe8574d63d00144e8170a"} Sep 16 07:59:53.451813 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:59:53.451361 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-c4bxp" event={"ID":"23ed5836-df96-41f7-bd69-129fd932f13c","Type":"ContainerStarted","Data":"8dd4e9766e9c334a298a71f95c3acbb0820a63e4f615e62bc166f3234e4aad75"} Sep 16 07:59:53.451813 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:59:53.451378 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-c4bxp" event={"ID":"23ed5836-df96-41f7-bd69-129fd932f13c","Type":"ContainerStarted","Data":"b2e975bd990dd19b4cb1c0fdaaef0519eea2bec21666767dfa79e43ae8df250a"} Sep 16 07:59:53.477346 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:59:53.477275 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-c4bxp" podStartSLOduration=13.477251737 podStartE2EDuration="13.477251737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:59:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:59:53.474512179 +0000 UTC m=+1926.501413217" watchObservedRunningTime="2026-09-16 07:59:53.477251737 +0000 UTC m=+1926.504152748" Sep 16 07:59:55.539627 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:59:55.539595 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-c4bxp" Sep 16 07:59:55.540189 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:59:55.540156 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-c4bxp" Sep 16 07:59:55.540351 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:59:55.540195 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-c4bxp" Sep 16 07:59:55.540351 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:59:55.540215 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-c4bxp" Sep 16 07:59:55.541526 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:59:55.541489 2557 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-c4bxp" podUID="23ed5836-df96-41f7-bd69-129fd932f13c" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.25:8816: connect: connection refused" Sep 16 07:59:55.541526 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:59:55.541507 2557 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-c4bxp" podUID="23ed5836-df96-41f7-bd69-129fd932f13c" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.25:6571: connect: connection refused" Sep 16 07:59:55.541795 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:59:55.541576 2557 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-c4bxp" podUID="23ed5836-df96-41f7-bd69-129fd932f13c" containerName="online" probeResult="failure" output="Get \"https://10.133.0.25:6567/health\": dial tcp 10.133.0.25:6567: connect: connection refused" Sep 16 07:59:55.541795 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:59:55.541510 2557 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-c4bxp" podUID="23ed5836-df96-41f7-bd69-129fd932f13c" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.25:8443: connect: connection refused" Sep 16 07:59:57.061124 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:59:57.061064 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-87h77" podUID="6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.24:8443: connect: connection refused" Sep 16 07:59:57.150210 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:59:57.150142 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-87h77" podUID="6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18" containerName="online" probeResult="failure" output="Get \"https://10.133.0.24:6567/health\": dial tcp 10.133.0.24:6567: connect: connection refused" Sep 16 07:59:58.539710 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:59:58.539668 2557 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-c4bxp" podUID="23ed5836-df96-41f7-bd69-129fd932f13c" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.25:8443: connect: connection refused" Sep 16 07:59:58.540186 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:59:58.539666 2557 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-c4bxp" podUID="23ed5836-df96-41f7-bd69-129fd932f13c" containerName="online" probeResult="failure" output="Get \"https://10.133.0.25:6567/health\": dial tcp 10.133.0.25:6567: connect: connection refused" Sep 16 07:59:58.540186 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:59:58.539654 2557 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-c4bxp" podUID="23ed5836-df96-41f7-bd69-129fd932f13c" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.25:8816: connect: connection refused" Sep 16 07:59:58.540344 ip-10-0-141-221 kubenswrapper[2557]: I0916 07:59:58.539692 2557 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-c4bxp" podUID="23ed5836-df96-41f7-bd69-129fd932f13c" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.25:6571: connect: connection refused" Sep 16 08:00:00.539551 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:00:00.539520 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-c4bxp" Sep 16 08:00:00.539930 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:00:00.539582 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-c4bxp" Sep 16 08:00:00.539930 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:00:00.539597 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-c4bxp" Sep 16 08:00:00.539930 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:00:00.539608 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-c4bxp" Sep 16 08:00:01.540131 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:00:01.540105 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-c4bxp" Sep 16 08:00:01.540516 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:00:01.540196 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-c4bxp" Sep 16 08:00:01.540516 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:00:01.540430 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-c4bxp" Sep 16 08:00:01.545055 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:00:01.545033 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-c4bxp" Sep 16 08:00:02.495857 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:00:02.495831 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-c4bxp" Sep 16 08:00:02.496055 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:00:02.495887 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-c4bxp" Sep 16 08:00:02.496055 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:00:02.495921 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-c4bxp" Sep 16 08:00:02.498488 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:00:02.498462 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-c4bxp" Sep 16 08:00:07.060485 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:00:07.060434 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-87h77" podUID="6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.24:8443: connect: connection refused" Sep 16 08:00:07.060933 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:00:07.060689 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-87h77" Sep 16 08:00:07.150955 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:00:07.150913 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-87h77" podUID="6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18" containerName="online" probeResult="failure" output="Get \"https://10.133.0.24:6567/health\": dial tcp 10.133.0.24:6567: connect: connection refused" Sep 16 08:00:07.151117 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:00:07.151031 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-87h77" Sep 16 08:00:10.395558 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:00:10.395529 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-87h77" Sep 16 08:00:10.493447 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:00:10.493408 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18-online-tls\") pod \"6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18\" (UID: \"6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18\") " Sep 16 08:00:10.493447 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:00:10.493450 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wkfr4\" (UniqueName: \"kubernetes.io/projected/6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18-kube-api-access-wkfr4\") pod \"6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18\" (UID: \"6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18\") " Sep 16 08:00:10.493693 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:00:10.493478 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18-offline-tls\") pod \"6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18\" (UID: \"6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18\") " Sep 16 08:00:10.493693 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:00:10.493506 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18-registry-tls\") pod \"6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18\" (UID: \"6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18\") " Sep 16 08:00:10.493693 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:00:10.493607 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18-ui-tls\") pod \"6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18\" (UID: \"6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18\") " Sep 16 08:00:10.493693 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:00:10.493673 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18-feast-data\") pod \"6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18\" (UID: \"6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18\") " Sep 16 08:00:10.494292 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:00:10.494230 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18-feast-data" (OuterVolumeSpecName: "feast-data") pod "6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18" (UID: "6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 08:00:10.495929 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:00:10.495905 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18-kube-api-access-wkfr4" (OuterVolumeSpecName: "kube-api-access-wkfr4") pod "6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18" (UID: "6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18"). InnerVolumeSpecName "kube-api-access-wkfr4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 08:00:10.495929 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:00:10.495912 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18" (UID: "6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 08:00:10.496090 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:00:10.495938 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18" (UID: "6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 08:00:10.496090 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:00:10.496061 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18" (UID: "6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 08:00:10.496090 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:00:10.496083 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18-online-tls" (OuterVolumeSpecName: "online-tls") pod "6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18" (UID: "6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 08:00:10.527493 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:00:10.527423 2557 generic.go:356] "Generic (PLEG): container finished" podID="6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18" containerID="6d4678e0b594abd1c0c7886b6c224f6103723c94978fe413e7341698ca917101" exitCode=137 Sep 16 08:00:10.527493 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:00:10.527443 2557 generic.go:356] "Generic (PLEG): container finished" podID="6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18" containerID="1bc14d1868aa738a870a4c48faa51123d949920302b52728ee025fa0609712c7" exitCode=137 Sep 16 08:00:10.527493 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:00:10.527477 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-87h77" event={"ID":"6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18","Type":"ContainerDied","Data":"6d4678e0b594abd1c0c7886b6c224f6103723c94978fe413e7341698ca917101"} Sep 16 08:00:10.527708 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:00:10.527495 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-87h77" Sep 16 08:00:10.527708 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:00:10.527508 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-87h77" event={"ID":"6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18","Type":"ContainerDied","Data":"1bc14d1868aa738a870a4c48faa51123d949920302b52728ee025fa0609712c7"} Sep 16 08:00:10.527708 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:00:10.527522 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-87h77" event={"ID":"6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18","Type":"ContainerDied","Data":"a000a911d5d7800f46e7581f94755cbdc14bf3722c9643d001df98eed5755ee5"} Sep 16 08:00:10.527708 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:00:10.527551 2557 scope.go:117] "RemoveContainer" containerID="27334b4ccd87f3c4e079e396d89ddabffec03f65a49ad610144da32a9eecea80" Sep 16 08:00:10.538323 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:00:10.538301 2557 scope.go:117] "RemoveContainer" containerID="6d4678e0b594abd1c0c7886b6c224f6103723c94978fe413e7341698ca917101" Sep 16 08:00:10.546008 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:00:10.545990 2557 scope.go:117] "RemoveContainer" containerID="e20f881ea6c25eb0343ea885c45eaf186823ab2c9c6d825153a8005f5f3f7323" Sep 16 08:00:10.549934 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:00:10.549911 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-87h77"] Sep 16 08:00:10.554108 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:00:10.554089 2557 scope.go:117] "RemoveContainer" containerID="1bc14d1868aa738a870a4c48faa51123d949920302b52728ee025fa0609712c7" Sep 16 08:00:10.554108 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:00:10.554100 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-87h77"] Sep 16 08:00:10.561620 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:00:10.561605 2557 scope.go:117] "RemoveContainer" containerID="82e59c96d3d54ac0e824f1db1cb3fe0b405f9439e875c0b31a578a05dc136424" Sep 16 08:00:10.568399 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:00:10.568386 2557 scope.go:117] "RemoveContainer" containerID="fd98684ba596b5141d9e48951f7877ba071c363f8b683feea58828338eb34c60" Sep 16 08:00:10.575101 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:00:10.575082 2557 scope.go:117] "RemoveContainer" containerID="27334b4ccd87f3c4e079e396d89ddabffec03f65a49ad610144da32a9eecea80" Sep 16 08:00:10.575389 ip-10-0-141-221 kubenswrapper[2557]: E0916 08:00:10.575370 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27334b4ccd87f3c4e079e396d89ddabffec03f65a49ad610144da32a9eecea80\": container with ID starting with 27334b4ccd87f3c4e079e396d89ddabffec03f65a49ad610144da32a9eecea80 not found: ID does not exist" containerID="27334b4ccd87f3c4e079e396d89ddabffec03f65a49ad610144da32a9eecea80" Sep 16 08:00:10.575487 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:00:10.575395 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27334b4ccd87f3c4e079e396d89ddabffec03f65a49ad610144da32a9eecea80"} err="failed to get container status \"27334b4ccd87f3c4e079e396d89ddabffec03f65a49ad610144da32a9eecea80\": rpc error: code = NotFound desc = could not find container \"27334b4ccd87f3c4e079e396d89ddabffec03f65a49ad610144da32a9eecea80\": container with ID starting with 27334b4ccd87f3c4e079e396d89ddabffec03f65a49ad610144da32a9eecea80 not found: ID does not exist" Sep 16 08:00:10.575487 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:00:10.575412 2557 scope.go:117] "RemoveContainer" containerID="6d4678e0b594abd1c0c7886b6c224f6103723c94978fe413e7341698ca917101" Sep 16 08:00:10.575683 ip-10-0-141-221 kubenswrapper[2557]: E0916 08:00:10.575664 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d4678e0b594abd1c0c7886b6c224f6103723c94978fe413e7341698ca917101\": container with ID starting with 6d4678e0b594abd1c0c7886b6c224f6103723c94978fe413e7341698ca917101 not found: ID does not exist" containerID="6d4678e0b594abd1c0c7886b6c224f6103723c94978fe413e7341698ca917101" Sep 16 08:00:10.575729 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:00:10.575688 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d4678e0b594abd1c0c7886b6c224f6103723c94978fe413e7341698ca917101"} err="failed to get container status \"6d4678e0b594abd1c0c7886b6c224f6103723c94978fe413e7341698ca917101\": rpc error: code = NotFound desc = could not find container \"6d4678e0b594abd1c0c7886b6c224f6103723c94978fe413e7341698ca917101\": container with ID starting with 6d4678e0b594abd1c0c7886b6c224f6103723c94978fe413e7341698ca917101 not found: ID does not exist" Sep 16 08:00:10.575729 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:00:10.575709 2557 scope.go:117] "RemoveContainer" containerID="e20f881ea6c25eb0343ea885c45eaf186823ab2c9c6d825153a8005f5f3f7323" Sep 16 08:00:10.575958 ip-10-0-141-221 kubenswrapper[2557]: E0916 08:00:10.575940 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e20f881ea6c25eb0343ea885c45eaf186823ab2c9c6d825153a8005f5f3f7323\": container with ID starting with e20f881ea6c25eb0343ea885c45eaf186823ab2c9c6d825153a8005f5f3f7323 not found: ID does not exist" containerID="e20f881ea6c25eb0343ea885c45eaf186823ab2c9c6d825153a8005f5f3f7323" Sep 16 08:00:10.575996 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:00:10.575963 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e20f881ea6c25eb0343ea885c45eaf186823ab2c9c6d825153a8005f5f3f7323"} err="failed to get container status \"e20f881ea6c25eb0343ea885c45eaf186823ab2c9c6d825153a8005f5f3f7323\": rpc error: code = NotFound desc = could not find container \"e20f881ea6c25eb0343ea885c45eaf186823ab2c9c6d825153a8005f5f3f7323\": container with ID starting with e20f881ea6c25eb0343ea885c45eaf186823ab2c9c6d825153a8005f5f3f7323 not found: ID does not exist" Sep 16 08:00:10.575996 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:00:10.575978 2557 scope.go:117] "RemoveContainer" containerID="1bc14d1868aa738a870a4c48faa51123d949920302b52728ee025fa0609712c7" Sep 16 08:00:10.576206 ip-10-0-141-221 kubenswrapper[2557]: E0916 08:00:10.576190 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bc14d1868aa738a870a4c48faa51123d949920302b52728ee025fa0609712c7\": container with ID starting with 1bc14d1868aa738a870a4c48faa51123d949920302b52728ee025fa0609712c7 not found: ID does not exist" containerID="1bc14d1868aa738a870a4c48faa51123d949920302b52728ee025fa0609712c7" Sep 16 08:00:10.576344 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:00:10.576209 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bc14d1868aa738a870a4c48faa51123d949920302b52728ee025fa0609712c7"} err="failed to get container status \"1bc14d1868aa738a870a4c48faa51123d949920302b52728ee025fa0609712c7\": rpc error: code = NotFound desc = could not find container \"1bc14d1868aa738a870a4c48faa51123d949920302b52728ee025fa0609712c7\": container with ID starting with 1bc14d1868aa738a870a4c48faa51123d949920302b52728ee025fa0609712c7 not found: ID does not exist" Sep 16 08:00:10.576449 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:00:10.576232 2557 scope.go:117] "RemoveContainer" containerID="82e59c96d3d54ac0e824f1db1cb3fe0b405f9439e875c0b31a578a05dc136424" Sep 16 08:00:10.576714 ip-10-0-141-221 kubenswrapper[2557]: E0916 08:00:10.576696 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82e59c96d3d54ac0e824f1db1cb3fe0b405f9439e875c0b31a578a05dc136424\": container with ID starting with 82e59c96d3d54ac0e824f1db1cb3fe0b405f9439e875c0b31a578a05dc136424 not found: ID does not exist" containerID="82e59c96d3d54ac0e824f1db1cb3fe0b405f9439e875c0b31a578a05dc136424" Sep 16 08:00:10.576778 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:00:10.576718 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82e59c96d3d54ac0e824f1db1cb3fe0b405f9439e875c0b31a578a05dc136424"} err="failed to get container status \"82e59c96d3d54ac0e824f1db1cb3fe0b405f9439e875c0b31a578a05dc136424\": rpc error: code = NotFound desc = could not find container \"82e59c96d3d54ac0e824f1db1cb3fe0b405f9439e875c0b31a578a05dc136424\": container with ID starting with 82e59c96d3d54ac0e824f1db1cb3fe0b405f9439e875c0b31a578a05dc136424 not found: ID does not exist" Sep 16 08:00:10.576778 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:00:10.576732 2557 scope.go:117] "RemoveContainer" containerID="fd98684ba596b5141d9e48951f7877ba071c363f8b683feea58828338eb34c60" Sep 16 08:00:10.576975 ip-10-0-141-221 kubenswrapper[2557]: E0916 08:00:10.576957 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd98684ba596b5141d9e48951f7877ba071c363f8b683feea58828338eb34c60\": container with ID starting with fd98684ba596b5141d9e48951f7877ba071c363f8b683feea58828338eb34c60 not found: ID does not exist" containerID="fd98684ba596b5141d9e48951f7877ba071c363f8b683feea58828338eb34c60" Sep 16 08:00:10.577020 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:00:10.576979 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd98684ba596b5141d9e48951f7877ba071c363f8b683feea58828338eb34c60"} err="failed to get container status \"fd98684ba596b5141d9e48951f7877ba071c363f8b683feea58828338eb34c60\": rpc error: code = NotFound desc = could not find container \"fd98684ba596b5141d9e48951f7877ba071c363f8b683feea58828338eb34c60\": container with ID starting with fd98684ba596b5141d9e48951f7877ba071c363f8b683feea58828338eb34c60 not found: ID does not exist" Sep 16 08:00:10.577020 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:00:10.576991 2557 scope.go:117] "RemoveContainer" containerID="27334b4ccd87f3c4e079e396d89ddabffec03f65a49ad610144da32a9eecea80" Sep 16 08:00:10.577229 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:00:10.577212 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27334b4ccd87f3c4e079e396d89ddabffec03f65a49ad610144da32a9eecea80"} err="failed to get container status \"27334b4ccd87f3c4e079e396d89ddabffec03f65a49ad610144da32a9eecea80\": rpc error: code = NotFound desc = could not find container \"27334b4ccd87f3c4e079e396d89ddabffec03f65a49ad610144da32a9eecea80\": container with ID starting with 27334b4ccd87f3c4e079e396d89ddabffec03f65a49ad610144da32a9eecea80 not found: ID does not exist" Sep 16 08:00:10.577295 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:00:10.577230 2557 scope.go:117] "RemoveContainer" containerID="6d4678e0b594abd1c0c7886b6c224f6103723c94978fe413e7341698ca917101" Sep 16 08:00:10.577496 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:00:10.577478 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d4678e0b594abd1c0c7886b6c224f6103723c94978fe413e7341698ca917101"} err="failed to get container status \"6d4678e0b594abd1c0c7886b6c224f6103723c94978fe413e7341698ca917101\": rpc error: code = NotFound desc = could not find container \"6d4678e0b594abd1c0c7886b6c224f6103723c94978fe413e7341698ca917101\": container with ID starting with 6d4678e0b594abd1c0c7886b6c224f6103723c94978fe413e7341698ca917101 not found: ID does not exist" Sep 16 08:00:10.577564 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:00:10.577498 2557 scope.go:117] "RemoveContainer" containerID="e20f881ea6c25eb0343ea885c45eaf186823ab2c9c6d825153a8005f5f3f7323" Sep 16 08:00:10.577746 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:00:10.577727 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e20f881ea6c25eb0343ea885c45eaf186823ab2c9c6d825153a8005f5f3f7323"} err="failed to get container status \"e20f881ea6c25eb0343ea885c45eaf186823ab2c9c6d825153a8005f5f3f7323\": rpc error: code = NotFound desc = could not find container \"e20f881ea6c25eb0343ea885c45eaf186823ab2c9c6d825153a8005f5f3f7323\": container with ID starting with e20f881ea6c25eb0343ea885c45eaf186823ab2c9c6d825153a8005f5f3f7323 not found: ID does not exist" Sep 16 08:00:10.577792 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:00:10.577748 2557 scope.go:117] "RemoveContainer" containerID="1bc14d1868aa738a870a4c48faa51123d949920302b52728ee025fa0609712c7" Sep 16 08:00:10.577983 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:00:10.577967 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bc14d1868aa738a870a4c48faa51123d949920302b52728ee025fa0609712c7"} err="failed to get container status \"1bc14d1868aa738a870a4c48faa51123d949920302b52728ee025fa0609712c7\": rpc error: code = NotFound desc = could not find container \"1bc14d1868aa738a870a4c48faa51123d949920302b52728ee025fa0609712c7\": container with ID starting with 1bc14d1868aa738a870a4c48faa51123d949920302b52728ee025fa0609712c7 not found: ID does not exist" Sep 16 08:00:10.578053 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:00:10.577985 2557 scope.go:117] "RemoveContainer" containerID="82e59c96d3d54ac0e824f1db1cb3fe0b405f9439e875c0b31a578a05dc136424" Sep 16 08:00:10.578209 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:00:10.578192 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82e59c96d3d54ac0e824f1db1cb3fe0b405f9439e875c0b31a578a05dc136424"} err="failed to get container status \"82e59c96d3d54ac0e824f1db1cb3fe0b405f9439e875c0b31a578a05dc136424\": rpc error: code = NotFound desc = could not find container \"82e59c96d3d54ac0e824f1db1cb3fe0b405f9439e875c0b31a578a05dc136424\": container with ID starting with 82e59c96d3d54ac0e824f1db1cb3fe0b405f9439e875c0b31a578a05dc136424 not found: ID does not exist" Sep 16 08:00:10.578279 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:00:10.578210 2557 scope.go:117] "RemoveContainer" containerID="fd98684ba596b5141d9e48951f7877ba071c363f8b683feea58828338eb34c60" Sep 16 08:00:10.578474 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:00:10.578457 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd98684ba596b5141d9e48951f7877ba071c363f8b683feea58828338eb34c60"} err="failed to get container status \"fd98684ba596b5141d9e48951f7877ba071c363f8b683feea58828338eb34c60\": rpc error: code = NotFound desc = could not find container \"fd98684ba596b5141d9e48951f7877ba071c363f8b683feea58828338eb34c60\": container with ID starting with fd98684ba596b5141d9e48951f7877ba071c363f8b683feea58828338eb34c60 not found: ID does not exist" Sep 16 08:00:10.594965 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:00:10.594946 2557 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18-feast-data\") on node \"ip-10-0-141-221.ec2.internal\" DevicePath \"\"" Sep 16 08:00:10.595037 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:00:10.594987 2557 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18-online-tls\") on node \"ip-10-0-141-221.ec2.internal\" DevicePath \"\"" Sep 16 08:00:10.595037 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:00:10.594997 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkfr4\" (UniqueName: \"kubernetes.io/projected/6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18-kube-api-access-wkfr4\") on node \"ip-10-0-141-221.ec2.internal\" DevicePath \"\"" Sep 16 08:00:10.595037 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:00:10.595006 2557 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18-offline-tls\") on node \"ip-10-0-141-221.ec2.internal\" DevicePath \"\"" Sep 16 08:00:10.595037 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:00:10.595015 2557 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18-registry-tls\") on node \"ip-10-0-141-221.ec2.internal\" DevicePath \"\"" Sep 16 08:00:10.595037 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:00:10.595023 2557 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18-ui-tls\") on node \"ip-10-0-141-221.ec2.internal\" DevicePath \"\"" Sep 16 08:00:11.523815 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:00:11.523786 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18" path="/var/lib/kubelet/pods/6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18/volumes" Sep 16 08:01:04.751957 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:01:04.751920 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-glszw"] Sep 16 08:01:04.752402 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:01:04.752200 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18" containerName="ui" Sep 16 08:01:04.752402 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:01:04.752210 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18" containerName="ui" Sep 16 08:01:04.752402 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:01:04.752220 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18" containerName="offline" Sep 16 08:01:04.752402 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:01:04.752227 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18" containerName="offline" Sep 16 08:01:04.752402 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:01:04.752255 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18" containerName="feast-apply" Sep 16 08:01:04.752402 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:01:04.752261 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18" containerName="feast-apply" Sep 16 08:01:04.752402 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:01:04.752268 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18" containerName="feast-init" Sep 16 08:01:04.752402 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:01:04.752273 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18" containerName="feast-init" Sep 16 08:01:04.752402 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:01:04.752286 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18" containerName="online" Sep 16 08:01:04.752402 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:01:04.752291 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18" containerName="online" Sep 16 08:01:04.752402 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:01:04.752302 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18" containerName="registry" Sep 16 08:01:04.752402 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:01:04.752309 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18" containerName="registry" Sep 16 08:01:04.752402 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:01:04.752353 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18" containerName="online" Sep 16 08:01:04.752402 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:01:04.752359 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18" containerName="offline" Sep 16 08:01:04.752402 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:01:04.752366 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18" containerName="registry" Sep 16 08:01:04.752402 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:01:04.752375 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c1047a7-8f58-4fce-b7ce-fcdc6ff34b18" containerName="ui" Sep 16 08:01:04.755464 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:01:04.755446 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-glszw" Sep 16 08:01:04.757802 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:01:04.757780 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-ui-tls\"" Sep 16 08:01:04.757901 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:01:04.757820 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-client-ca\"" Sep 16 08:01:04.757901 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:01:04.757882 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-dockercfg-ttgql\"" Sep 16 08:01:04.757901 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:01:04.757883 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"openshift-service-ca.crt\"" Sep 16 08:01:04.758058 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:01:04.757910 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"kube-root-ca.crt\"" Sep 16 08:01:04.758169 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:01:04.758150 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-online-tls\"" Sep 16 08:01:04.758325 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:01:04.758153 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-offline-tls\"" Sep 16 08:01:04.766925 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:01:04.766905 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-glszw"] Sep 16 08:01:04.879229 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:01:04.879188 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/0c4c4aa4-4483-41ad-b383-a03cf353e55b-ui-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-glszw\" (UID: \"0c4c4aa4-4483-41ad-b383-a03cf353e55b\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-glszw" Sep 16 08:01:04.879229 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:01:04.879253 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/0c4c4aa4-4483-41ad-b383-a03cf353e55b-feast-data\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-glszw\" (UID: \"0c4c4aa4-4483-41ad-b383-a03cf353e55b\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-glszw" Sep 16 08:01:04.879490 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:01:04.879290 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/0c4c4aa4-4483-41ad-b383-a03cf353e55b-offline-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-glszw\" (UID: \"0c4c4aa4-4483-41ad-b383-a03cf353e55b\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-glszw" Sep 16 08:01:04.879490 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:01:04.879354 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/0c4c4aa4-4483-41ad-b383-a03cf353e55b-online-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-glszw\" (UID: \"0c4c4aa4-4483-41ad-b383-a03cf353e55b\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-glszw" Sep 16 08:01:04.879490 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:01:04.879394 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slqmg\" (UniqueName: \"kubernetes.io/projected/0c4c4aa4-4483-41ad-b383-a03cf353e55b-kube-api-access-slqmg\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-glszw\" (UID: \"0c4c4aa4-4483-41ad-b383-a03cf353e55b\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-glszw" Sep 16 08:01:04.879490 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:01:04.879452 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/0c4c4aa4-4483-41ad-b383-a03cf353e55b-registry-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-glszw\" (UID: \"0c4c4aa4-4483-41ad-b383-a03cf353e55b\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-glszw" Sep 16 08:01:04.979998 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:01:04.979953 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/0c4c4aa4-4483-41ad-b383-a03cf353e55b-offline-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-glszw\" (UID: \"0c4c4aa4-4483-41ad-b383-a03cf353e55b\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-glszw" Sep 16 08:01:04.979998 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:01:04.980000 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/0c4c4aa4-4483-41ad-b383-a03cf353e55b-online-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-glszw\" (UID: \"0c4c4aa4-4483-41ad-b383-a03cf353e55b\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-glszw" Sep 16 08:01:04.980259 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:01:04.980030 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slqmg\" (UniqueName: \"kubernetes.io/projected/0c4c4aa4-4483-41ad-b383-a03cf353e55b-kube-api-access-slqmg\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-glszw\" (UID: \"0c4c4aa4-4483-41ad-b383-a03cf353e55b\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-glszw" Sep 16 08:01:04.980259 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:01:04.980058 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/0c4c4aa4-4483-41ad-b383-a03cf353e55b-registry-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-glszw\" (UID: \"0c4c4aa4-4483-41ad-b383-a03cf353e55b\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-glszw" Sep 16 08:01:04.980259 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:01:04.980092 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/0c4c4aa4-4483-41ad-b383-a03cf353e55b-ui-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-glszw\" (UID: \"0c4c4aa4-4483-41ad-b383-a03cf353e55b\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-glszw" Sep 16 08:01:04.980259 ip-10-0-141-221 kubenswrapper[2557]: E0916 08:01:04.980095 2557 secret.go:189] Couldn't get secret test-ns-remote-registry/feast-simple-feast-remote-setup-offline-tls: secret "feast-simple-feast-remote-setup-offline-tls" not found Sep 16 08:01:04.980259 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:01:04.980114 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/0c4c4aa4-4483-41ad-b383-a03cf353e55b-feast-data\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-glszw\" (UID: \"0c4c4aa4-4483-41ad-b383-a03cf353e55b\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-glszw" Sep 16 08:01:04.980259 ip-10-0-141-221 kubenswrapper[2557]: E0916 08:01:04.980175 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c4c4aa4-4483-41ad-b383-a03cf353e55b-offline-tls podName:0c4c4aa4-4483-41ad-b383-a03cf353e55b nodeName:}" failed. No retries permitted until 2026-09-16 08:01:05.480155471 +0000 UTC m=+1998.507056479 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/0c4c4aa4-4483-41ad-b383-a03cf353e55b-offline-tls") pod "feast-simple-feast-remote-setup-c5d49d8c6-glszw" (UID: "0c4c4aa4-4483-41ad-b383-a03cf353e55b") : secret "feast-simple-feast-remote-setup-offline-tls" not found Sep 16 08:01:04.980579 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:01:04.980562 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/0c4c4aa4-4483-41ad-b383-a03cf353e55b-feast-data\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-glszw\" (UID: \"0c4c4aa4-4483-41ad-b383-a03cf353e55b\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-glszw" Sep 16 08:01:04.980918 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:01:04.980900 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/0c4c4aa4-4483-41ad-b383-a03cf353e55b-registry-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-glszw\" (UID: \"0c4c4aa4-4483-41ad-b383-a03cf353e55b\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-glszw" Sep 16 08:01:04.982540 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:01:04.982513 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/0c4c4aa4-4483-41ad-b383-a03cf353e55b-online-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-glszw\" (UID: \"0c4c4aa4-4483-41ad-b383-a03cf353e55b\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-glszw" Sep 16 08:01:04.982641 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:01:04.982567 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/0c4c4aa4-4483-41ad-b383-a03cf353e55b-ui-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-glszw\" (UID: \"0c4c4aa4-4483-41ad-b383-a03cf353e55b\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-glszw" Sep 16 08:01:04.988375 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:01:04.988352 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-slqmg\" (UniqueName: \"kubernetes.io/projected/0c4c4aa4-4483-41ad-b383-a03cf353e55b-kube-api-access-slqmg\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-glszw\" (UID: \"0c4c4aa4-4483-41ad-b383-a03cf353e55b\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-glszw" Sep 16 08:01:05.484598 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:01:05.484564 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/0c4c4aa4-4483-41ad-b383-a03cf353e55b-offline-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-glszw\" (UID: \"0c4c4aa4-4483-41ad-b383-a03cf353e55b\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-glszw" Sep 16 08:01:05.486847 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:01:05.486825 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/0c4c4aa4-4483-41ad-b383-a03cf353e55b-offline-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-glszw\" (UID: \"0c4c4aa4-4483-41ad-b383-a03cf353e55b\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-glszw" Sep 16 08:01:05.664451 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:01:05.664418 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-glszw" Sep 16 08:01:05.937647 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:01:05.937622 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-glszw"] Sep 16 08:01:05.939353 ip-10-0-141-221 kubenswrapper[2557]: W0916 08:01:05.939327 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0c4c4aa4_4483_41ad_b383_a03cf353e55b.slice/crio-ebc56ab6ab4cd6f3efe27597fbcad2c6c4c961153b4ad2e15d21d734b5c0637b WatchSource:0}: Error finding container ebc56ab6ab4cd6f3efe27597fbcad2c6c4c961153b4ad2e15d21d734b5c0637b: Status 404 returned error can't find the container with id ebc56ab6ab4cd6f3efe27597fbcad2c6c4c961153b4ad2e15d21d734b5c0637b Sep 16 08:01:06.734654 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:01:06.734617 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-glszw" event={"ID":"0c4c4aa4-4483-41ad-b383-a03cf353e55b","Type":"ContainerStarted","Data":"0f8b8411a38d09b2bca3800cfc07d6261fbb55ce17e069b1e7618fc7d4581e97"} Sep 16 08:01:06.734794 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:01:06.734670 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-glszw" event={"ID":"0c4c4aa4-4483-41ad-b383-a03cf353e55b","Type":"ContainerStarted","Data":"ebc56ab6ab4cd6f3efe27597fbcad2c6c4c961153b4ad2e15d21d734b5c0637b"} Sep 16 08:01:11.756058 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:01:11.756026 2557 generic.go:356] "Generic (PLEG): container finished" podID="0c4c4aa4-4483-41ad-b383-a03cf353e55b" containerID="0f8b8411a38d09b2bca3800cfc07d6261fbb55ce17e069b1e7618fc7d4581e97" exitCode=0 Sep 16 08:01:11.756448 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:01:11.756070 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-glszw" event={"ID":"0c4c4aa4-4483-41ad-b383-a03cf353e55b","Type":"ContainerDied","Data":"0f8b8411a38d09b2bca3800cfc07d6261fbb55ce17e069b1e7618fc7d4581e97"} Sep 16 08:01:12.761378 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:01:12.761332 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-glszw" event={"ID":"0c4c4aa4-4483-41ad-b383-a03cf353e55b","Type":"ContainerStarted","Data":"8c541ee377f603f45b70a7be20e662d2cb9a8e35f41b6c7657ecf3061435a337"} Sep 16 08:01:32.838045 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:01:32.838010 2557 generic.go:356] "Generic (PLEG): container finished" podID="0c4c4aa4-4483-41ad-b383-a03cf353e55b" containerID="8c541ee377f603f45b70a7be20e662d2cb9a8e35f41b6c7657ecf3061435a337" exitCode=0 Sep 16 08:01:32.838455 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:01:32.838085 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-glszw" event={"ID":"0c4c4aa4-4483-41ad-b383-a03cf353e55b","Type":"ContainerDied","Data":"8c541ee377f603f45b70a7be20e662d2cb9a8e35f41b6c7657ecf3061435a337"} Sep 16 08:01:33.847630 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:01:33.847592 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-glszw" event={"ID":"0c4c4aa4-4483-41ad-b383-a03cf353e55b","Type":"ContainerStarted","Data":"9a7a694cce4d348d324488149757d24d92553f517eff9303b8d39570d0ebeb9e"} Sep 16 08:01:33.848105 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:01:33.847640 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-glszw" event={"ID":"0c4c4aa4-4483-41ad-b383-a03cf353e55b","Type":"ContainerStarted","Data":"66547bce9bc1543c4ef25c9618a32afc2c923f9c1d17b1f40e7aa99e95553919"} Sep 16 08:01:33.848105 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:01:33.847657 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-glszw" event={"ID":"0c4c4aa4-4483-41ad-b383-a03cf353e55b","Type":"ContainerStarted","Data":"bd3c728a44b50f2bb9e1a2feda3899f03566d70776f59040654b7669c64dada1"} Sep 16 08:01:33.869444 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:01:33.869389 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-glszw" podStartSLOduration=29.869372734 podStartE2EDuration="29.869372734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 08:01:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 08:01:33.868618656 +0000 UTC m=+2026.895519671" watchObservedRunningTime="2026-09-16 08:01:33.869372734 +0000 UTC m=+2026.896273746" Sep 16 08:01:35.665354 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:01:35.665300 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-glszw" Sep 16 08:01:35.665825 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:01:35.665381 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-glszw" Sep 16 08:01:35.665825 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:01:35.665401 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-glszw" Sep 16 08:01:35.665825 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:01:35.665417 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-glszw" Sep 16 08:01:35.665825 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:01:35.665433 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-glszw" Sep 16 08:01:35.665825 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:01:35.665448 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-glszw" Sep 16 08:01:35.667399 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:01:35.667360 2557 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-glszw" podUID="0c4c4aa4-4483-41ad-b383-a03cf353e55b" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.26:8816: connect: connection refused" Sep 16 08:01:35.667544 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:01:35.667362 2557 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-glszw" podUID="0c4c4aa4-4483-41ad-b383-a03cf353e55b" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.26:8443: connect: connection refused" Sep 16 08:01:35.667544 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:01:35.667441 2557 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-glszw" podUID="0c4c4aa4-4483-41ad-b383-a03cf353e55b" containerName="online" probeResult="failure" output="Get \"https://10.133.0.26:6567/health\": dial tcp 10.133.0.26:6567: connect: connection refused" Sep 16 08:01:38.665090 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:01:38.665042 2557 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-glszw" podUID="0c4c4aa4-4483-41ad-b383-a03cf353e55b" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.26:8816: connect: connection refused" Sep 16 08:01:38.665090 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:01:38.665077 2557 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-glszw" podUID="0c4c4aa4-4483-41ad-b383-a03cf353e55b" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.26:8443: connect: connection refused" Sep 16 08:01:38.665677 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:01:38.665114 2557 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-glszw" podUID="0c4c4aa4-4483-41ad-b383-a03cf353e55b" containerName="online" probeResult="failure" output="Get \"https://10.133.0.26:6567/health\": dial tcp 10.133.0.26:6567: connect: connection refused" Sep 16 08:01:41.665466 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:01:41.665426 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-glszw" Sep 16 08:01:41.665930 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:01:41.665629 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-glszw" Sep 16 08:01:41.666216 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:01:41.666197 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-glszw" Sep 16 08:01:41.666355 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:01:41.666338 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-glszw" Sep 16 08:01:42.668349 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:01:42.668312 2557 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-glszw" podUID="0c4c4aa4-4483-41ad-b383-a03cf353e55b" containerName="online" probeResult="failure" output="Get \"https://10.133.0.26:6567/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Sep 16 08:01:45.332555 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:01:45.332497 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-glszw" Sep 16 08:01:46.077890 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:01:46.077859 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-glszw" Sep 16 08:01:46.670389 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:01:46.670348 2557 prober.go:120] "Probe failed" probeType="Liveness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-glszw" podUID="0c4c4aa4-4483-41ad-b383-a03cf353e55b" containerName="online" probeResult="failure" output="Get \"https://10.133.0.26:6567/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Sep 16 08:02:46.845970 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:02:46.845933 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-glszw"] Sep 16 08:02:46.846820 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:02:46.846161 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-glszw" podUID="0c4c4aa4-4483-41ad-b383-a03cf353e55b" containerName="online" containerID="cri-o://bd3c728a44b50f2bb9e1a2feda3899f03566d70776f59040654b7669c64dada1" gracePeriod=30 Sep 16 08:02:46.846820 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:02:46.846272 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-glszw" podUID="0c4c4aa4-4483-41ad-b383-a03cf353e55b" containerName="offline" containerID="cri-o://66547bce9bc1543c4ef25c9618a32afc2c923f9c1d17b1f40e7aa99e95553919" gracePeriod=30 Sep 16 08:02:46.846820 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:02:46.846279 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-glszw" podUID="0c4c4aa4-4483-41ad-b383-a03cf353e55b" containerName="ui" containerID="cri-o://9a7a694cce4d348d324488149757d24d92553f517eff9303b8d39570d0ebeb9e" gracePeriod=30 Sep 16 08:02:47.004585 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:02:47.004549 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-c4bxp"] Sep 16 08:02:47.004898 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:02:47.004859 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-c4bxp" podUID="23ed5836-df96-41f7-bd69-129fd932f13c" containerName="registry" containerID="cri-o://b2e975bd990dd19b4cb1c0fdaaef0519eea2bec21666767dfa79e43ae8df250a" gracePeriod=30 Sep 16 08:02:47.005140 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:02:47.004906 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-c4bxp" podUID="23ed5836-df96-41f7-bd69-129fd932f13c" containerName="offline" containerID="cri-o://013a1b2fe4d94dd257ebfa55a3deefb39aaf6dd5ebdbe8574d63d00144e8170a" gracePeriod=30 Sep 16 08:02:47.005289 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:02:47.004917 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-c4bxp" podUID="23ed5836-df96-41f7-bd69-129fd932f13c" containerName="online" containerID="cri-o://8dd4e9766e9c334a298a71f95c3acbb0820a63e4f615e62bc166f3234e4aad75" gracePeriod=30 Sep 16 08:02:47.005289 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:02:47.004904 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-c4bxp" podUID="23ed5836-df96-41f7-bd69-129fd932f13c" containerName="ui" containerID="cri-o://7a02e32b0b1fe24f47bbb850df8fffbe6aafa15a7fc7303851ff4dfbd5c679e4" gracePeriod=30 Sep 16 08:02:48.145836 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:02:48.145802 2557 generic.go:356] "Generic (PLEG): container finished" podID="23ed5836-df96-41f7-bd69-129fd932f13c" containerID="7a02e32b0b1fe24f47bbb850df8fffbe6aafa15a7fc7303851ff4dfbd5c679e4" exitCode=0 Sep 16 08:02:48.145836 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:02:48.145829 2557 generic.go:356] "Generic (PLEG): container finished" podID="23ed5836-df96-41f7-bd69-129fd932f13c" containerID="8dd4e9766e9c334a298a71f95c3acbb0820a63e4f615e62bc166f3234e4aad75" exitCode=0 Sep 16 08:02:48.146272 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:02:48.145873 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-c4bxp" event={"ID":"23ed5836-df96-41f7-bd69-129fd932f13c","Type":"ContainerDied","Data":"7a02e32b0b1fe24f47bbb850df8fffbe6aafa15a7fc7303851ff4dfbd5c679e4"} Sep 16 08:02:48.146272 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:02:48.145913 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-c4bxp" event={"ID":"23ed5836-df96-41f7-bd69-129fd932f13c","Type":"ContainerDied","Data":"8dd4e9766e9c334a298a71f95c3acbb0820a63e4f615e62bc166f3234e4aad75"} Sep 16 08:02:48.148466 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:02:48.148447 2557 generic.go:356] "Generic (PLEG): container finished" podID="0c4c4aa4-4483-41ad-b383-a03cf353e55b" containerID="9a7a694cce4d348d324488149757d24d92553f517eff9303b8d39570d0ebeb9e" exitCode=0 Sep 16 08:02:48.148466 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:02:48.148465 2557 generic.go:356] "Generic (PLEG): container finished" podID="0c4c4aa4-4483-41ad-b383-a03cf353e55b" containerID="bd3c728a44b50f2bb9e1a2feda3899f03566d70776f59040654b7669c64dada1" exitCode=0 Sep 16 08:02:48.148614 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:02:48.148503 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-glszw" event={"ID":"0c4c4aa4-4483-41ad-b383-a03cf353e55b","Type":"ContainerDied","Data":"9a7a694cce4d348d324488149757d24d92553f517eff9303b8d39570d0ebeb9e"} Sep 16 08:02:48.148614 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:02:48.148529 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-glszw" event={"ID":"0c4c4aa4-4483-41ad-b383-a03cf353e55b","Type":"ContainerDied","Data":"bd3c728a44b50f2bb9e1a2feda3899f03566d70776f59040654b7669c64dada1"} Sep 16 08:02:51.666596 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:02:51.666553 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-glszw" podUID="0c4c4aa4-4483-41ad-b383-a03cf353e55b" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.26:8443: connect: connection refused" Sep 16 08:02:52.496131 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:02:52.496069 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-c4bxp" podUID="23ed5836-df96-41f7-bd69-129fd932f13c" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.25:8443: connect: connection refused" Sep 16 08:02:52.496344 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:02:52.496071 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-c4bxp" podUID="23ed5836-df96-41f7-bd69-129fd932f13c" containerName="online" probeResult="failure" output="Get \"https://10.133.0.25:6567/health\": dial tcp 10.133.0.25:6567: connect: connection refused" Sep 16 08:02:55.333725 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:02:55.333689 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-glszw" podUID="0c4c4aa4-4483-41ad-b383-a03cf353e55b" containerName="online" probeResult="failure" output="Get \"https://10.133.0.26:6567/health\": dial tcp 10.133.0.26:6567: connect: connection refused" Sep 16 08:03:01.666697 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:03:01.666658 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-glszw" podUID="0c4c4aa4-4483-41ad-b383-a03cf353e55b" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.26:8443: connect: connection refused" Sep 16 08:03:02.495517 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:03:02.495475 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-c4bxp" podUID="23ed5836-df96-41f7-bd69-129fd932f13c" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.25:8443: connect: connection refused" Sep 16 08:03:02.495695 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:03:02.495517 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-c4bxp" podUID="23ed5836-df96-41f7-bd69-129fd932f13c" containerName="online" probeResult="failure" output="Get \"https://10.133.0.25:6567/health\": dial tcp 10.133.0.25:6567: connect: connection refused" Sep 16 08:03:05.333753 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:03:05.333711 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-glszw" podUID="0c4c4aa4-4483-41ad-b383-a03cf353e55b" containerName="online" probeResult="failure" output="Get \"https://10.133.0.26:6567/health\": dial tcp 10.133.0.26:6567: connect: connection refused" Sep 16 08:03:11.666520 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:03:11.666476 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-glszw" podUID="0c4c4aa4-4483-41ad-b383-a03cf353e55b" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.26:8443: connect: connection refused" Sep 16 08:03:11.666954 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:03:11.666609 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-glszw" Sep 16 08:03:12.495421 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:03:12.495383 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-c4bxp" podUID="23ed5836-df96-41f7-bd69-129fd932f13c" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.25:8443: connect: connection refused" Sep 16 08:03:12.495607 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:03:12.495440 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-c4bxp" podUID="23ed5836-df96-41f7-bd69-129fd932f13c" containerName="online" probeResult="failure" output="Get \"https://10.133.0.25:6567/health\": dial tcp 10.133.0.25:6567: connect: connection refused" Sep 16 08:03:12.495607 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:03:12.495541 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-c4bxp" Sep 16 08:03:12.495607 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:03:12.495606 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-c4bxp" Sep 16 08:03:15.333430 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:03:15.333384 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-glszw" podUID="0c4c4aa4-4483-41ad-b383-a03cf353e55b" containerName="online" probeResult="failure" output="Get \"https://10.133.0.26:6567/health\": dial tcp 10.133.0.26:6567: connect: connection refused" Sep 16 08:03:15.333853 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:03:15.333527 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-glszw" Sep 16 08:03:17.264097 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:03:17.264073 2557 generic.go:356] "Generic (PLEG): container finished" podID="23ed5836-df96-41f7-bd69-129fd932f13c" containerID="013a1b2fe4d94dd257ebfa55a3deefb39aaf6dd5ebdbe8574d63d00144e8170a" exitCode=137 Sep 16 08:03:17.264097 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:03:17.264097 2557 generic.go:356] "Generic (PLEG): container finished" podID="23ed5836-df96-41f7-bd69-129fd932f13c" containerID="b2e975bd990dd19b4cb1c0fdaaef0519eea2bec21666767dfa79e43ae8df250a" exitCode=137 Sep 16 08:03:17.264460 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:03:17.264144 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-c4bxp" event={"ID":"23ed5836-df96-41f7-bd69-129fd932f13c","Type":"ContainerDied","Data":"013a1b2fe4d94dd257ebfa55a3deefb39aaf6dd5ebdbe8574d63d00144e8170a"} Sep 16 08:03:17.264460 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:03:17.264185 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-c4bxp" event={"ID":"23ed5836-df96-41f7-bd69-129fd932f13c","Type":"ContainerDied","Data":"b2e975bd990dd19b4cb1c0fdaaef0519eea2bec21666767dfa79e43ae8df250a"} Sep 16 08:03:17.266897 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:03:17.266877 2557 generic.go:356] "Generic (PLEG): container finished" podID="0c4c4aa4-4483-41ad-b383-a03cf353e55b" containerID="66547bce9bc1543c4ef25c9618a32afc2c923f9c1d17b1f40e7aa99e95553919" exitCode=137 Sep 16 08:03:17.266988 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:03:17.266919 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-glszw" event={"ID":"0c4c4aa4-4483-41ad-b383-a03cf353e55b","Type":"ContainerDied","Data":"66547bce9bc1543c4ef25c9618a32afc2c923f9c1d17b1f40e7aa99e95553919"} Sep 16 08:03:17.477941 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:03:17.477919 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-glszw" Sep 16 08:03:17.570962 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:03:17.570936 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/0c4c4aa4-4483-41ad-b383-a03cf353e55b-ui-tls\") pod \"0c4c4aa4-4483-41ad-b383-a03cf353e55b\" (UID: \"0c4c4aa4-4483-41ad-b383-a03cf353e55b\") " Sep 16 08:03:17.571123 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:03:17.570971 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/0c4c4aa4-4483-41ad-b383-a03cf353e55b-registry-tls\") pod \"0c4c4aa4-4483-41ad-b383-a03cf353e55b\" (UID: \"0c4c4aa4-4483-41ad-b383-a03cf353e55b\") " Sep 16 08:03:17.571123 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:03:17.571006 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/0c4c4aa4-4483-41ad-b383-a03cf353e55b-online-tls\") pod \"0c4c4aa4-4483-41ad-b383-a03cf353e55b\" (UID: \"0c4c4aa4-4483-41ad-b383-a03cf353e55b\") " Sep 16 08:03:17.571123 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:03:17.571037 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-slqmg\" (UniqueName: \"kubernetes.io/projected/0c4c4aa4-4483-41ad-b383-a03cf353e55b-kube-api-access-slqmg\") pod \"0c4c4aa4-4483-41ad-b383-a03cf353e55b\" (UID: \"0c4c4aa4-4483-41ad-b383-a03cf353e55b\") " Sep 16 08:03:17.571123 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:03:17.571095 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/0c4c4aa4-4483-41ad-b383-a03cf353e55b-offline-tls\") pod \"0c4c4aa4-4483-41ad-b383-a03cf353e55b\" (UID: \"0c4c4aa4-4483-41ad-b383-a03cf353e55b\") " Sep 16 08:03:17.571352 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:03:17.571139 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/0c4c4aa4-4483-41ad-b383-a03cf353e55b-feast-data\") pod \"0c4c4aa4-4483-41ad-b383-a03cf353e55b\" (UID: \"0c4c4aa4-4483-41ad-b383-a03cf353e55b\") " Sep 16 08:03:17.571462 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:03:17.571391 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c4c4aa4-4483-41ad-b383-a03cf353e55b-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "0c4c4aa4-4483-41ad-b383-a03cf353e55b" (UID: "0c4c4aa4-4483-41ad-b383-a03cf353e55b"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 08:03:17.571770 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:03:17.571715 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c4c4aa4-4483-41ad-b383-a03cf353e55b-feast-data" (OuterVolumeSpecName: "feast-data") pod "0c4c4aa4-4483-41ad-b383-a03cf353e55b" (UID: "0c4c4aa4-4483-41ad-b383-a03cf353e55b"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 08:03:17.573216 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:03:17.573189 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c4c4aa4-4483-41ad-b383-a03cf353e55b-online-tls" (OuterVolumeSpecName: "online-tls") pod "0c4c4aa4-4483-41ad-b383-a03cf353e55b" (UID: "0c4c4aa4-4483-41ad-b383-a03cf353e55b"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 08:03:17.573216 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:03:17.573202 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c4c4aa4-4483-41ad-b383-a03cf353e55b-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "0c4c4aa4-4483-41ad-b383-a03cf353e55b" (UID: "0c4c4aa4-4483-41ad-b383-a03cf353e55b"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 08:03:17.573503 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:03:17.573481 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c4c4aa4-4483-41ad-b383-a03cf353e55b-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "0c4c4aa4-4483-41ad-b383-a03cf353e55b" (UID: "0c4c4aa4-4483-41ad-b383-a03cf353e55b"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 08:03:17.573594 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:03:17.573506 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c4c4aa4-4483-41ad-b383-a03cf353e55b-kube-api-access-slqmg" (OuterVolumeSpecName: "kube-api-access-slqmg") pod "0c4c4aa4-4483-41ad-b383-a03cf353e55b" (UID: "0c4c4aa4-4483-41ad-b383-a03cf353e55b"). InnerVolumeSpecName "kube-api-access-slqmg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 08:03:17.641729 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:03:17.641709 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-c4bxp" Sep 16 08:03:17.671949 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:03:17.671927 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/23ed5836-df96-41f7-bd69-129fd932f13c-online-tls\") pod \"23ed5836-df96-41f7-bd69-129fd932f13c\" (UID: \"23ed5836-df96-41f7-bd69-129fd932f13c\") " Sep 16 08:03:17.672051 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:03:17.671955 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/23ed5836-df96-41f7-bd69-129fd932f13c-feast-data\") pod \"23ed5836-df96-41f7-bd69-129fd932f13c\" (UID: \"23ed5836-df96-41f7-bd69-129fd932f13c\") " Sep 16 08:03:17.672051 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:03:17.671975 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/23ed5836-df96-41f7-bd69-129fd932f13c-offline-tls\") pod \"23ed5836-df96-41f7-bd69-129fd932f13c\" (UID: \"23ed5836-df96-41f7-bd69-129fd932f13c\") " Sep 16 08:03:17.672051 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:03:17.671994 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kb4nh\" (UniqueName: \"kubernetes.io/projected/23ed5836-df96-41f7-bd69-129fd932f13c-kube-api-access-kb4nh\") pod \"23ed5836-df96-41f7-bd69-129fd932f13c\" (UID: \"23ed5836-df96-41f7-bd69-129fd932f13c\") " Sep 16 08:03:17.672051 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:03:17.672012 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/23ed5836-df96-41f7-bd69-129fd932f13c-registry-tls\") pod \"23ed5836-df96-41f7-bd69-129fd932f13c\" (UID: \"23ed5836-df96-41f7-bd69-129fd932f13c\") " Sep 16 08:03:17.672171 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:03:17.672075 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/23ed5836-df96-41f7-bd69-129fd932f13c-ui-tls\") pod \"23ed5836-df96-41f7-bd69-129fd932f13c\" (UID: \"23ed5836-df96-41f7-bd69-129fd932f13c\") " Sep 16 08:03:17.672210 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:03:17.672182 2557 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/0c4c4aa4-4483-41ad-b383-a03cf353e55b-feast-data\") on node \"ip-10-0-141-221.ec2.internal\" DevicePath \"\"" Sep 16 08:03:17.672210 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:03:17.672191 2557 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/0c4c4aa4-4483-41ad-b383-a03cf353e55b-ui-tls\") on node \"ip-10-0-141-221.ec2.internal\" DevicePath \"\"" Sep 16 08:03:17.672210 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:03:17.672199 2557 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/0c4c4aa4-4483-41ad-b383-a03cf353e55b-registry-tls\") on node \"ip-10-0-141-221.ec2.internal\" DevicePath \"\"" Sep 16 08:03:17.672210 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:03:17.672207 2557 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/0c4c4aa4-4483-41ad-b383-a03cf353e55b-online-tls\") on node \"ip-10-0-141-221.ec2.internal\" DevicePath \"\"" Sep 16 08:03:17.672418 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:03:17.672215 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slqmg\" (UniqueName: \"kubernetes.io/projected/0c4c4aa4-4483-41ad-b383-a03cf353e55b-kube-api-access-slqmg\") on node \"ip-10-0-141-221.ec2.internal\" DevicePath \"\"" Sep 16 08:03:17.672418 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:03:17.672223 2557 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/0c4c4aa4-4483-41ad-b383-a03cf353e55b-offline-tls\") on node \"ip-10-0-141-221.ec2.internal\" DevicePath \"\"" Sep 16 08:03:17.672880 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:03:17.672818 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23ed5836-df96-41f7-bd69-129fd932f13c-feast-data" (OuterVolumeSpecName: "feast-data") pod "23ed5836-df96-41f7-bd69-129fd932f13c" (UID: "23ed5836-df96-41f7-bd69-129fd932f13c"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 08:03:17.674424 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:03:17.674366 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23ed5836-df96-41f7-bd69-129fd932f13c-online-tls" (OuterVolumeSpecName: "online-tls") pod "23ed5836-df96-41f7-bd69-129fd932f13c" (UID: "23ed5836-df96-41f7-bd69-129fd932f13c"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 08:03:17.674424 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:03:17.674383 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23ed5836-df96-41f7-bd69-129fd932f13c-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "23ed5836-df96-41f7-bd69-129fd932f13c" (UID: "23ed5836-df96-41f7-bd69-129fd932f13c"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 08:03:17.674606 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:03:17.674583 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23ed5836-df96-41f7-bd69-129fd932f13c-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "23ed5836-df96-41f7-bd69-129fd932f13c" (UID: "23ed5836-df96-41f7-bd69-129fd932f13c"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 08:03:17.674656 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:03:17.674640 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23ed5836-df96-41f7-bd69-129fd932f13c-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "23ed5836-df96-41f7-bd69-129fd932f13c" (UID: "23ed5836-df96-41f7-bd69-129fd932f13c"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 08:03:17.674740 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:03:17.674725 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23ed5836-df96-41f7-bd69-129fd932f13c-kube-api-access-kb4nh" (OuterVolumeSpecName: "kube-api-access-kb4nh") pod "23ed5836-df96-41f7-bd69-129fd932f13c" (UID: "23ed5836-df96-41f7-bd69-129fd932f13c"). InnerVolumeSpecName "kube-api-access-kb4nh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 08:03:17.772908 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:03:17.772885 2557 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/23ed5836-df96-41f7-bd69-129fd932f13c-ui-tls\") on node \"ip-10-0-141-221.ec2.internal\" DevicePath \"\"" Sep 16 08:03:17.772908 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:03:17.772904 2557 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/23ed5836-df96-41f7-bd69-129fd932f13c-online-tls\") on node \"ip-10-0-141-221.ec2.internal\" DevicePath \"\"" Sep 16 08:03:17.773086 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:03:17.772914 2557 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/23ed5836-df96-41f7-bd69-129fd932f13c-feast-data\") on node \"ip-10-0-141-221.ec2.internal\" DevicePath \"\"" Sep 16 08:03:17.773086 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:03:17.772922 2557 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/23ed5836-df96-41f7-bd69-129fd932f13c-offline-tls\") on node \"ip-10-0-141-221.ec2.internal\" DevicePath \"\"" Sep 16 08:03:17.773086 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:03:17.772931 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kb4nh\" (UniqueName: \"kubernetes.io/projected/23ed5836-df96-41f7-bd69-129fd932f13c-kube-api-access-kb4nh\") on node \"ip-10-0-141-221.ec2.internal\" DevicePath \"\"" Sep 16 08:03:17.773086 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:03:17.772940 2557 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/23ed5836-df96-41f7-bd69-129fd932f13c-registry-tls\") on node \"ip-10-0-141-221.ec2.internal\" DevicePath \"\"" Sep 16 08:03:18.273280 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:03:18.273252 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-glszw" Sep 16 08:03:18.273707 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:03:18.273252 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-glszw" event={"ID":"0c4c4aa4-4483-41ad-b383-a03cf353e55b","Type":"ContainerDied","Data":"ebc56ab6ab4cd6f3efe27597fbcad2c6c4c961153b4ad2e15d21d734b5c0637b"} Sep 16 08:03:18.273707 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:03:18.273379 2557 scope.go:117] "RemoveContainer" containerID="9a7a694cce4d348d324488149757d24d92553f517eff9303b8d39570d0ebeb9e" Sep 16 08:03:18.276423 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:03:18.276399 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-c4bxp" event={"ID":"23ed5836-df96-41f7-bd69-129fd932f13c","Type":"ContainerDied","Data":"d517b192adf7d469f6f7760f14cbd008d497c1d13dbe4aebe471c9f9893fe1ee"} Sep 16 08:03:18.276574 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:03:18.276489 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-c4bxp" Sep 16 08:03:18.282468 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:03:18.282448 2557 scope.go:117] "RemoveContainer" containerID="66547bce9bc1543c4ef25c9618a32afc2c923f9c1d17b1f40e7aa99e95553919" Sep 16 08:03:18.289786 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:03:18.289769 2557 scope.go:117] "RemoveContainer" containerID="bd3c728a44b50f2bb9e1a2feda3899f03566d70776f59040654b7669c64dada1" Sep 16 08:03:18.296266 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:03:18.296231 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-glszw"] Sep 16 08:03:18.299170 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:03:18.299137 2557 scope.go:117] "RemoveContainer" containerID="8c541ee377f603f45b70a7be20e662d2cb9a8e35f41b6c7657ecf3061435a337" Sep 16 08:03:18.301427 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:03:18.301411 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-glszw"] Sep 16 08:03:18.306800 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:03:18.306783 2557 scope.go:117] "RemoveContainer" containerID="0f8b8411a38d09b2bca3800cfc07d6261fbb55ce17e069b1e7618fc7d4581e97" Sep 16 08:03:18.312207 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:03:18.312187 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-c4bxp"] Sep 16 08:03:18.314055 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:03:18.314041 2557 scope.go:117] "RemoveContainer" containerID="7a02e32b0b1fe24f47bbb850df8fffbe6aafa15a7fc7303851ff4dfbd5c679e4" Sep 16 08:03:18.318016 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:03:18.317998 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-c4bxp"] Sep 16 08:03:18.321310 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:03:18.321294 2557 scope.go:117] "RemoveContainer" containerID="013a1b2fe4d94dd257ebfa55a3deefb39aaf6dd5ebdbe8574d63d00144e8170a" Sep 16 08:03:18.327690 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:03:18.327677 2557 scope.go:117] "RemoveContainer" containerID="8dd4e9766e9c334a298a71f95c3acbb0820a63e4f615e62bc166f3234e4aad75" Sep 16 08:03:18.334138 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:03:18.334124 2557 scope.go:117] "RemoveContainer" containerID="b2e975bd990dd19b4cb1c0fdaaef0519eea2bec21666767dfa79e43ae8df250a" Sep 16 08:03:18.340783 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:03:18.340767 2557 scope.go:117] "RemoveContainer" containerID="354a581f7e8d5d3d9cde3c5fc653038c5a84b170c7cb21693e60828bab62edb8" Sep 16 08:03:18.347070 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:03:18.347054 2557 scope.go:117] "RemoveContainer" containerID="25c4b319b22cbeec8576f34ab28af3441b18d884596145296aa5803cb75be3e7" Sep 16 08:03:19.523510 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:03:19.523475 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c4c4aa4-4483-41ad-b383-a03cf353e55b" path="/var/lib/kubelet/pods/0c4c4aa4-4483-41ad-b383-a03cf353e55b/volumes" Sep 16 08:03:19.524162 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:03:19.524140 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="23ed5836-df96-41f7-bd69-129fd932f13c" path="/var/lib/kubelet/pods/23ed5836-df96-41f7-bd69-129fd932f13c/volumes" Sep 16 08:03:54.113680 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:03:54.113649 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-n9r4p/must-gather-64jx7"] Sep 16 08:03:54.114118 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:03:54.113900 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c4c4aa4-4483-41ad-b383-a03cf353e55b" containerName="online" Sep 16 08:03:54.114118 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:03:54.113910 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c4c4aa4-4483-41ad-b383-a03cf353e55b" containerName="online" Sep 16 08:03:54.114118 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:03:54.113921 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c4c4aa4-4483-41ad-b383-a03cf353e55b" containerName="feast-init" Sep 16 08:03:54.114118 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:03:54.113927 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c4c4aa4-4483-41ad-b383-a03cf353e55b" containerName="feast-init" Sep 16 08:03:54.114118 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:03:54.113934 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c4c4aa4-4483-41ad-b383-a03cf353e55b" containerName="feast-apply" Sep 16 08:03:54.114118 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:03:54.113939 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c4c4aa4-4483-41ad-b383-a03cf353e55b" containerName="feast-apply" Sep 16 08:03:54.114118 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:03:54.113947 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23ed5836-df96-41f7-bd69-129fd932f13c" containerName="registry" Sep 16 08:03:54.114118 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:03:54.113953 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="23ed5836-df96-41f7-bd69-129fd932f13c" containerName="registry" Sep 16 08:03:54.114118 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:03:54.113958 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23ed5836-df96-41f7-bd69-129fd932f13c" containerName="online" Sep 16 08:03:54.114118 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:03:54.113963 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="23ed5836-df96-41f7-bd69-129fd932f13c" containerName="online" Sep 16 08:03:54.114118 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:03:54.113970 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23ed5836-df96-41f7-bd69-129fd932f13c" containerName="offline" Sep 16 08:03:54.114118 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:03:54.113976 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="23ed5836-df96-41f7-bd69-129fd932f13c" containerName="offline" Sep 16 08:03:54.114118 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:03:54.113982 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c4c4aa4-4483-41ad-b383-a03cf353e55b" containerName="ui" Sep 16 08:03:54.114118 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:03:54.113987 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c4c4aa4-4483-41ad-b383-a03cf353e55b" containerName="ui" Sep 16 08:03:54.114118 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:03:54.113998 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c4c4aa4-4483-41ad-b383-a03cf353e55b" containerName="offline" Sep 16 08:03:54.114118 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:03:54.114003 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c4c4aa4-4483-41ad-b383-a03cf353e55b" containerName="offline" Sep 16 08:03:54.114118 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:03:54.114008 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23ed5836-df96-41f7-bd69-129fd932f13c" containerName="feast-init" Sep 16 08:03:54.114118 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:03:54.114013 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="23ed5836-df96-41f7-bd69-129fd932f13c" containerName="feast-init" Sep 16 08:03:54.114118 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:03:54.114020 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23ed5836-df96-41f7-bd69-129fd932f13c" containerName="feast-apply" Sep 16 08:03:54.114118 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:03:54.114024 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="23ed5836-df96-41f7-bd69-129fd932f13c" containerName="feast-apply" Sep 16 08:03:54.114118 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:03:54.114031 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23ed5836-df96-41f7-bd69-129fd932f13c" containerName="ui" Sep 16 08:03:54.114118 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:03:54.114039 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="23ed5836-df96-41f7-bd69-129fd932f13c" containerName="ui" Sep 16 08:03:54.114118 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:03:54.114074 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="23ed5836-df96-41f7-bd69-129fd932f13c" containerName="online" Sep 16 08:03:54.114118 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:03:54.114081 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="23ed5836-df96-41f7-bd69-129fd932f13c" containerName="registry" Sep 16 08:03:54.114118 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:03:54.114090 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c4c4aa4-4483-41ad-b383-a03cf353e55b" containerName="offline" Sep 16 08:03:54.114118 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:03:54.114097 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="23ed5836-df96-41f7-bd69-129fd932f13c" containerName="offline" Sep 16 08:03:54.114118 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:03:54.114102 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="23ed5836-df96-41f7-bd69-129fd932f13c" containerName="ui" Sep 16 08:03:54.114118 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:03:54.114108 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c4c4aa4-4483-41ad-b383-a03cf353e55b" containerName="ui" Sep 16 08:03:54.114118 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:03:54.114113 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c4c4aa4-4483-41ad-b383-a03cf353e55b" containerName="online" Sep 16 08:03:54.116078 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:03:54.116055 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-n9r4p/must-gather-64jx7" Sep 16 08:03:54.118189 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:03:54.118170 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-n9r4p\"/\"default-dockercfg-n5fgq\"" Sep 16 08:03:54.118303 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:03:54.118201 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-n9r4p\"/\"openshift-service-ca.crt\"" Sep 16 08:03:54.118901 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:03:54.118883 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-n9r4p\"/\"kube-root-ca.crt\"" Sep 16 08:03:54.124055 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:03:54.124034 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-n9r4p/must-gather-64jx7"] Sep 16 08:03:54.216883 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:03:54.216857 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/439b197c-997c-41c9-a485-b42c234ed65e-must-gather-output\") pod \"must-gather-64jx7\" (UID: \"439b197c-997c-41c9-a485-b42c234ed65e\") " pod="openshift-must-gather-n9r4p/must-gather-64jx7" Sep 16 08:03:54.217008 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:03:54.216904 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62kwn\" (UniqueName: \"kubernetes.io/projected/439b197c-997c-41c9-a485-b42c234ed65e-kube-api-access-62kwn\") pod \"must-gather-64jx7\" (UID: \"439b197c-997c-41c9-a485-b42c234ed65e\") " pod="openshift-must-gather-n9r4p/must-gather-64jx7" Sep 16 08:03:54.317159 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:03:54.317124 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/439b197c-997c-41c9-a485-b42c234ed65e-must-gather-output\") pod \"must-gather-64jx7\" (UID: \"439b197c-997c-41c9-a485-b42c234ed65e\") " pod="openshift-must-gather-n9r4p/must-gather-64jx7" Sep 16 08:03:54.317287 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:03:54.317179 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62kwn\" (UniqueName: \"kubernetes.io/projected/439b197c-997c-41c9-a485-b42c234ed65e-kube-api-access-62kwn\") pod \"must-gather-64jx7\" (UID: \"439b197c-997c-41c9-a485-b42c234ed65e\") " pod="openshift-must-gather-n9r4p/must-gather-64jx7" Sep 16 08:03:54.317434 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:03:54.317418 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/439b197c-997c-41c9-a485-b42c234ed65e-must-gather-output\") pod \"must-gather-64jx7\" (UID: \"439b197c-997c-41c9-a485-b42c234ed65e\") " pod="openshift-must-gather-n9r4p/must-gather-64jx7" Sep 16 08:03:54.325351 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:03:54.325334 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-62kwn\" (UniqueName: \"kubernetes.io/projected/439b197c-997c-41c9-a485-b42c234ed65e-kube-api-access-62kwn\") pod \"must-gather-64jx7\" (UID: \"439b197c-997c-41c9-a485-b42c234ed65e\") " pod="openshift-must-gather-n9r4p/must-gather-64jx7" Sep 16 08:03:54.425278 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:03:54.425255 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-n9r4p/must-gather-64jx7" Sep 16 08:03:54.690130 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:03:54.690059 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-n9r4p/must-gather-64jx7"] Sep 16 08:03:54.692821 ip-10-0-141-221 kubenswrapper[2557]: W0916 08:03:54.692795 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod439b197c_997c_41c9_a485_b42c234ed65e.slice/crio-abb3e8dbf61f428444efba74193a812cac4dfde4fc5683d262e85294330dd42f WatchSource:0}: Error finding container abb3e8dbf61f428444efba74193a812cac4dfde4fc5683d262e85294330dd42f: Status 404 returned error can't find the container with id abb3e8dbf61f428444efba74193a812cac4dfde4fc5683d262e85294330dd42f Sep 16 08:03:54.694690 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:03:54.694677 2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 16 08:03:55.405157 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:03:55.405118 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-n9r4p/must-gather-64jx7" event={"ID":"439b197c-997c-41c9-a485-b42c234ed65e","Type":"ContainerStarted","Data":"abb3e8dbf61f428444efba74193a812cac4dfde4fc5683d262e85294330dd42f"} Sep 16 08:03:56.413159 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:03:56.413121 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-n9r4p/must-gather-64jx7" event={"ID":"439b197c-997c-41c9-a485-b42c234ed65e","Type":"ContainerStarted","Data":"8c2deea87f24de8382baef748d854669ec512d883211cab6e8af6ed51daa2213"} Sep 16 08:03:56.413564 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:03:56.413173 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-n9r4p/must-gather-64jx7" event={"ID":"439b197c-997c-41c9-a485-b42c234ed65e","Type":"ContainerStarted","Data":"03fb28ff234294b4fec85cff4c9311c3168caad6cd95c070d11e92953577e827"} Sep 16 08:03:56.429316 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:03:56.429255 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-n9r4p/must-gather-64jx7" podStartSLOduration=1.265412121 podStartE2EDuration="2.429216519s" totalImagesPullingTime="1.163804398s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 08:03:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 08:03:56.427260007 +0000 UTC m=+2169.454161019" watchObservedRunningTime="2026-09-16 08:03:56.429216519 +0000 UTC m=+2169.456117532" Sep 16 08:03:57.121219 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:03:57.121186 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-t8thj_90e5e23a-060b-4612-80f5-f9a600b6d476/global-pull-secret-syncer/0.log" Sep 16 08:03:57.268207 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:03:57.268180 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-lq72c_1ae92448-74b5-4b28-b337-758f6884d09a/konnectivity-agent/0.log" Sep 16 08:03:57.308125 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:03:57.308094 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-221.ec2.internal_3825bcd696c373ea98c5e3521a26b861/haproxy/0.log" Sep 16 08:04:01.396277 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:04:01.396227 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-mkpdq_5e8b982e-e811-49dd-9ca7-fcdbd0cf36d1/kube-state-metrics/0.log" Sep 16 08:04:01.415980 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:04:01.415945 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-mkpdq_5e8b982e-e811-49dd-9ca7-fcdbd0cf36d1/kube-rbac-proxy-main/0.log" Sep 16 08:04:01.439540 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:04:01.439508 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-mkpdq_5e8b982e-e811-49dd-9ca7-fcdbd0cf36d1/kube-rbac-proxy-self/0.log" Sep 16 08:04:01.496627 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:04:01.496591 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5d85778b9d-tzgsd_d31607fb-50b7-4280-a7bc-b7242a48207c/monitoring-plugin/0.log" Sep 16 08:04:01.536759 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:04:01.536730 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5tdrc_e0c3b327-c203-42b9-92ab-7ee31f7153ba/node-exporter/0.log" Sep 16 08:04:01.558338 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:04:01.558287 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5tdrc_e0c3b327-c203-42b9-92ab-7ee31f7153ba/kube-rbac-proxy/0.log" Sep 16 08:04:01.586496 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:04:01.586468 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5tdrc_e0c3b327-c203-42b9-92ab-7ee31f7153ba/init-textfile/0.log" Sep 16 08:04:02.159750 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:04:02.159674 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7dc978598b-7j5h4_7ca2b45d-2c2d-4d42-8f53-285ff2febfdc/thanos-query/0.log" Sep 16 08:04:02.189534 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:04:02.189489 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7dc978598b-7j5h4_7ca2b45d-2c2d-4d42-8f53-285ff2febfdc/kube-rbac-proxy-web/0.log" Sep 16 08:04:02.213147 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:04:02.213086 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7dc978598b-7j5h4_7ca2b45d-2c2d-4d42-8f53-285ff2febfdc/kube-rbac-proxy/0.log" Sep 16 08:04:02.245293 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:04:02.245269 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7dc978598b-7j5h4_7ca2b45d-2c2d-4d42-8f53-285ff2febfdc/prom-label-proxy/0.log" Sep 16 08:04:02.277202 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:04:02.277167 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7dc978598b-7j5h4_7ca2b45d-2c2d-4d42-8f53-285ff2febfdc/kube-rbac-proxy-rules/0.log" Sep 16 08:04:02.305821 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:04:02.305285 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7dc978598b-7j5h4_7ca2b45d-2c2d-4d42-8f53-285ff2febfdc/kube-rbac-proxy-metrics/0.log" Sep 16 08:04:04.212885 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:04:04.212854 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-747795c55-lfcd2_a120f5cd-d5b4-4c52-8067-0bb771c34f6c/console/0.log" Sep 16 08:04:04.597258 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:04:04.597147 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-n9r4p/perf-node-gather-daemonset-c6pf6"] Sep 16 08:04:04.602066 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:04:04.602036 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-n9r4p/perf-node-gather-daemonset-c6pf6" Sep 16 08:04:04.613913 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:04:04.613824 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-n9r4p/perf-node-gather-daemonset-c6pf6"] Sep 16 08:04:04.700575 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:04:04.700541 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/65e27bb8-83fb-4fd8-91da-7f0cc2aa96d9-sys\") pod \"perf-node-gather-daemonset-c6pf6\" (UID: \"65e27bb8-83fb-4fd8-91da-7f0cc2aa96d9\") " pod="openshift-must-gather-n9r4p/perf-node-gather-daemonset-c6pf6" Sep 16 08:04:04.700733 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:04:04.700591 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7hk6\" (UniqueName: \"kubernetes.io/projected/65e27bb8-83fb-4fd8-91da-7f0cc2aa96d9-kube-api-access-j7hk6\") pod \"perf-node-gather-daemonset-c6pf6\" (UID: \"65e27bb8-83fb-4fd8-91da-7f0cc2aa96d9\") " pod="openshift-must-gather-n9r4p/perf-node-gather-daemonset-c6pf6" Sep 16 08:04:04.700733 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:04:04.700698 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/65e27bb8-83fb-4fd8-91da-7f0cc2aa96d9-lib-modules\") pod \"perf-node-gather-daemonset-c6pf6\" (UID: \"65e27bb8-83fb-4fd8-91da-7f0cc2aa96d9\") " pod="openshift-must-gather-n9r4p/perf-node-gather-daemonset-c6pf6" Sep 16 08:04:04.700846 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:04:04.700831 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/65e27bb8-83fb-4fd8-91da-7f0cc2aa96d9-proc\") pod \"perf-node-gather-daemonset-c6pf6\" (UID: \"65e27bb8-83fb-4fd8-91da-7f0cc2aa96d9\") " pod="openshift-must-gather-n9r4p/perf-node-gather-daemonset-c6pf6" Sep 16 08:04:04.700910 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:04:04.700860 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/65e27bb8-83fb-4fd8-91da-7f0cc2aa96d9-podres\") pod \"perf-node-gather-daemonset-c6pf6\" (UID: \"65e27bb8-83fb-4fd8-91da-7f0cc2aa96d9\") " pod="openshift-must-gather-n9r4p/perf-node-gather-daemonset-c6pf6" Sep 16 08:04:04.802127 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:04:04.802088 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7hk6\" (UniqueName: \"kubernetes.io/projected/65e27bb8-83fb-4fd8-91da-7f0cc2aa96d9-kube-api-access-j7hk6\") pod \"perf-node-gather-daemonset-c6pf6\" (UID: \"65e27bb8-83fb-4fd8-91da-7f0cc2aa96d9\") " pod="openshift-must-gather-n9r4p/perf-node-gather-daemonset-c6pf6" Sep 16 08:04:04.802352 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:04:04.802152 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/65e27bb8-83fb-4fd8-91da-7f0cc2aa96d9-lib-modules\") pod \"perf-node-gather-daemonset-c6pf6\" (UID: \"65e27bb8-83fb-4fd8-91da-7f0cc2aa96d9\") " pod="openshift-must-gather-n9r4p/perf-node-gather-daemonset-c6pf6" Sep 16 08:04:04.802352 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:04:04.802268 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/65e27bb8-83fb-4fd8-91da-7f0cc2aa96d9-proc\") pod \"perf-node-gather-daemonset-c6pf6\" (UID: \"65e27bb8-83fb-4fd8-91da-7f0cc2aa96d9\") " pod="openshift-must-gather-n9r4p/perf-node-gather-daemonset-c6pf6" Sep 16 08:04:04.802352 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:04:04.802299 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/65e27bb8-83fb-4fd8-91da-7f0cc2aa96d9-podres\") pod \"perf-node-gather-daemonset-c6pf6\" (UID: \"65e27bb8-83fb-4fd8-91da-7f0cc2aa96d9\") " pod="openshift-must-gather-n9r4p/perf-node-gather-daemonset-c6pf6" Sep 16 08:04:04.802352 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:04:04.802335 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/65e27bb8-83fb-4fd8-91da-7f0cc2aa96d9-sys\") pod \"perf-node-gather-daemonset-c6pf6\" (UID: \"65e27bb8-83fb-4fd8-91da-7f0cc2aa96d9\") " pod="openshift-must-gather-n9r4p/perf-node-gather-daemonset-c6pf6" Sep 16 08:04:04.802605 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:04:04.802433 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/65e27bb8-83fb-4fd8-91da-7f0cc2aa96d9-sys\") pod \"perf-node-gather-daemonset-c6pf6\" (UID: \"65e27bb8-83fb-4fd8-91da-7f0cc2aa96d9\") " pod="openshift-must-gather-n9r4p/perf-node-gather-daemonset-c6pf6" Sep 16 08:04:04.802792 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:04:04.802765 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/65e27bb8-83fb-4fd8-91da-7f0cc2aa96d9-proc\") pod \"perf-node-gather-daemonset-c6pf6\" (UID: \"65e27bb8-83fb-4fd8-91da-7f0cc2aa96d9\") " pod="openshift-must-gather-n9r4p/perf-node-gather-daemonset-c6pf6" Sep 16 08:04:04.802932 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:04:04.802881 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/65e27bb8-83fb-4fd8-91da-7f0cc2aa96d9-podres\") pod \"perf-node-gather-daemonset-c6pf6\" (UID: \"65e27bb8-83fb-4fd8-91da-7f0cc2aa96d9\") " pod="openshift-must-gather-n9r4p/perf-node-gather-daemonset-c6pf6" Sep 16 08:04:04.803028 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:04:04.802977 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/65e27bb8-83fb-4fd8-91da-7f0cc2aa96d9-lib-modules\") pod \"perf-node-gather-daemonset-c6pf6\" (UID: \"65e27bb8-83fb-4fd8-91da-7f0cc2aa96d9\") " pod="openshift-must-gather-n9r4p/perf-node-gather-daemonset-c6pf6" Sep 16 08:04:04.810827 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:04:04.810801 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7hk6\" (UniqueName: \"kubernetes.io/projected/65e27bb8-83fb-4fd8-91da-7f0cc2aa96d9-kube-api-access-j7hk6\") pod \"perf-node-gather-daemonset-c6pf6\" (UID: \"65e27bb8-83fb-4fd8-91da-7f0cc2aa96d9\") " pod="openshift-must-gather-n9r4p/perf-node-gather-daemonset-c6pf6" Sep 16 08:04:04.916015 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:04:04.915989 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-n9r4p/perf-node-gather-daemonset-c6pf6" Sep 16 08:04:05.192202 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:04:05.191788 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-n9r4p/perf-node-gather-daemonset-c6pf6"] Sep 16 08:04:05.461267 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:04:05.461184 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-v8n9m_dbc6f54b-f7e7-4ede-b9a2-58f81e7a3f4c/dns/0.log" Sep 16 08:04:05.474335 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:04:05.474311 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-n9r4p/perf-node-gather-daemonset-c6pf6" event={"ID":"65e27bb8-83fb-4fd8-91da-7f0cc2aa96d9","Type":"ContainerStarted","Data":"dd256a4e7608fff7075f9aee962249080387eebd802a32a8c7f2ee71e493c66b"} Sep 16 08:04:05.474483 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:04:05.474363 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-n9r4p/perf-node-gather-daemonset-c6pf6" event={"ID":"65e27bb8-83fb-4fd8-91da-7f0cc2aa96d9","Type":"ContainerStarted","Data":"72a33d94bce4c35b0aa8f4449a8b9b4d78d4191e0be5b61fcc357ac1d3a846ea"} Sep 16 08:04:05.474554 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:04:05.474507 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-n9r4p/perf-node-gather-daemonset-c6pf6" Sep 16 08:04:05.480179 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:04:05.480161 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-v8n9m_dbc6f54b-f7e7-4ede-b9a2-58f81e7a3f4c/kube-rbac-proxy/0.log" Sep 16 08:04:05.489828 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:04:05.489793 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-n9r4p/perf-node-gather-daemonset-c6pf6" podStartSLOduration=1.489780414 podStartE2EDuration="1.489780414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 08:04:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 08:04:05.488513399 +0000 UTC m=+2178.515414433" watchObservedRunningTime="2026-09-16 08:04:05.489780414 +0000 UTC m=+2178.516681426" Sep 16 08:04:05.588085 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:04:05.588065 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-8gnk6_20f6d954-1104-49b9-ab10-32e0de4ef8e0/dns-node-resolver/0.log" Sep 16 08:04:06.034122 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:04:06.034089 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-vbxvc_89f6995b-d178-4fe9-9593-ba8dc8f49351/node-ca/0.log" Sep 16 08:04:06.989517 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:04:06.989490 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-lprn9_7c9bb910-6eda-4d97-9d79-7d78fd9abb58/serve-healthcheck-canary/0.log" Sep 16 08:04:07.470698 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:04:07.470673 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xq64b_44a442da-5d1a-4b97-8697-3ce085773de6/kube-rbac-proxy/0.log" Sep 16 08:04:07.489113 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:04:07.489090 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xq64b_44a442da-5d1a-4b97-8697-3ce085773de6/exporter/0.log" Sep 16 08:04:07.507990 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:04:07.507971 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xq64b_44a442da-5d1a-4b97-8697-3ce085773de6/extractor/0.log" Sep 16 08:04:11.488536 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:04:11.488511 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-n9r4p/perf-node-gather-daemonset-c6pf6" Sep 16 08:04:12.289923 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:04:12.289896 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-4cb95_280250dc-6c09-4766-822c-b8bdf97e4a49/kube-multus/0.log" Sep 16 08:04:12.653069 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:04:12.653038 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qprhb_5f7afb78-2955-49e0-adec-6e8aae7ae51a/kube-multus-additional-cni-plugins/0.log" Sep 16 08:04:12.673043 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:04:12.673018 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qprhb_5f7afb78-2955-49e0-adec-6e8aae7ae51a/egress-router-binary-copy/0.log" Sep 16 08:04:12.695253 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:04:12.695212 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qprhb_5f7afb78-2955-49e0-adec-6e8aae7ae51a/cni-plugins/0.log" Sep 16 08:04:12.715322 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:04:12.715290 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qprhb_5f7afb78-2955-49e0-adec-6e8aae7ae51a/bond-cni-plugin/0.log" Sep 16 08:04:12.736938 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:04:12.736912 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qprhb_5f7afb78-2955-49e0-adec-6e8aae7ae51a/routeoverride-cni/0.log" Sep 16 08:04:12.758172 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:04:12.758126 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qprhb_5f7afb78-2955-49e0-adec-6e8aae7ae51a/whereabouts-cni-bincopy/0.log" Sep 16 08:04:12.780372 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:04:12.780340 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qprhb_5f7afb78-2955-49e0-adec-6e8aae7ae51a/whereabouts-cni/0.log" Sep 16 08:04:12.853215 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:04:12.853190 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-kqc7r_296c5dcf-403c-4a1f-9d6f-05ffb7d75e38/network-metrics-daemon/0.log" Sep 16 08:04:12.870737 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:04:12.870720 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-kqc7r_296c5dcf-403c-4a1f-9d6f-05ffb7d75e38/kube-rbac-proxy/0.log" Sep 16 08:04:13.706824 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:04:13.706794 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4ztgb_8d5c17cc-f191-4354-9c73-ab7f25dd203b/ovn-controller/0.log" Sep 16 08:04:13.724347 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:04:13.724319 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4ztgb_8d5c17cc-f191-4354-9c73-ab7f25dd203b/ovn-acl-logging/0.log" Sep 16 08:04:13.751644 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:04:13.751609 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4ztgb_8d5c17cc-f191-4354-9c73-ab7f25dd203b/kube-rbac-proxy-node/0.log" Sep 16 08:04:13.772795 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:04:13.772747 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4ztgb_8d5c17cc-f191-4354-9c73-ab7f25dd203b/kube-rbac-proxy-ovn-metrics/0.log" Sep 16 08:04:13.790217 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:04:13.790192 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4ztgb_8d5c17cc-f191-4354-9c73-ab7f25dd203b/northd/0.log" Sep 16 08:04:13.811708 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:04:13.811686 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4ztgb_8d5c17cc-f191-4354-9c73-ab7f25dd203b/nbdb/0.log" Sep 16 08:04:13.831432 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:04:13.831408 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4ztgb_8d5c17cc-f191-4354-9c73-ab7f25dd203b/sbdb/0.log" Sep 16 08:04:13.873303 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:04:13.873215 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4ztgb_8d5c17cc-f191-4354-9c73-ab7f25dd203b/ovnkube-controller/0.log" Sep 16 08:04:15.318296 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:04:15.318208 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-jtqz9_bfde2ba8-6a5f-413f-9ce9-d9ce0af2c758/network-check-target-container/0.log" Sep 16 08:04:16.129219 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:04:16.129150 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-fhnn7_53ab9fb8-54fd-478d-9b2b-f69c8aac0d02/iptables-alerter/0.log" Sep 16 08:04:16.818851 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:04:16.818822 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-zjxc4_eb688a0e-2bd2-4df2-b475-7eba4237e526/tuned/0.log" Sep 16 08:04:19.939307 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:04:19.939274 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-vj2rd_17757d39-4eae-4f8d-8c3a-679d3502d985/csi-driver/0.log" Sep 16 08:04:19.957419 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:04:19.957399 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-vj2rd_17757d39-4eae-4f8d-8c3a-679d3502d985/csi-node-driver-registrar/0.log" Sep 16 08:04:19.975573 ip-10-0-141-221 kubenswrapper[2557]: I0916 08:04:19.975554 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-vj2rd_17757d39-4eae-4f8d-8c3a-679d3502d985/csi-liveness-probe/0.log"