Sep 17 07:25:44.830740 ip-10-0-139-100 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Sep 17 07:25:44.830755 ip-10-0-139-100 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Sep 17 07:25:44.830764 ip-10-0-139-100 systemd[1]: kubelet.service: Failed with result 'resources'. Sep 17 07:25:44.831093 ip-10-0-139-100 systemd[1]: Failed to start Kubernetes Kubelet. Sep 17 07:25:55.079735 ip-10-0-139-100 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Sep 17 07:25:55.079758 ip-10-0-139-100 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 1d194449761d4b0ca2abf2e441ed5399 -- Sep 17 07:27:34.987476 ip-10-0-139-100 systemd[1]: Starting Kubernetes Kubelet... Sep 17 07:27:35.412254 ip-10-0-139-100 kubenswrapper[2578]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 17 07:27:35.412254 ip-10-0-139-100 kubenswrapper[2578]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 17 07:27:35.412254 ip-10-0-139-100 kubenswrapper[2578]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 17 07:27:35.412254 ip-10-0-139-100 kubenswrapper[2578]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 17 07:27:35.412254 ip-10-0-139-100 kubenswrapper[2578]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 17 07:27:35.412891 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.412324 2578 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 17 07:27:35.416215 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.416200 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 17 07:27:35.416215 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.416214 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 17 07:27:35.416276 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.416219 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 17 07:27:35.416276 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.416224 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 17 07:27:35.416276 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.416229 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 17 07:27:35.416276 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.416232 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 17 07:27:35.416276 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.416236 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 17 07:27:35.416276 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.416239 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 17 07:27:35.416276 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.416242 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 17 07:27:35.416276 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.416244 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 17 07:27:35.416276 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.416247 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 17 07:27:35.416276 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.416249 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 17 07:27:35.416276 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.416252 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 17 07:27:35.416276 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.416255 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 17 07:27:35.416276 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.416257 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 17 07:27:35.416276 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.416260 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 17 07:27:35.416276 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.416262 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 17 07:27:35.416276 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.416266 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 17 07:27:35.416276 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.416268 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 17 07:27:35.416276 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.416271 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 17 07:27:35.416276 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.416274 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 17 07:27:35.416722 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.416277 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 17 07:27:35.416722 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.416280 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 17 07:27:35.416722 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.416283 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 17 07:27:35.416722 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.416285 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 17 07:27:35.416722 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.416288 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 17 07:27:35.416722 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.416291 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 17 07:27:35.416722 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.416294 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 17 07:27:35.416722 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.416296 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 17 07:27:35.416722 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.416300 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 17 07:27:35.416722 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.416302 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 17 07:27:35.416722 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.416305 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 17 07:27:35.416722 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.416308 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 17 07:27:35.416722 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.416310 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 17 07:27:35.416722 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.416313 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 17 07:27:35.416722 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.416316 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 17 07:27:35.416722 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.416318 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 17 07:27:35.416722 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.416323 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 17 07:27:35.416722 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.416326 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 17 07:27:35.416722 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.416329 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 17 07:27:35.416722 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.416331 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 17 07:27:35.417446 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.416334 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 17 07:27:35.417446 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.416337 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 17 07:27:35.417446 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.416339 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 17 07:27:35.417446 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.416342 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 17 07:27:35.417446 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.416344 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 17 07:27:35.417446 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.416347 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 17 07:27:35.417446 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.416349 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 17 07:27:35.417446 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.416352 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 17 07:27:35.417446 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.416355 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 17 07:27:35.417446 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.416357 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 17 07:27:35.417446 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.416360 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 17 07:27:35.417446 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.416362 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 17 07:27:35.417446 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.416365 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 17 07:27:35.417446 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.416368 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 17 07:27:35.417446 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.416371 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 17 07:27:35.417446 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.416373 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 17 07:27:35.417446 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.416376 2578 feature_gate.go:328] unrecognized feature gate: Example Sep 17 07:27:35.417446 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.416378 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 17 07:27:35.417446 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.416381 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 17 07:27:35.417446 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.416383 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 17 07:27:35.417986 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.416386 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 17 07:27:35.417986 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.416388 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 17 07:27:35.417986 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.416391 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 17 07:27:35.417986 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.416395 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 17 07:27:35.417986 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.416400 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 17 07:27:35.417986 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.416402 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 17 07:27:35.417986 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.416405 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 17 07:27:35.417986 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.416408 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 17 07:27:35.417986 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.416411 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 17 07:27:35.417986 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.416413 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 17 07:27:35.417986 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.416416 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 17 07:27:35.417986 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.416419 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 17 07:27:35.417986 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.416422 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 17 07:27:35.417986 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.416425 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 17 07:27:35.417986 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.416427 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 17 07:27:35.417986 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.416430 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 17 07:27:35.417986 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.416433 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 17 07:27:35.417986 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.416436 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 17 07:27:35.417986 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.416439 2578 feature_gate.go:328] unrecognized feature gate: Example2 Sep 17 07:27:35.418453 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.416441 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 17 07:27:35.418453 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.416444 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 17 07:27:35.418453 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.416446 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 17 07:27:35.418453 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.416449 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 17 07:27:35.418453 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.416452 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 17 07:27:35.418453 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.416454 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 17 07:27:35.418453 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.416458 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 17 07:27:35.418453 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.416460 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 17 07:27:35.418453 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.416463 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 17 07:27:35.418453 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.416466 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 17 07:27:35.418453 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.416468 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 17 07:27:35.418453 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.416471 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 17 07:27:35.418453 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.416473 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 17 07:27:35.418453 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.416476 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 17 07:27:35.418453 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.416478 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 17 07:27:35.418453 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.416482 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 17 07:27:35.418453 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.416484 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 17 07:27:35.418453 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.416487 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 17 07:27:35.418453 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.416490 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 17 07:27:35.420977 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.420957 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 17 07:27:35.420977 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.420974 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 17 07:27:35.420977 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.420980 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 17 07:27:35.421202 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.420984 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 17 07:27:35.421202 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.420989 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 17 07:27:35.421202 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.420994 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 17 07:27:35.421202 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.420999 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 17 07:27:35.421202 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.421003 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 17 07:27:35.421202 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.421006 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 17 07:27:35.421202 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.421010 2578 feature_gate.go:328] unrecognized feature gate: Example2 Sep 17 07:27:35.421202 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.421015 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 17 07:27:35.421202 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.421020 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 17 07:27:35.421202 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.421024 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 17 07:27:35.421202 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.421027 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 17 07:27:35.421202 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.421031 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 17 07:27:35.421202 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.421035 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 17 07:27:35.421202 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.421038 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 17 07:27:35.421202 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.421042 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 17 07:27:35.421202 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.421046 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 17 07:27:35.421202 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.421050 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 17 07:27:35.421202 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.421054 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 17 07:27:35.421202 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.421058 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 17 07:27:35.421202 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.421064 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 17 07:27:35.422038 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.421070 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 17 07:27:35.422038 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.421075 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 17 07:27:35.422038 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.421079 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 17 07:27:35.422038 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.421084 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 17 07:27:35.422038 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.421097 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 17 07:27:35.422038 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.421102 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 17 07:27:35.422038 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.421106 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 17 07:27:35.422038 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.421115 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 17 07:27:35.422038 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.421120 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 17 07:27:35.422038 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.421124 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 17 07:27:35.422038 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.421131 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 17 07:27:35.422038 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.421136 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 17 07:27:35.422038 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.421141 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 17 07:27:35.422038 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.421145 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 17 07:27:35.422038 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.421150 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 17 07:27:35.422038 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.421154 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 17 07:27:35.422038 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.421159 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 17 07:27:35.422038 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.421163 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 17 07:27:35.422038 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.421167 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 17 07:27:35.422038 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.421171 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 17 07:27:35.422910 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.421175 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 17 07:27:35.422910 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.421180 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 17 07:27:35.422910 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.421199 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 17 07:27:35.422910 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.421203 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 17 07:27:35.422910 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.421207 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 17 07:27:35.422910 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.421211 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 17 07:27:35.422910 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.421215 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 17 07:27:35.422910 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.421220 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 17 07:27:35.422910 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.421224 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 17 07:27:35.422910 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.421228 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 17 07:27:35.422910 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.421233 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 17 07:27:35.422910 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.421237 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 17 07:27:35.422910 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.421241 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 17 07:27:35.422910 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.421245 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 17 07:27:35.422910 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.421250 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 17 07:27:35.422910 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.421254 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 17 07:27:35.422910 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.421258 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 17 07:27:35.422910 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.421263 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 17 07:27:35.422910 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.421267 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 17 07:27:35.422910 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.421273 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 17 07:27:35.423783 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.421277 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 17 07:27:35.423783 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.421281 2578 feature_gate.go:328] unrecognized feature gate: Example Sep 17 07:27:35.423783 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.421285 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 17 07:27:35.423783 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.421290 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 17 07:27:35.423783 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.421295 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 17 07:27:35.423783 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.421300 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 17 07:27:35.423783 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.421304 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 17 07:27:35.423783 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.421308 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 17 07:27:35.423783 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.421313 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 17 07:27:35.423783 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.421316 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 17 07:27:35.423783 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.421321 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 17 07:27:35.423783 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.421325 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 17 07:27:35.423783 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.421329 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 17 07:27:35.423783 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.421334 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 17 07:27:35.423783 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.421338 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 17 07:27:35.423783 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.421342 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 17 07:27:35.423783 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.421347 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 17 07:27:35.423783 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.421351 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 17 07:27:35.423783 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.421355 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 17 07:27:35.423783 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.421359 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 17 07:27:35.424477 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.421368 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 17 07:27:35.424477 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.421375 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 17 07:27:35.424477 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.421380 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 17 07:27:35.424477 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.421384 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 17 07:27:35.424477 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.421388 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 17 07:27:35.424477 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.421392 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 17 07:27:35.424477 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.421396 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 17 07:27:35.424477 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.421400 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 17 07:27:35.424477 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.421405 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 17 07:27:35.424477 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.421410 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 17 07:27:35.424477 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.421413 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 17 07:27:35.424477 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.421418 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 17 07:27:35.424477 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.421422 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 17 07:27:35.424477 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.421426 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 17 07:27:35.424477 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.421430 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 17 07:27:35.424477 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.421434 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 17 07:27:35.424477 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.422072 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 17 07:27:35.424477 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.422082 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 17 07:27:35.424477 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.422088 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 17 07:27:35.424939 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.422093 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 17 07:27:35.424939 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.422097 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 17 07:27:35.424939 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.422102 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 17 07:27:35.424939 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.422107 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 17 07:27:35.424939 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.422112 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 17 07:27:35.424939 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.422116 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 17 07:27:35.424939 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.422121 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 17 07:27:35.424939 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.422125 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 17 07:27:35.424939 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.422129 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 17 07:27:35.424939 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.422133 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 17 07:27:35.424939 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.422137 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 17 07:27:35.424939 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.422143 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 17 07:27:35.424939 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.422150 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 17 07:27:35.424939 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.422156 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 17 07:27:35.424939 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.422160 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 17 07:27:35.424939 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.422165 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 17 07:27:35.424939 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.422170 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 17 07:27:35.424939 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.422174 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 17 07:27:35.424939 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.422178 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 17 07:27:35.424939 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.422202 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 17 07:27:35.425539 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.422208 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 17 07:27:35.425539 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.422213 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 17 07:27:35.425539 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.422217 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 17 07:27:35.425539 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.422221 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 17 07:27:35.425539 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.422226 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 17 07:27:35.425539 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.422230 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 17 07:27:35.425539 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.422234 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 17 07:27:35.425539 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.422238 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 17 07:27:35.425539 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.422242 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 17 07:27:35.425539 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.422246 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 17 07:27:35.425539 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.422250 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 17 07:27:35.425539 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.422254 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 17 07:27:35.425539 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.422260 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 17 07:27:35.425539 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.422266 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 17 07:27:35.425539 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.422271 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 17 07:27:35.425539 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.422275 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 17 07:27:35.425539 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.422279 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 17 07:27:35.425539 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.422283 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 17 07:27:35.425539 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.422290 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 17 07:27:35.426007 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.422296 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 17 07:27:35.426007 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.422301 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 17 07:27:35.426007 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.422306 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 17 07:27:35.426007 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.422311 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 17 07:27:35.426007 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.422315 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 17 07:27:35.426007 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.422320 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 17 07:27:35.426007 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.422324 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 17 07:27:35.426007 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.422329 2578 feature_gate.go:328] unrecognized feature gate: Example2 Sep 17 07:27:35.426007 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.422333 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 17 07:27:35.426007 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.422338 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 17 07:27:35.426007 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.422342 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 17 07:27:35.426007 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.422346 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 17 07:27:35.426007 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.422351 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 17 07:27:35.426007 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.422356 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 17 07:27:35.426007 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.422361 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 17 07:27:35.426007 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.422365 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 17 07:27:35.426007 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.422370 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 17 07:27:35.426007 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.422374 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 17 07:27:35.426007 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.422378 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 17 07:27:35.426007 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.422394 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 17 07:27:35.426538 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.422398 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 17 07:27:35.426538 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.422402 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 17 07:27:35.426538 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.422407 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 17 07:27:35.426538 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.422411 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 17 07:27:35.426538 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.422416 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 17 07:27:35.426538 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.422420 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 17 07:27:35.426538 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.422424 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 17 07:27:35.426538 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.422429 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 17 07:27:35.426538 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.422433 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 17 07:27:35.426538 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.422437 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 17 07:27:35.426538 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.422442 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 17 07:27:35.426538 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.422448 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 17 07:27:35.426538 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.422452 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 17 07:27:35.426538 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.422456 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 17 07:27:35.426538 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.422460 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 17 07:27:35.426538 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.422464 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 17 07:27:35.426538 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.422469 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 17 07:27:35.426538 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.422473 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 17 07:27:35.426538 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.422477 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 17 07:27:35.426538 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.422481 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 17 07:27:35.427242 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.422487 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 17 07:27:35.427242 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.422491 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 17 07:27:35.427242 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.422495 2578 feature_gate.go:328] unrecognized feature gate: Example Sep 17 07:27:35.427242 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.422499 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 17 07:27:35.427242 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.422504 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 17 07:27:35.427242 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.422508 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 17 07:27:35.427242 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.422513 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 17 07:27:35.427242 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.422517 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 17 07:27:35.427242 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.422521 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 17 07:27:35.427242 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.422525 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 17 07:27:35.427242 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.422529 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 17 07:27:35.427242 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.422533 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 17 07:27:35.427242 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.422537 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 17 07:27:35.427242 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.422541 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 17 07:27:35.427242 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.422545 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 17 07:27:35.427242 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.422550 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 17 07:27:35.427242 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.422554 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 17 07:27:35.427242 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.422651 2578 flags.go:64] FLAG: --address="0.0.0.0" Sep 17 07:27:35.427242 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.422703 2578 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 17 07:27:35.427242 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.422716 2578 flags.go:64] FLAG: --anonymous-auth="true" Sep 17 07:27:35.427729 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.422723 2578 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 17 07:27:35.427729 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.422729 2578 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 17 07:27:35.427729 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.422734 2578 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 17 07:27:35.427729 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.422741 2578 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 17 07:27:35.427729 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.422748 2578 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 17 07:27:35.427729 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.422753 2578 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 17 07:27:35.427729 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.422758 2578 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 17 07:27:35.427729 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.422764 2578 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 17 07:27:35.427729 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.422769 2578 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 17 07:27:35.427729 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.422774 2578 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 17 07:27:35.427729 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.422778 2578 flags.go:64] FLAG: --cgroup-root="" Sep 17 07:27:35.427729 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.422783 2578 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 17 07:27:35.427729 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.422787 2578 flags.go:64] FLAG: --client-ca-file="" Sep 17 07:27:35.427729 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.422793 2578 flags.go:64] FLAG: --cloud-provider="external" Sep 17 07:27:35.427729 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.422798 2578 flags.go:64] FLAG: --cluster-dns="[]" Sep 17 07:27:35.427729 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.422804 2578 flags.go:64] FLAG: --cluster-domain="" Sep 17 07:27:35.427729 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.422808 2578 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 17 07:27:35.427729 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.422813 2578 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 17 07:27:35.427729 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.422818 2578 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 17 07:27:35.427729 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.422823 2578 flags.go:64] FLAG: --container-log-max-files="5" Sep 17 07:27:35.427729 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.422829 2578 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 17 07:27:35.427729 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.422834 2578 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 17 07:27:35.427729 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.422839 2578 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 17 07:27:35.428412 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.422844 2578 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 17 07:27:35.428412 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.422849 2578 flags.go:64] FLAG: --contention-profiling="false" Sep 17 07:27:35.428412 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.422854 2578 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 17 07:27:35.428412 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.422858 2578 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 17 07:27:35.428412 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.422863 2578 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 17 07:27:35.428412 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.422868 2578 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 17 07:27:35.428412 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.422877 2578 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 17 07:27:35.428412 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.422883 2578 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 17 07:27:35.428412 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.422888 2578 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 17 07:27:35.428412 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.422892 2578 flags.go:64] FLAG: --enable-load-reader="false" Sep 17 07:27:35.428412 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.422897 2578 flags.go:64] FLAG: --enable-server="true" Sep 17 07:27:35.428412 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.422902 2578 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 17 07:27:35.428412 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.422909 2578 flags.go:64] FLAG: --event-burst="100" Sep 17 07:27:35.428412 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.422913 2578 flags.go:64] FLAG: --event-qps="50" Sep 17 07:27:35.428412 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.422918 2578 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 17 07:27:35.428412 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.422942 2578 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 17 07:27:35.428412 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.422948 2578 flags.go:64] FLAG: --eviction-hard="" Sep 17 07:27:35.428412 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.422955 2578 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 17 07:27:35.428412 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.422959 2578 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 17 07:27:35.428412 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.422964 2578 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 17 07:27:35.428412 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.422969 2578 flags.go:64] FLAG: --eviction-soft="" Sep 17 07:27:35.428412 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.422974 2578 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 17 07:27:35.428412 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.422978 2578 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 17 07:27:35.428412 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.422984 2578 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 17 07:27:35.428412 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.422989 2578 flags.go:64] FLAG: --experimental-mounter-path="" Sep 17 07:27:35.429018 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.422995 2578 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 17 07:27:35.429018 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.423000 2578 flags.go:64] FLAG: --fail-swap-on="true" Sep 17 07:27:35.429018 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.423004 2578 flags.go:64] FLAG: --feature-gates="" Sep 17 07:27:35.429018 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.423010 2578 flags.go:64] FLAG: --file-check-frequency="20s" Sep 17 07:27:35.429018 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.423015 2578 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 17 07:27:35.429018 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.423020 2578 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 17 07:27:35.429018 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.423025 2578 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 17 07:27:35.429018 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.423030 2578 flags.go:64] FLAG: --healthz-port="10248" Sep 17 07:27:35.429018 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.423035 2578 flags.go:64] FLAG: --help="false" Sep 17 07:27:35.429018 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.423040 2578 flags.go:64] FLAG: --hostname-override="ip-10-0-139-100.ec2.internal" Sep 17 07:27:35.429018 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.423045 2578 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 17 07:27:35.429018 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.423050 2578 flags.go:64] FLAG: --http-check-frequency="20s" Sep 17 07:27:35.429018 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.423055 2578 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 17 07:27:35.429018 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.423062 2578 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 17 07:27:35.429018 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.423068 2578 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 17 07:27:35.429018 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.423072 2578 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 17 07:27:35.429018 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.423077 2578 flags.go:64] FLAG: --image-service-endpoint="" Sep 17 07:27:35.429018 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.423082 2578 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 17 07:27:35.429018 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.423086 2578 flags.go:64] FLAG: --kube-api-burst="100" Sep 17 07:27:35.429018 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.423091 2578 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 17 07:27:35.429018 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.423096 2578 flags.go:64] FLAG: --kube-api-qps="50" Sep 17 07:27:35.429018 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.423101 2578 flags.go:64] FLAG: --kube-reserved="" Sep 17 07:27:35.429018 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.423106 2578 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 17 07:27:35.429018 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.423111 2578 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 17 07:27:35.429643 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.423116 2578 flags.go:64] FLAG: --kubelet-cgroups="" Sep 17 07:27:35.429643 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.423121 2578 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 17 07:27:35.429643 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.423125 2578 flags.go:64] FLAG: --lock-file="" Sep 17 07:27:35.429643 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.423130 2578 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 17 07:27:35.429643 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.423134 2578 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 17 07:27:35.429643 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.423139 2578 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 17 07:27:35.429643 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.423148 2578 flags.go:64] FLAG: --log-json-split-stream="false" Sep 17 07:27:35.429643 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.423153 2578 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 17 07:27:35.429643 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.423160 2578 flags.go:64] FLAG: --log-text-split-stream="false" Sep 17 07:27:35.429643 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.423165 2578 flags.go:64] FLAG: --logging-format="text" Sep 17 07:27:35.429643 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.423169 2578 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 17 07:27:35.429643 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.423175 2578 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 17 07:27:35.429643 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.423179 2578 flags.go:64] FLAG: --manifest-url="" Sep 17 07:27:35.429643 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.423201 2578 flags.go:64] FLAG: --manifest-url-header="" Sep 17 07:27:35.429643 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.423208 2578 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 17 07:27:35.429643 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.423213 2578 flags.go:64] FLAG: --max-open-files="1000000" Sep 17 07:27:35.429643 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.423219 2578 flags.go:64] FLAG: --max-pods="110" Sep 17 07:27:35.429643 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.423224 2578 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 17 07:27:35.429643 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.423229 2578 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 17 07:27:35.429643 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.423233 2578 flags.go:64] FLAG: --memory-manager-policy="None" Sep 17 07:27:35.429643 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.423238 2578 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 17 07:27:35.429643 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.423244 2578 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 17 07:27:35.429643 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.423250 2578 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 17 07:27:35.429643 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.423254 2578 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 17 07:27:35.429643 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.423266 2578 flags.go:64] FLAG: --node-status-max-images="50" Sep 17 07:27:35.430265 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.423271 2578 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 17 07:27:35.430265 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.423276 2578 flags.go:64] FLAG: --oom-score-adj="-999" Sep 17 07:27:35.430265 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.423281 2578 flags.go:64] FLAG: --pod-cidr="" Sep 17 07:27:35.430265 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.423286 2578 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10" Sep 17 07:27:35.430265 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.423295 2578 flags.go:64] FLAG: --pod-manifest-path="" Sep 17 07:27:35.430265 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.423300 2578 flags.go:64] FLAG: --pod-max-pids="-1" Sep 17 07:27:35.430265 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.423305 2578 flags.go:64] FLAG: --pods-per-core="0" Sep 17 07:27:35.430265 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.423309 2578 flags.go:64] FLAG: --port="10250" Sep 17 07:27:35.430265 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.423315 2578 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 17 07:27:35.430265 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.423319 2578 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-06a9ac78e339e5e0c" Sep 17 07:27:35.430265 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.423324 2578 flags.go:64] FLAG: --qos-reserved="" Sep 17 07:27:35.430265 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.423329 2578 flags.go:64] FLAG: --read-only-port="10255" Sep 17 07:27:35.430265 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.423334 2578 flags.go:64] FLAG: --register-node="true" Sep 17 07:27:35.430265 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.423339 2578 flags.go:64] FLAG: --register-with-taints="" Sep 17 07:27:35.430265 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.423345 2578 flags.go:64] FLAG: --registry-burst="10" Sep 17 07:27:35.430265 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.423351 2578 flags.go:64] FLAG: --registry-qps="5" Sep 17 07:27:35.430265 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.423358 2578 flags.go:64] FLAG: --reserved-cpus="" Sep 17 07:27:35.430265 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.423363 2578 flags.go:64] FLAG: --reserved-memory="" Sep 17 07:27:35.430265 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.423368 2578 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 17 07:27:35.430265 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.423373 2578 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 17 07:27:35.430265 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.423378 2578 flags.go:64] FLAG: --rotate-certificates="false" Sep 17 07:27:35.430265 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.423383 2578 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 17 07:27:35.430265 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.423388 2578 flags.go:64] FLAG: --runonce="false" Sep 17 07:27:35.430265 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.423393 2578 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 17 07:27:35.430265 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.423398 2578 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 17 07:27:35.430895 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.423403 2578 flags.go:64] FLAG: --seccomp-default="false" Sep 17 07:27:35.430895 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.423408 2578 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 17 07:27:35.430895 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.423412 2578 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 17 07:27:35.430895 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.423419 2578 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 17 07:27:35.430895 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.423423 2578 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 17 07:27:35.430895 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.423429 2578 flags.go:64] FLAG: --storage-driver-password="root" Sep 17 07:27:35.430895 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.423433 2578 flags.go:64] FLAG: --storage-driver-secure="false" Sep 17 07:27:35.430895 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.423438 2578 flags.go:64] FLAG: --storage-driver-table="stats" Sep 17 07:27:35.430895 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.423443 2578 flags.go:64] FLAG: --storage-driver-user="root" Sep 17 07:27:35.430895 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.423447 2578 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 17 07:27:35.430895 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.423452 2578 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 17 07:27:35.430895 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.423457 2578 flags.go:64] FLAG: --system-cgroups="" Sep 17 07:27:35.430895 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.423461 2578 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 17 07:27:35.430895 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.423470 2578 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 17 07:27:35.430895 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.423475 2578 flags.go:64] FLAG: --tls-cert-file="" Sep 17 07:27:35.430895 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.423479 2578 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 17 07:27:35.430895 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.423485 2578 flags.go:64] FLAG: --tls-min-version="" Sep 17 07:27:35.430895 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.423490 2578 flags.go:64] FLAG: --tls-private-key-file="" Sep 17 07:27:35.430895 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.423494 2578 flags.go:64] FLAG: --topology-manager-policy="none" Sep 17 07:27:35.430895 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.423499 2578 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 17 07:27:35.430895 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.423504 2578 flags.go:64] FLAG: --topology-manager-scope="container" Sep 17 07:27:35.430895 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.423508 2578 flags.go:64] FLAG: --v="2" Sep 17 07:27:35.430895 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.423515 2578 flags.go:64] FLAG: --version="false" Sep 17 07:27:35.430895 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.423525 2578 flags.go:64] FLAG: --vmodule="" Sep 17 07:27:35.430895 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.423531 2578 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 17 07:27:35.431583 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.423537 2578 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 17 07:27:35.431583 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.423724 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 17 07:27:35.431583 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.423732 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 17 07:27:35.431583 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.423736 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 17 07:27:35.431583 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.423741 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 17 07:27:35.431583 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.423746 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 17 07:27:35.431583 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.423750 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 17 07:27:35.431583 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.423755 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 17 07:27:35.431583 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.423759 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 17 07:27:35.431583 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.423764 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 17 07:27:35.431583 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.423769 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 17 07:27:35.431583 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.423774 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 17 07:27:35.431583 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.423778 2578 feature_gate.go:328] unrecognized feature gate: Example Sep 17 07:27:35.431583 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.423782 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 17 07:27:35.431583 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.423786 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 17 07:27:35.431583 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.423793 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 17 07:27:35.431583 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.423798 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 17 07:27:35.431583 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.423803 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 17 07:27:35.431583 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.423807 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 17 07:27:35.432047 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.423812 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 17 07:27:35.432047 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.423817 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 17 07:27:35.432047 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.423822 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 17 07:27:35.432047 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.423826 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 17 07:27:35.432047 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.423831 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 17 07:27:35.432047 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.423835 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 17 07:27:35.432047 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.423840 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 17 07:27:35.432047 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.423845 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 17 07:27:35.432047 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.423849 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 17 07:27:35.432047 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.423854 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 17 07:27:35.432047 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.423858 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 17 07:27:35.432047 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.423864 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 17 07:27:35.432047 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.423869 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 17 07:27:35.432047 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.423873 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 17 07:27:35.432047 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.423877 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 17 07:27:35.432047 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.423881 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 17 07:27:35.432047 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.423886 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 17 07:27:35.432047 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.423890 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 17 07:27:35.432047 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.423895 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 17 07:27:35.432530 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.423899 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 17 07:27:35.432530 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.423903 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 17 07:27:35.432530 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.423907 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 17 07:27:35.432530 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.423912 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 17 07:27:35.432530 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.423917 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 17 07:27:35.432530 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.423922 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 17 07:27:35.432530 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.423926 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 17 07:27:35.432530 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.423930 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 17 07:27:35.432530 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.423935 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 17 07:27:35.432530 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.423939 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 17 07:27:35.432530 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.423943 2578 feature_gate.go:328] unrecognized feature gate: Example2 Sep 17 07:27:35.432530 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.423947 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 17 07:27:35.432530 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.423952 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 17 07:27:35.432530 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.423957 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 17 07:27:35.432530 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.423963 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 17 07:27:35.432530 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.423969 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 17 07:27:35.432530 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.423974 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 17 07:27:35.432530 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.423979 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 17 07:27:35.432530 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.423983 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 17 07:27:35.432530 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.423987 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 17 07:27:35.433020 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.423991 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 17 07:27:35.433020 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.423996 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 17 07:27:35.433020 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.424000 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 17 07:27:35.433020 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.424004 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 17 07:27:35.433020 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.424010 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 17 07:27:35.433020 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.424014 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 17 07:27:35.433020 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.424020 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 17 07:27:35.433020 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.424024 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 17 07:27:35.433020 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.424028 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 17 07:27:35.433020 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.424033 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 17 07:27:35.433020 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.424037 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 17 07:27:35.433020 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.424041 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 17 07:27:35.433020 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.424046 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 17 07:27:35.433020 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.424050 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 17 07:27:35.433020 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.424054 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 17 07:27:35.433020 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.424059 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 17 07:27:35.433020 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.424065 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 17 07:27:35.433020 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.424069 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 17 07:27:35.433020 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.424073 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 17 07:27:35.433020 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.424077 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 17 07:27:35.433539 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.424082 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 17 07:27:35.433539 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.424086 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 17 07:27:35.433539 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.424090 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 17 07:27:35.433539 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.424094 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 17 07:27:35.433539 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.424098 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 17 07:27:35.433539 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.424102 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 17 07:27:35.433539 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.424106 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 17 07:27:35.433539 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.424110 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 17 07:27:35.433539 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.424114 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 17 07:27:35.433539 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.424119 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 17 07:27:35.433539 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.424123 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 17 07:27:35.433539 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.424127 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 17 07:27:35.433539 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.424131 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 17 07:27:35.433539 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.424135 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 17 07:27:35.433539 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.424139 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 17 07:27:35.433539 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.424143 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 17 07:27:35.433539 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.424149 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 17 07:27:35.433539 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.424153 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 17 07:27:35.433539 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.424158 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 17 07:27:35.433539 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.424163 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 17 07:27:35.434015 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.424170 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 17 07:27:35.434015 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.424175 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 17 07:27:35.434015 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.424955 2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 17 07:27:35.434015 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.432675 2578 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 17 07:27:35.434015 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.432689 2578 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 17 07:27:35.434015 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.432708 2578 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 17 07:27:35.434015 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.432714 2578 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 17 07:27:35.434015 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.432771 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 17 07:27:35.434015 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.432775 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 17 07:27:35.434015 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.432779 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 17 07:27:35.434015 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.432782 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 17 07:27:35.434015 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.432786 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 17 07:27:35.434015 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.432789 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 17 07:27:35.434015 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.432791 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 17 07:27:35.434015 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.432794 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 17 07:27:35.434439 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.432797 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 17 07:27:35.434439 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.432800 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 17 07:27:35.434439 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.432802 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 17 07:27:35.434439 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.432805 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 17 07:27:35.434439 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.432810 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 17 07:27:35.434439 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.432814 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 17 07:27:35.434439 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.432817 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 17 07:27:35.434439 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.432820 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 17 07:27:35.434439 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.432823 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 17 07:27:35.434439 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.432826 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 17 07:27:35.434439 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.432829 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 17 07:27:35.434439 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.432832 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 17 07:27:35.434439 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.432835 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 17 07:27:35.434439 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.432838 2578 feature_gate.go:328] unrecognized feature gate: Example2 Sep 17 07:27:35.434439 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.432840 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 17 07:27:35.434439 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.432843 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 17 07:27:35.434439 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.432846 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 17 07:27:35.434439 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.432849 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 17 07:27:35.434439 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.432852 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 17 07:27:35.434969 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.432854 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 17 07:27:35.434969 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.432857 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 17 07:27:35.434969 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.432860 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 17 07:27:35.434969 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.432863 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 17 07:27:35.434969 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.432866 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 17 07:27:35.434969 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.432869 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 17 07:27:35.434969 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.432872 2578 feature_gate.go:328] unrecognized feature gate: Example Sep 17 07:27:35.434969 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.432874 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 17 07:27:35.434969 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.432877 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 17 07:27:35.434969 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.432880 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 17 07:27:35.434969 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.432882 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 17 07:27:35.434969 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.432885 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 17 07:27:35.434969 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.432887 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 17 07:27:35.434969 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.432890 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 17 07:27:35.434969 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.432893 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 17 07:27:35.434969 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.432896 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 17 07:27:35.434969 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.432898 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 17 07:27:35.434969 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.432901 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 17 07:27:35.434969 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.432903 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 17 07:27:35.434969 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.432906 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 17 07:27:35.435465 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.432909 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 17 07:27:35.435465 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.432911 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 17 07:27:35.435465 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.432914 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 17 07:27:35.435465 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.432917 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 17 07:27:35.435465 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.432919 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 17 07:27:35.435465 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.432922 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 17 07:27:35.435465 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.432924 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 17 07:27:35.435465 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.432927 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 17 07:27:35.435465 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.432930 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 17 07:27:35.435465 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.432932 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 17 07:27:35.435465 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.432936 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 17 07:27:35.435465 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.432938 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 17 07:27:35.435465 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.432941 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 17 07:27:35.435465 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.432945 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 17 07:27:35.435465 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.432948 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 17 07:27:35.435465 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.432951 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 17 07:27:35.435465 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.432954 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 17 07:27:35.435465 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.432956 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 17 07:27:35.435465 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.432959 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 17 07:27:35.435914 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.432962 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 17 07:27:35.435914 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.432964 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 17 07:27:35.435914 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.432967 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 17 07:27:35.435914 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.432970 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 17 07:27:35.435914 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.432973 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 17 07:27:35.435914 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.432975 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 17 07:27:35.435914 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.432978 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 17 07:27:35.435914 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.432981 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 17 07:27:35.435914 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.432984 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 17 07:27:35.435914 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.432986 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 17 07:27:35.435914 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.432989 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 17 07:27:35.435914 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.432993 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 17 07:27:35.435914 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.432997 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 17 07:27:35.435914 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.433000 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 17 07:27:35.435914 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.433003 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 17 07:27:35.435914 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.433006 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 17 07:27:35.435914 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.433009 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 17 07:27:35.435914 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.433012 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 17 07:27:35.435914 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.433015 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 17 07:27:35.436449 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.433019 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 17 07:27:35.436449 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.433022 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 17 07:27:35.436449 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.433025 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 17 07:27:35.436449 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.433028 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 17 07:27:35.436449 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.433030 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 17 07:27:35.436449 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.433033 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 17 07:27:35.436449 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.433036 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 17 07:27:35.436449 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.433039 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 17 07:27:35.436449 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.433041 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 17 07:27:35.436449 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.433044 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 17 07:27:35.436449 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.433047 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 17 07:27:35.436449 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.433050 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 17 07:27:35.436449 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.433052 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 17 07:27:35.436449 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.433055 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 17 07:27:35.436449 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.433060 2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 17 07:27:35.436808 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.433206 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 17 07:27:35.436808 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.433211 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 17 07:27:35.436808 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.433214 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 17 07:27:35.436808 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.433217 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 17 07:27:35.436808 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.433220 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 17 07:27:35.436808 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.433223 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 17 07:27:35.436808 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.433226 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 17 07:27:35.436808 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.433229 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 17 07:27:35.436808 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.433232 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 17 07:27:35.436808 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.433235 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 17 07:27:35.436808 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.433237 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 17 07:27:35.436808 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.433240 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 17 07:27:35.436808 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.433242 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 17 07:27:35.436808 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.433245 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 17 07:27:35.436808 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.433248 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 17 07:27:35.436808 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.433250 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 17 07:27:35.436808 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.433254 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 17 07:27:35.436808 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.433258 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 17 07:27:35.436808 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.433261 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 17 07:27:35.436808 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.433264 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 17 07:27:35.437324 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.433267 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 17 07:27:35.437324 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.433269 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 17 07:27:35.437324 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.433272 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 17 07:27:35.437324 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.433275 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 17 07:27:35.437324 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.433279 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 17 07:27:35.437324 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.433282 2578 feature_gate.go:328] unrecognized feature gate: Example2 Sep 17 07:27:35.437324 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.433284 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 17 07:27:35.437324 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.433287 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 17 07:27:35.437324 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.433290 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 17 07:27:35.437324 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.433292 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 17 07:27:35.437324 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.433295 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 17 07:27:35.437324 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.433297 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 17 07:27:35.437324 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.433300 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 17 07:27:35.437324 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.433303 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 17 07:27:35.437324 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.433305 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 17 07:27:35.437324 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.433308 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 17 07:27:35.437324 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.433310 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 17 07:27:35.437324 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.433313 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 17 07:27:35.437324 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.433315 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 17 07:27:35.437764 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.433318 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 17 07:27:35.437764 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.433321 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 17 07:27:35.437764 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.433323 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 17 07:27:35.437764 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.433326 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 17 07:27:35.437764 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.433328 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 17 07:27:35.437764 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.433331 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 17 07:27:35.437764 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.433334 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 17 07:27:35.437764 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.433336 2578 feature_gate.go:328] unrecognized feature gate: Example Sep 17 07:27:35.437764 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.433339 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 17 07:27:35.437764 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.433342 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 17 07:27:35.437764 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.433344 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 17 07:27:35.437764 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.433348 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 17 07:27:35.437764 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.433351 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 17 07:27:35.437764 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.433354 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 17 07:27:35.437764 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.433356 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 17 07:27:35.437764 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.433359 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 17 07:27:35.437764 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.433361 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 17 07:27:35.437764 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.433364 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 17 07:27:35.437764 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.433367 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 17 07:27:35.437764 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.433370 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 17 07:27:35.438258 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.433373 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 17 07:27:35.438258 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.433375 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 17 07:27:35.438258 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.433378 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 17 07:27:35.438258 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.433380 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 17 07:27:35.438258 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.433383 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 17 07:27:35.438258 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.433386 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 17 07:27:35.438258 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.433388 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 17 07:27:35.438258 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.433391 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 17 07:27:35.438258 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.433394 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 17 07:27:35.438258 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.433396 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 17 07:27:35.438258 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.433399 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 17 07:27:35.438258 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.433401 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 17 07:27:35.438258 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.433404 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 17 07:27:35.438258 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.433406 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 17 07:27:35.438258 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.433409 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 17 07:27:35.438258 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.433412 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 17 07:27:35.438258 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.433414 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 17 07:27:35.438258 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.433417 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 17 07:27:35.438258 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.433420 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 17 07:27:35.438258 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.433422 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 17 07:27:35.438740 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.433427 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 17 07:27:35.438740 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.433430 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 17 07:27:35.438740 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.433434 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 17 07:27:35.438740 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.433436 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 17 07:27:35.438740 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.433439 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 17 07:27:35.438740 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.433442 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 17 07:27:35.438740 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.433445 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 17 07:27:35.438740 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.433447 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 17 07:27:35.438740 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.433450 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 17 07:27:35.438740 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.433452 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 17 07:27:35.438740 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.433455 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 17 07:27:35.438740 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.433458 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 17 07:27:35.438740 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.433460 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 17 07:27:35.438740 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.433463 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 17 07:27:35.438740 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.433466 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 17 07:27:35.438740 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.433468 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 17 07:27:35.438740 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.433471 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 17 07:27:35.438740 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.433473 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 17 07:27:35.438740 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.433476 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 17 07:27:35.439206 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:35.433479 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 17 07:27:35.439206 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.433483 2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 17 07:27:35.439206 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.433581 2578 server.go:962] "Client rotation is on, will bootstrap in background" Sep 17 07:27:35.439206 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.435819 2578 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 17 07:27:35.439206 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.436907 2578 server.go:1019] "Starting client certificate rotation" Sep 17 07:27:35.439206 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.437008 2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 17 07:27:35.439206 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.437045 2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 17 07:27:35.439992 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.439979 2578 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 17 07:27:35.441382 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.441359 2578 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 17 07:27:35.451700 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.451684 2578 log.go:25] "Validated CRI v1 runtime API" Sep 17 07:27:35.454792 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.454779 2578 log.go:25] "Validated CRI v1 image API" Sep 17 07:27:35.455586 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.455572 2578 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 17 07:27:35.460943 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.460922 2578 fs.go:135] Filesystem UUIDs: map[0a5bd8d9-3af9-4afa-8b77-40e92c5f1f4a:/dev/nvme0n1p4 65d6753b-ec27-4004-bbf9-0a0f4ee7c092:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Sep 17 07:27:35.461004 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.460941 2578 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 17 07:27:35.466959 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.466854 2578 manager.go:216] Machine: {Timestamp:2026-09-17 07:27:35.464632861 +0000 UTC m=+0.370341427 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100032 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec27925d194cd563e400d080b77efd39 SystemUUID:ec27925d-194c-d563-e400-d080b77efd39 BootID:1d194449-761d-4b0c-a2ab-f2e441ed5399 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:d5:2b:54:71 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:d5:2b:54:71 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:b6:f5:18:3e:93:84 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 17 07:27:35.466959 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.466954 2578 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 17 07:27:35.467075 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.467027 2578 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 17 07:27:35.471232 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.471197 2578 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 17 07:27:35.471365 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.471237 2578 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-100.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 17 07:27:35.472091 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.472082 2578 topology_manager.go:138] "Creating topology manager with none policy" Sep 17 07:27:35.472157 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.472093 2578 container_manager_linux.go:309] "Creating device plugin manager" Sep 17 07:27:35.472157 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.472101 2578 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 17 07:27:35.472804 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.472794 2578 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 17 07:27:35.473938 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.473928 2578 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 17 07:27:35.478251 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.478237 2578 state_mem.go:36] "Initialized new in-memory state store" Sep 17 07:27:35.478356 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.478346 2578 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 17 07:27:35.478409 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.478374 2578 kubelet.go:486] "Attempting to sync node with API server" Sep 17 07:27:35.478409 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.478383 2578 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 17 07:27:35.478409 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.478396 2578 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 17 07:27:35.478409 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.478404 2578 kubelet.go:398] "Adding apiserver pod source" Sep 17 07:27:35.478550 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.478417 2578 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 17 07:27:35.478830 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.478818 2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 17 07:27:35.478911 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.478835 2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 17 07:27:35.480639 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.480620 2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 17 07:27:35.483621 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.483605 2578 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 17 07:27:35.486826 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.486805 2578 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 17 07:27:35.486876 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.486857 2578 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 17 07:27:35.488780 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.488769 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 17 07:27:35.488813 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.488786 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 17 07:27:35.488813 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.488792 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 17 07:27:35.488813 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.488798 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 17 07:27:35.488813 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.488807 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 17 07:27:35.488914 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.488817 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 17 07:27:35.488914 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.488833 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 17 07:27:35.488914 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.488838 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 17 07:27:35.488914 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.488844 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 17 07:27:35.488914 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.488849 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 17 07:27:35.488914 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.488857 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 17 07:27:35.488914 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.488862 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 17 07:27:35.489755 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.489746 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 17 07:27:35.489755 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.489755 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 17 07:27:35.493565 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.493552 2578 server.go:1268] "Started kubelet" Sep 17 07:27:35.493680 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.493619 2578 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 17 07:27:35.493748 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.493667 2578 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 17 07:27:35.493796 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.493764 2578 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 17 07:27:35.494094 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.494068 2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-100.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 17 07:27:35.494173 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:27:35.494127 2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-100.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 17 07:27:35.494463 ip-10-0-139-100 systemd[1]: Started Kubernetes Kubelet. Sep 17 07:27:35.496518 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:27:35.496288 2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 17 07:27:35.496600 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.496563 2578 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 17 07:27:35.498632 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.498619 2578 server.go:310] "Adding debug handlers to kubelet server" Sep 17 07:27:35.499464 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:27:35.499430 2578 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 17 07:27:35.502297 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:27:35.501219 2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-100.ec2.internal.18d60b0db90dc9c4 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-100.ec2.internal,UID:ip-10-0-139-100.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-100.ec2.internal,},FirstTimestamp:2026-09-17 07:27:35.493519812 +0000 UTC m=+0.399228385,LastTimestamp:2026-09-17 07:27:35.493519812 +0000 UTC m=+0.399228385,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-100.ec2.internal,}" Sep 17 07:27:35.505122 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.505109 2578 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 17 07:27:35.505202 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.505137 2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 17 07:27:35.505463 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:27:35.505379 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-100.ec2.internal\" not found" Sep 17 07:27:35.505797 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.505783 2578 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 17 07:27:35.505882 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.505828 2578 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 17 07:27:35.505939 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.505888 2578 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 17 07:27:35.505992 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.505956 2578 reconstruct.go:97] "Volume reconstruction finished" Sep 17 07:27:35.505992 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.505967 2578 reconciler.go:29] "Reconciler: start to sync state" Sep 17 07:27:35.505992 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.505970 2578 factory.go:55] Registering systemd factory Sep 17 07:27:35.505992 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.505985 2578 factory.go:223] Registration of the systemd container factory successfully Sep 17 07:27:35.506345 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.506199 2578 factory.go:153] Registering CRI-O factory Sep 17 07:27:35.506413 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.506349 2578 factory.go:223] Registration of the crio container factory successfully Sep 17 07:27:35.506413 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.506400 2578 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 17 07:27:35.506525 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.506418 2578 factory.go:103] Registering Raw factory Sep 17 07:27:35.506525 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.506433 2578 manager.go:1196] Started watching for new ooms in manager Sep 17 07:27:35.506819 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.506804 2578 manager.go:349] Starting recovery of all containers Sep 17 07:27:35.515565 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:27:35.515538 2578 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 17 07:27:35.517040 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.517027 2578 manager.go:354] Recovery completed Sep 17 07:27:35.517503 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:27:35.517466 2578 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-139-100.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 17 07:27:35.520915 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.520902 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 17 07:27:35.522831 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.522798 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-100.ec2.internal" event="NodeHasSufficientMemory" Sep 17 07:27:35.522894 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.522848 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-100.ec2.internal" event="NodeHasNoDiskPressure" Sep 17 07:27:35.522894 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.522863 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-100.ec2.internal" event="NodeHasSufficientPID" Sep 17 07:27:35.523547 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.523472 2578 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 17 07:27:35.523547 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.523485 2578 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 17 07:27:35.523547 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.523504 2578 state_mem.go:36] "Initialized new in-memory state store" Sep 17 07:27:35.523754 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.523486 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 17 07:27:35.524258 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.524238 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-100.ec2.internal" event="NodeHasSufficientMemory" Sep 17 07:27:35.524349 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.524269 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-100.ec2.internal" event="NodeHasNoDiskPressure" Sep 17 07:27:35.524349 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.524279 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-100.ec2.internal" event="NodeHasSufficientPID" Sep 17 07:27:35.524581 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.524568 2578 policy_none.go:49] "None policy: Start" Sep 17 07:27:35.524635 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.524591 2578 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 17 07:27:35.524635 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.524606 2578 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 17 07:27:35.525909 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.525895 2578 policy_none.go:47] "Start" Sep 17 07:27:35.526472 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.526455 2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-t5ww9" Sep 17 07:27:35.535988 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:27:35.535907 2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-100.ec2.internal.18d60b0dbacd09d9 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-100.ec2.internal,UID:ip-10-0-139-100.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-139-100.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-139-100.ec2.internal,},FirstTimestamp:2026-09-17 07:27:35.522830809 +0000 UTC m=+0.428539376,LastTimestamp:2026-09-17 07:27:35.522830809 +0000 UTC m=+0.428539376,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-100.ec2.internal,}" Sep 17 07:27:35.536078 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.536005 2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-t5ww9" Sep 17 07:27:35.565898 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.563813 2578 manager.go:337] "Starting Device Plugin manager" Sep 17 07:27:35.565898 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:27:35.563844 2578 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 17 07:27:35.565898 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.563852 2578 server.go:87] "Starting device plugin registration server" Sep 17 07:27:35.565898 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.564129 2578 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 17 07:27:35.565898 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.564141 2578 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 17 07:27:35.565898 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.564245 2578 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 17 07:27:35.565898 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.564325 2578 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 17 07:27:35.565898 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.564334 2578 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 17 07:27:35.565898 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:27:35.564785 2578 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 17 07:27:35.565898 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:27:35.564829 2578 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-100.ec2.internal\" not found" Sep 17 07:27:35.597125 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.597104 2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 17 07:27:35.598287 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.598268 2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 17 07:27:35.598369 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.598321 2578 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 17 07:27:35.598369 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.598345 2578 kubelet.go:2448] "Starting kubelet main sync loop" Sep 17 07:27:35.598455 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:27:35.598386 2578 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 17 07:27:35.601324 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.601307 2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 17 07:27:35.664283 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.664214 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 17 07:27:35.668986 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.668968 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-100.ec2.internal" event="NodeHasSufficientMemory" Sep 17 07:27:35.669082 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.668996 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-100.ec2.internal" event="NodeHasNoDiskPressure" Sep 17 07:27:35.669082 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.669012 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-100.ec2.internal" event="NodeHasSufficientPID" Sep 17 07:27:35.669082 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.669037 2578 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-139-100.ec2.internal" Sep 17 07:27:35.695967 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.695946 2578 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-139-100.ec2.internal" Sep 17 07:27:35.695967 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:27:35.695966 2578 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-100.ec2.internal\": node \"ip-10-0-139-100.ec2.internal\" not found" Sep 17 07:27:35.699408 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.699377 2578 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-100.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-100.ec2.internal"] Sep 17 07:27:35.699486 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.699473 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 17 07:27:35.700298 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.700283 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-100.ec2.internal" event="NodeHasSufficientMemory" Sep 17 07:27:35.700411 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.700314 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-100.ec2.internal" event="NodeHasNoDiskPressure" Sep 17 07:27:35.700411 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.700325 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-100.ec2.internal" event="NodeHasSufficientPID" Sep 17 07:27:35.701580 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.701565 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 17 07:27:35.701715 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.701698 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-100.ec2.internal" Sep 17 07:27:35.701772 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.701733 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 17 07:27:35.702430 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.702413 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-100.ec2.internal" event="NodeHasSufficientMemory" Sep 17 07:27:35.702430 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.702424 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-100.ec2.internal" event="NodeHasSufficientMemory" Sep 17 07:27:35.702580 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.702441 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-100.ec2.internal" event="NodeHasNoDiskPressure" Sep 17 07:27:35.702580 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.702448 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-100.ec2.internal" event="NodeHasNoDiskPressure" Sep 17 07:27:35.702580 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.702453 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-100.ec2.internal" event="NodeHasSufficientPID" Sep 17 07:27:35.702580 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.702462 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-100.ec2.internal" event="NodeHasSufficientPID" Sep 17 07:27:35.703543 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.703526 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-100.ec2.internal" Sep 17 07:27:35.703612 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.703555 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 17 07:27:35.704219 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.704202 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-100.ec2.internal" event="NodeHasSufficientMemory" Sep 17 07:27:35.704300 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.704232 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-100.ec2.internal" event="NodeHasNoDiskPressure" Sep 17 07:27:35.704300 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.704244 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-100.ec2.internal" event="NodeHasSufficientPID" Sep 17 07:27:35.732779 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:27:35.732765 2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-100.ec2.internal\" not found" node="ip-10-0-139-100.ec2.internal" Sep 17 07:27:35.737082 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:27:35.737069 2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-100.ec2.internal\" not found" node="ip-10-0-139-100.ec2.internal" Sep 17 07:27:35.778765 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:27:35.778746 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-100.ec2.internal\" not found" Sep 17 07:27:35.807329 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.807309 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4f0d3305f8ce3ecc2367db4b48860b3b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-100.ec2.internal\" (UID: \"4f0d3305f8ce3ecc2367db4b48860b3b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-100.ec2.internal" Sep 17 07:27:35.807405 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.807332 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4f0d3305f8ce3ecc2367db4b48860b3b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-100.ec2.internal\" (UID: \"4f0d3305f8ce3ecc2367db4b48860b3b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-100.ec2.internal" Sep 17 07:27:35.807405 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.807351 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9f09f680a0274abd6edf499bc1ec7a29-config\") pod \"kube-apiserver-proxy-ip-10-0-139-100.ec2.internal\" (UID: \"9f09f680a0274abd6edf499bc1ec7a29\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-100.ec2.internal" Sep 17 07:27:35.879056 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:27:35.879035 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-100.ec2.internal\" not found" Sep 17 07:27:35.908340 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.908316 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4f0d3305f8ce3ecc2367db4b48860b3b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-100.ec2.internal\" (UID: \"4f0d3305f8ce3ecc2367db4b48860b3b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-100.ec2.internal" Sep 17 07:27:35.908439 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.908346 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4f0d3305f8ce3ecc2367db4b48860b3b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-100.ec2.internal\" (UID: \"4f0d3305f8ce3ecc2367db4b48860b3b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-100.ec2.internal" Sep 17 07:27:35.908439 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.908371 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9f09f680a0274abd6edf499bc1ec7a29-config\") pod \"kube-apiserver-proxy-ip-10-0-139-100.ec2.internal\" (UID: \"9f09f680a0274abd6edf499bc1ec7a29\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-100.ec2.internal" Sep 17 07:27:35.908439 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.908421 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4f0d3305f8ce3ecc2367db4b48860b3b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-100.ec2.internal\" (UID: \"4f0d3305f8ce3ecc2367db4b48860b3b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-100.ec2.internal" Sep 17 07:27:35.908439 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.908428 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4f0d3305f8ce3ecc2367db4b48860b3b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-100.ec2.internal\" (UID: \"4f0d3305f8ce3ecc2367db4b48860b3b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-100.ec2.internal" Sep 17 07:27:35.908581 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:35.908420 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9f09f680a0274abd6edf499bc1ec7a29-config\") pod \"kube-apiserver-proxy-ip-10-0-139-100.ec2.internal\" (UID: \"9f09f680a0274abd6edf499bc1ec7a29\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-100.ec2.internal" Sep 17 07:27:35.979513 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:27:35.979457 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-100.ec2.internal\" not found" Sep 17 07:27:36.035324 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:36.035296 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-100.ec2.internal" Sep 17 07:27:36.039379 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:36.039140 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-100.ec2.internal" Sep 17 07:27:36.079734 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:27:36.079715 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-100.ec2.internal\" not found" Sep 17 07:27:36.180306 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:27:36.180252 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-100.ec2.internal\" not found" Sep 17 07:27:36.280852 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:27:36.280776 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-100.ec2.internal\" not found" Sep 17 07:27:37.406020 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:27:37.405994 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-100.ec2.internal\" not found" Sep 17 07:27:37.461514 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:37.461495 2578 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 17 07:27:37.461934 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:37.461634 2578 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 17 07:27:37.489409 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:37.489388 2578 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 17 07:27:37.506730 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:27:37.506707 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-100.ec2.internal\" not found" Sep 17 07:27:37.566141 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:37.566091 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-16 07:22:36 +0000 UTC" deadline="2028-02-19 03:50:47.186908123 +0000 UTC" Sep 17 07:27:37.566141 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:37.566129 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12476h23m9.620783799s" Sep 17 07:27:37.572571 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:37.572541 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f09f680a0274abd6edf499bc1ec7a29.slice/crio-4267798a67a5296372ff0c312f7d864b45ab726284a30b9afec3fd820eb85bbd WatchSource:0}: Error finding container 4267798a67a5296372ff0c312f7d864b45ab726284a30b9afec3fd820eb85bbd: Status 404 returned error can't find the container with id 4267798a67a5296372ff0c312f7d864b45ab726284a30b9afec3fd820eb85bbd Sep 17 07:27:37.572953 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:37.572931 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f0d3305f8ce3ecc2367db4b48860b3b.slice/crio-3b7ced86b8c6d7cf8670b3fb37eef79d249e148b9c224793b8e4de8b534aaeb6 WatchSource:0}: Error finding container 3b7ced86b8c6d7cf8670b3fb37eef79d249e148b9c224793b8e4de8b534aaeb6: Status 404 returned error can't find the container with id 3b7ced86b8c6d7cf8670b3fb37eef79d249e148b9c224793b8e4de8b534aaeb6 Sep 17 07:27:37.577041 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:37.577026 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 17 07:27:37.607027 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:27:37.606993 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-100.ec2.internal\" not found" Sep 17 07:27:37.627253 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:37.627208 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-100.ec2.internal" event={"ID":"9f09f680a0274abd6edf499bc1ec7a29","Type":"ContainerStarted","Data":"4267798a67a5296372ff0c312f7d864b45ab726284a30b9afec3fd820eb85bbd"} Sep 17 07:27:37.628304 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:37.628285 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-100.ec2.internal" event={"ID":"4f0d3305f8ce3ecc2367db4b48860b3b","Type":"ContainerStarted","Data":"3b7ced86b8c6d7cf8670b3fb37eef79d249e148b9c224793b8e4de8b534aaeb6"} Sep 17 07:27:37.649730 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:37.649710 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 17 07:27:37.655232 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:37.655212 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-100.ec2.internal" event="NodeHasSufficientMemory" Sep 17 07:27:37.655317 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:37.655246 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-100.ec2.internal" event="NodeHasNoDiskPressure" Sep 17 07:27:37.655317 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:37.655261 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-100.ec2.internal" event="NodeHasSufficientPID" Sep 17 07:27:37.691991 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:37.691970 2578 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 17 07:27:37.730146 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:37.730128 2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-100.ec2.internal" Sep 17 07:27:37.743019 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:37.743002 2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 17 07:27:37.744572 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:37.744557 2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-100.ec2.internal" Sep 17 07:27:37.762166 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:37.762149 2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 17 07:27:38.084395 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.084364 2578 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 17 07:27:38.504137 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.504105 2578 apiserver.go:52] "Watching apiserver" Sep 17 07:27:38.513626 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.513364 2578 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 17 07:27:38.514251 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.514224 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-c8gq4","kube-system/global-pull-secret-syncer-lqcw7","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dtt5v","openshift-dns/node-resolver-xfggl","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-100.ec2.internal","openshift-multus/multus-additional-cni-plugins-lh5z8","openshift-multus/multus-zpwkj","openshift-network-diagnostics/network-check-target-bvxfq","openshift-ovn-kubernetes/ovnkube-node-hnzgv","kube-system/konnectivity-agent-qbfvv","kube-system/kube-apiserver-proxy-ip-10-0-139-100.ec2.internal","openshift-cluster-node-tuning-operator/tuned-wxwzn","openshift-image-registry/node-ca-fwdd5","openshift-multus/network-metrics-daemon-lnpjx"] Sep 17 07:27:38.515792 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.515769 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-zpwkj" Sep 17 07:27:38.517267 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.517247 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-hnzgv" Sep 17 07:27:38.518568 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.518547 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lnpjx" Sep 17 07:27:38.518657 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:27:38.518611 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lnpjx" podUID="737e0243-b8eb-47e4-bf0f-c6b31e6ba787" Sep 17 07:27:38.519018 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.518999 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 17 07:27:38.519288 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.519267 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-6h45q\"" Sep 17 07:27:38.519475 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.519461 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 17 07:27:38.519711 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.519693 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 17 07:27:38.519929 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.519907 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 17 07:27:38.520132 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.520093 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xfggl" Sep 17 07:27:38.521144 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.521126 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-wxwzn" Sep 17 07:27:38.522143 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.522126 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 17 07:27:38.522539 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.522518 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-fwdd5" Sep 17 07:27:38.523425 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.523275 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 17 07:27:38.524994 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.523958 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-lh5z8" Sep 17 07:27:38.525487 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.525388 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lqcw7" Sep 17 07:27:38.525487 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:27:38.525451 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lqcw7" podUID="a236d1bd-81b1-4c9e-a14f-997204ad1b98" Sep 17 07:27:38.526247 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.526225 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 17 07:27:38.526361 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.526334 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 17 07:27:38.526457 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.526436 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 17 07:27:38.526520 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.526454 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-rz6m9\"" Sep 17 07:27:38.528056 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.528036 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 17 07:27:38.528318 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.527855 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-qbfvv" Sep 17 07:27:38.529244 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.529227 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-c8gq4" Sep 17 07:27:38.529597 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.529580 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-222jg\"" Sep 17 07:27:38.530202 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.530184 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 17 07:27:38.530430 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.530416 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bvxfq" Sep 17 07:27:38.530495 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.530466 2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 17 07:27:38.530495 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:27:38.530470 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bvxfq" podUID="82555a36-653a-4106-a31e-b4da6445b838" Sep 17 07:27:38.530657 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.530644 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 17 07:27:38.530961 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.530939 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 17 07:27:38.532331 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.532311 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dtt5v" Sep 17 07:27:38.536151 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.536133 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-ldbgj\"" Sep 17 07:27:38.536411 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.536349 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 17 07:27:38.536513 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.536441 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-24fx6\"" Sep 17 07:27:38.536513 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.536467 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 17 07:27:38.536621 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.536552 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 17 07:27:38.536699 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.536678 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-fb6xq\"" Sep 17 07:27:38.536774 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.536742 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 17 07:27:38.536902 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.536886 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 17 07:27:38.536975 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.536930 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 17 07:27:38.537380 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.537127 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 17 07:27:38.537380 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.537248 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 17 07:27:38.537380 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.537264 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 17 07:27:38.537380 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.537271 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 17 07:27:38.537606 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.537564 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 17 07:27:38.537723 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.537707 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-kk6bm\"" Sep 17 07:27:38.537783 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.537770 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-wvj2d\"" Sep 17 07:27:38.541010 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.540991 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 17 07:27:38.541099 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.540996 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 17 07:27:38.541906 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.541886 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a236d1bd-81b1-4c9e-a14f-997204ad1b98-kubelet-config\") pod \"global-pull-secret-syncer-lqcw7\" (UID: \"a236d1bd-81b1-4c9e-a14f-997204ad1b98\") " pod="kube-system/global-pull-secret-syncer-lqcw7" Sep 17 07:27:38.542000 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.541917 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d2d890e8-ee86-4c3d-b7a7-81f5f57b3e43-device-dir\") pod \"aws-ebs-csi-driver-node-dtt5v\" (UID: \"d2d890e8-ee86-4c3d-b7a7-81f5f57b3e43\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dtt5v" Sep 17 07:27:38.542000 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.541939 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/60c84ae0-1da1-4845-8d12-d9656a247588-host-run-k8s-cni-cncf-io\") pod \"multus-zpwkj\" (UID: \"60c84ae0-1da1-4845-8d12-d9656a247588\") " pod="openshift-multus/multus-zpwkj" Sep 17 07:27:38.542000 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.541964 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/60c84ae0-1da1-4845-8d12-d9656a247588-host-var-lib-kubelet\") pod \"multus-zpwkj\" (UID: \"60c84ae0-1da1-4845-8d12-d9656a247588\") " pod="openshift-multus/multus-zpwkj" Sep 17 07:27:38.542000 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.541990 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/eeb1a9ba-ece4-4e09-8bc7-de92eb38f029-systemd-units\") pod \"ovnkube-node-hnzgv\" (UID: \"eeb1a9ba-ece4-4e09-8bc7-de92eb38f029\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnzgv" Sep 17 07:27:38.542188 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.542014 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/eeb1a9ba-ece4-4e09-8bc7-de92eb38f029-ovn-node-metrics-cert\") pod \"ovnkube-node-hnzgv\" (UID: \"eeb1a9ba-ece4-4e09-8bc7-de92eb38f029\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnzgv" Sep 17 07:27:38.542188 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.542036 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e9046265-84c2-44b5-9d4a-da21902077aa-lib-modules\") pod \"tuned-wxwzn\" (UID: \"e9046265-84c2-44b5-9d4a-da21902077aa\") " pod="openshift-cluster-node-tuning-operator/tuned-wxwzn" Sep 17 07:27:38.542188 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.542058 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7qqf\" (UniqueName: \"kubernetes.io/projected/e9046265-84c2-44b5-9d4a-da21902077aa-kube-api-access-j7qqf\") pod \"tuned-wxwzn\" (UID: \"e9046265-84c2-44b5-9d4a-da21902077aa\") " pod="openshift-cluster-node-tuning-operator/tuned-wxwzn" Sep 17 07:27:38.542188 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.542109 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d9df1c13-2108-44d8-9137-730e35cf2165-hosts-file\") pod \"node-resolver-xfggl\" (UID: \"d9df1c13-2108-44d8-9137-730e35cf2165\") " pod="openshift-dns/node-resolver-xfggl" Sep 17 07:27:38.542188 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.542164 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9df1c13-2108-44d8-9137-730e35cf2165-tmp-dir\") pod \"node-resolver-xfggl\" (UID: \"d9df1c13-2108-44d8-9137-730e35cf2165\") " pod="openshift-dns/node-resolver-xfggl" Sep 17 07:27:38.542188 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.542182 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-78z2g\"" Sep 17 07:27:38.542466 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.542191 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eeb1a9ba-ece4-4e09-8bc7-de92eb38f029-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hnzgv\" (UID: \"eeb1a9ba-ece4-4e09-8bc7-de92eb38f029\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnzgv" Sep 17 07:27:38.542466 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.542217 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e9046265-84c2-44b5-9d4a-da21902077aa-etc-sysconfig\") pod \"tuned-wxwzn\" (UID: \"e9046265-84c2-44b5-9d4a-da21902077aa\") " pod="openshift-cluster-node-tuning-operator/tuned-wxwzn" Sep 17 07:27:38.542466 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.542273 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnz4n\" (UniqueName: \"kubernetes.io/projected/82555a36-653a-4106-a31e-b4da6445b838-kube-api-access-gnz4n\") pod \"network-check-target-bvxfq\" (UID: \"82555a36-653a-4106-a31e-b4da6445b838\") " pod="openshift-network-diagnostics/network-check-target-bvxfq" Sep 17 07:27:38.542466 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.542300 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d2d890e8-ee86-4c3d-b7a7-81f5f57b3e43-registration-dir\") pod \"aws-ebs-csi-driver-node-dtt5v\" (UID: \"d2d890e8-ee86-4c3d-b7a7-81f5f57b3e43\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dtt5v" Sep 17 07:27:38.542466 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.542323 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/60c84ae0-1da1-4845-8d12-d9656a247588-multus-daemon-config\") pod \"multus-zpwkj\" (UID: \"60c84ae0-1da1-4845-8d12-d9656a247588\") " pod="openshift-multus/multus-zpwkj" Sep 17 07:27:38.542466 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.542347 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/60c84ae0-1da1-4845-8d12-d9656a247588-etc-kubernetes\") pod \"multus-zpwkj\" (UID: \"60c84ae0-1da1-4845-8d12-d9656a247588\") " pod="openshift-multus/multus-zpwkj" Sep 17 07:27:38.542466 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.542367 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/eeb1a9ba-ece4-4e09-8bc7-de92eb38f029-host-slash\") pod \"ovnkube-node-hnzgv\" (UID: \"eeb1a9ba-ece4-4e09-8bc7-de92eb38f029\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnzgv" Sep 17 07:27:38.542466 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.542388 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eeb1a9ba-ece4-4e09-8bc7-de92eb38f029-etc-openvswitch\") pod \"ovnkube-node-hnzgv\" (UID: \"eeb1a9ba-ece4-4e09-8bc7-de92eb38f029\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnzgv" Sep 17 07:27:38.542466 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.542428 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e9046265-84c2-44b5-9d4a-da21902077aa-etc-systemd\") pod \"tuned-wxwzn\" (UID: \"e9046265-84c2-44b5-9d4a-da21902077aa\") " pod="openshift-cluster-node-tuning-operator/tuned-wxwzn" Sep 17 07:27:38.542466 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.542451 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a236d1bd-81b1-4c9e-a14f-997204ad1b98-original-pull-secret\") pod \"global-pull-secret-syncer-lqcw7\" (UID: \"a236d1bd-81b1-4c9e-a14f-997204ad1b98\") " pod="kube-system/global-pull-secret-syncer-lqcw7" Sep 17 07:27:38.542958 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.542474 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5vw8\" (UniqueName: \"kubernetes.io/projected/d9df1c13-2108-44d8-9137-730e35cf2165-kube-api-access-f5vw8\") pod \"node-resolver-xfggl\" (UID: \"d9df1c13-2108-44d8-9137-730e35cf2165\") " pod="openshift-dns/node-resolver-xfggl" Sep 17 07:27:38.542958 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.542497 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/bfe3853f-127c-4124-8df0-cdd6439052bb-agent-certs\") pod \"konnectivity-agent-qbfvv\" (UID: \"bfe3853f-127c-4124-8df0-cdd6439052bb\") " pod="kube-system/konnectivity-agent-qbfvv" Sep 17 07:27:38.542958 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.542521 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c42682fc-ab24-4492-a27b-c03a53648e5b-iptables-alerter-script\") pod \"iptables-alerter-c8gq4\" (UID: \"c42682fc-ab24-4492-a27b-c03a53648e5b\") " pod="openshift-network-operator/iptables-alerter-c8gq4" Sep 17 07:27:38.542958 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.542557 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c42682fc-ab24-4492-a27b-c03a53648e5b-host-slash\") pod \"iptables-alerter-c8gq4\" (UID: \"c42682fc-ab24-4492-a27b-c03a53648e5b\") " pod="openshift-network-operator/iptables-alerter-c8gq4" Sep 17 07:27:38.542958 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.542579 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/eeb1a9ba-ece4-4e09-8bc7-de92eb38f029-host-run-netns\") pod \"ovnkube-node-hnzgv\" (UID: \"eeb1a9ba-ece4-4e09-8bc7-de92eb38f029\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnzgv" Sep 17 07:27:38.542958 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.542601 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/eeb1a9ba-ece4-4e09-8bc7-de92eb38f029-host-cni-bin\") pod \"ovnkube-node-hnzgv\" (UID: \"eeb1a9ba-ece4-4e09-8bc7-de92eb38f029\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnzgv" Sep 17 07:27:38.542958 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.542640 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e9046265-84c2-44b5-9d4a-da21902077aa-host\") pod \"tuned-wxwzn\" (UID: \"e9046265-84c2-44b5-9d4a-da21902077aa\") " pod="openshift-cluster-node-tuning-operator/tuned-wxwzn" Sep 17 07:27:38.542958 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.542687 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/60c84ae0-1da1-4845-8d12-d9656a247588-system-cni-dir\") pod \"multus-zpwkj\" (UID: \"60c84ae0-1da1-4845-8d12-d9656a247588\") " pod="openshift-multus/multus-zpwkj" Sep 17 07:27:38.542958 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.542712 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/60c84ae0-1da1-4845-8d12-d9656a247588-cnibin\") pod \"multus-zpwkj\" (UID: \"60c84ae0-1da1-4845-8d12-d9656a247588\") " pod="openshift-multus/multus-zpwkj" Sep 17 07:27:38.542958 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.542755 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/60c84ae0-1da1-4845-8d12-d9656a247588-cni-binary-copy\") pod \"multus-zpwkj\" (UID: \"60c84ae0-1da1-4845-8d12-d9656a247588\") " pod="openshift-multus/multus-zpwkj" Sep 17 07:27:38.542958 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.542776 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/60c84ae0-1da1-4845-8d12-d9656a247588-hostroot\") pod \"multus-zpwkj\" (UID: \"60c84ae0-1da1-4845-8d12-d9656a247588\") " pod="openshift-multus/multus-zpwkj" Sep 17 07:27:38.542958 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.542930 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5z7j\" (UniqueName: \"kubernetes.io/projected/60c84ae0-1da1-4845-8d12-d9656a247588-kube-api-access-v5z7j\") pod \"multus-zpwkj\" (UID: \"60c84ae0-1da1-4845-8d12-d9656a247588\") " pod="openshift-multus/multus-zpwkj" Sep 17 07:27:38.542958 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.542956 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d2d890e8-ee86-4c3d-b7a7-81f5f57b3e43-sys-fs\") pod \"aws-ebs-csi-driver-node-dtt5v\" (UID: \"d2d890e8-ee86-4c3d-b7a7-81f5f57b3e43\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dtt5v" Sep 17 07:27:38.543702 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.542977 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/60c84ae0-1da1-4845-8d12-d9656a247588-host-run-netns\") pod \"multus-zpwkj\" (UID: \"60c84ae0-1da1-4845-8d12-d9656a247588\") " pod="openshift-multus/multus-zpwkj" Sep 17 07:27:38.543702 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.543011 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/eeb1a9ba-ece4-4e09-8bc7-de92eb38f029-host-kubelet\") pod \"ovnkube-node-hnzgv\" (UID: \"eeb1a9ba-ece4-4e09-8bc7-de92eb38f029\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnzgv" Sep 17 07:27:38.543702 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.543039 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6642bc78-77e5-44af-af76-a8f87d25c89a-cni-binary-copy\") pod \"multus-additional-cni-plugins-lh5z8\" (UID: \"6642bc78-77e5-44af-af76-a8f87d25c89a\") " pod="openshift-multus/multus-additional-cni-plugins-lh5z8" Sep 17 07:27:38.543702 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.543070 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/60c84ae0-1da1-4845-8d12-d9656a247588-multus-cni-dir\") pod \"multus-zpwkj\" (UID: \"60c84ae0-1da1-4845-8d12-d9656a247588\") " pod="openshift-multus/multus-zpwkj" Sep 17 07:27:38.543702 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.543098 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/eeb1a9ba-ece4-4e09-8bc7-de92eb38f029-node-log\") pod \"ovnkube-node-hnzgv\" (UID: \"eeb1a9ba-ece4-4e09-8bc7-de92eb38f029\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnzgv" Sep 17 07:27:38.543702 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.543122 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/eeb1a9ba-ece4-4e09-8bc7-de92eb38f029-log-socket\") pod \"ovnkube-node-hnzgv\" (UID: \"eeb1a9ba-ece4-4e09-8bc7-de92eb38f029\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnzgv" Sep 17 07:27:38.543702 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.543165 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eeb1a9ba-ece4-4e09-8bc7-de92eb38f029-host-run-ovn-kubernetes\") pod \"ovnkube-node-hnzgv\" (UID: \"eeb1a9ba-ece4-4e09-8bc7-de92eb38f029\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnzgv" Sep 17 07:27:38.543702 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.543192 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjmvw\" (UniqueName: \"kubernetes.io/projected/eeb1a9ba-ece4-4e09-8bc7-de92eb38f029-kube-api-access-kjmvw\") pod \"ovnkube-node-hnzgv\" (UID: \"eeb1a9ba-ece4-4e09-8bc7-de92eb38f029\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnzgv" Sep 17 07:27:38.543702 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.543245 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e9046265-84c2-44b5-9d4a-da21902077aa-sys\") pod \"tuned-wxwzn\" (UID: \"e9046265-84c2-44b5-9d4a-da21902077aa\") " pod="openshift-cluster-node-tuning-operator/tuned-wxwzn" Sep 17 07:27:38.543702 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.543262 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6642bc78-77e5-44af-af76-a8f87d25c89a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lh5z8\" (UID: \"6642bc78-77e5-44af-af76-a8f87d25c89a\") " pod="openshift-multus/multus-additional-cni-plugins-lh5z8" Sep 17 07:27:38.543702 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.543294 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a236d1bd-81b1-4c9e-a14f-997204ad1b98-dbus\") pod \"global-pull-secret-syncer-lqcw7\" (UID: \"a236d1bd-81b1-4c9e-a14f-997204ad1b98\") " pod="kube-system/global-pull-secret-syncer-lqcw7" Sep 17 07:27:38.543702 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.543322 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/737e0243-b8eb-47e4-bf0f-c6b31e6ba787-metrics-certs\") pod \"network-metrics-daemon-lnpjx\" (UID: \"737e0243-b8eb-47e4-bf0f-c6b31e6ba787\") " pod="openshift-multus/network-metrics-daemon-lnpjx" Sep 17 07:27:38.543702 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.543356 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d2d890e8-ee86-4c3d-b7a7-81f5f57b3e43-socket-dir\") pod \"aws-ebs-csi-driver-node-dtt5v\" (UID: \"d2d890e8-ee86-4c3d-b7a7-81f5f57b3e43\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dtt5v" Sep 17 07:27:38.543702 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.543379 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d2d890e8-ee86-4c3d-b7a7-81f5f57b3e43-etc-selinux\") pod \"aws-ebs-csi-driver-node-dtt5v\" (UID: \"d2d890e8-ee86-4c3d-b7a7-81f5f57b3e43\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dtt5v" Sep 17 07:27:38.543702 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.543427 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eeb1a9ba-ece4-4e09-8bc7-de92eb38f029-var-lib-openvswitch\") pod \"ovnkube-node-hnzgv\" (UID: \"eeb1a9ba-ece4-4e09-8bc7-de92eb38f029\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnzgv" Sep 17 07:27:38.543702 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.543449 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e9046265-84c2-44b5-9d4a-da21902077aa-var-lib-kubelet\") pod \"tuned-wxwzn\" (UID: \"e9046265-84c2-44b5-9d4a-da21902077aa\") " pod="openshift-cluster-node-tuning-operator/tuned-wxwzn" Sep 17 07:27:38.544451 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.543475 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6642bc78-77e5-44af-af76-a8f87d25c89a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lh5z8\" (UID: \"6642bc78-77e5-44af-af76-a8f87d25c89a\") " pod="openshift-multus/multus-additional-cni-plugins-lh5z8" Sep 17 07:27:38.544451 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.543500 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6642bc78-77e5-44af-af76-a8f87d25c89a-system-cni-dir\") pod \"multus-additional-cni-plugins-lh5z8\" (UID: \"6642bc78-77e5-44af-af76-a8f87d25c89a\") " pod="openshift-multus/multus-additional-cni-plugins-lh5z8" Sep 17 07:27:38.544451 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.543523 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2v6lb\" (UniqueName: \"kubernetes.io/projected/6642bc78-77e5-44af-af76-a8f87d25c89a-kube-api-access-2v6lb\") pod \"multus-additional-cni-plugins-lh5z8\" (UID: \"6642bc78-77e5-44af-af76-a8f87d25c89a\") " pod="openshift-multus/multus-additional-cni-plugins-lh5z8" Sep 17 07:27:38.544451 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.543546 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f50cb832-b497-4a6a-bbe6-b758ec939688-host\") pod \"node-ca-fwdd5\" (UID: \"f50cb832-b497-4a6a-bbe6-b758ec939688\") " pod="openshift-image-registry/node-ca-fwdd5" Sep 17 07:27:38.544451 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.543570 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92f4v\" (UniqueName: \"kubernetes.io/projected/f50cb832-b497-4a6a-bbe6-b758ec939688-kube-api-access-92f4v\") pod \"node-ca-fwdd5\" (UID: \"f50cb832-b497-4a6a-bbe6-b758ec939688\") " pod="openshift-image-registry/node-ca-fwdd5" Sep 17 07:27:38.544451 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.543597 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/eeb1a9ba-ece4-4e09-8bc7-de92eb38f029-env-overrides\") pod \"ovnkube-node-hnzgv\" (UID: \"eeb1a9ba-ece4-4e09-8bc7-de92eb38f029\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnzgv" Sep 17 07:27:38.544451 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.543621 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e9046265-84c2-44b5-9d4a-da21902077aa-etc-modprobe-d\") pod \"tuned-wxwzn\" (UID: \"e9046265-84c2-44b5-9d4a-da21902077aa\") " pod="openshift-cluster-node-tuning-operator/tuned-wxwzn" Sep 17 07:27:38.544451 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.543643 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e9046265-84c2-44b5-9d4a-da21902077aa-etc-tuned\") pod \"tuned-wxwzn\" (UID: \"e9046265-84c2-44b5-9d4a-da21902077aa\") " pod="openshift-cluster-node-tuning-operator/tuned-wxwzn" Sep 17 07:27:38.544451 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.543663 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f50cb832-b497-4a6a-bbe6-b758ec939688-serviceca\") pod \"node-ca-fwdd5\" (UID: \"f50cb832-b497-4a6a-bbe6-b758ec939688\") " pod="openshift-image-registry/node-ca-fwdd5" Sep 17 07:27:38.544451 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.543682 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/60c84ae0-1da1-4845-8d12-d9656a247588-multus-conf-dir\") pod \"multus-zpwkj\" (UID: \"60c84ae0-1da1-4845-8d12-d9656a247588\") " pod="openshift-multus/multus-zpwkj" Sep 17 07:27:38.544451 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.543730 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eeb1a9ba-ece4-4e09-8bc7-de92eb38f029-run-openvswitch\") pod \"ovnkube-node-hnzgv\" (UID: \"eeb1a9ba-ece4-4e09-8bc7-de92eb38f029\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnzgv" Sep 17 07:27:38.544451 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.543759 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/eeb1a9ba-ece4-4e09-8bc7-de92eb38f029-ovnkube-config\") pod \"ovnkube-node-hnzgv\" (UID: \"eeb1a9ba-ece4-4e09-8bc7-de92eb38f029\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnzgv" Sep 17 07:27:38.544451 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.543783 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e9046265-84c2-44b5-9d4a-da21902077aa-etc-sysctl-d\") pod \"tuned-wxwzn\" (UID: \"e9046265-84c2-44b5-9d4a-da21902077aa\") " pod="openshift-cluster-node-tuning-operator/tuned-wxwzn" Sep 17 07:27:38.544451 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.543806 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/60c84ae0-1da1-4845-8d12-d9656a247588-multus-socket-dir-parent\") pod \"multus-zpwkj\" (UID: \"60c84ae0-1da1-4845-8d12-d9656a247588\") " pod="openshift-multus/multus-zpwkj" Sep 17 07:27:38.544451 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.543828 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/60c84ae0-1da1-4845-8d12-d9656a247588-host-run-multus-certs\") pod \"multus-zpwkj\" (UID: \"60c84ae0-1da1-4845-8d12-d9656a247588\") " pod="openshift-multus/multus-zpwkj" Sep 17 07:27:38.544451 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.543882 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/eeb1a9ba-ece4-4e09-8bc7-de92eb38f029-run-systemd\") pod \"ovnkube-node-hnzgv\" (UID: \"eeb1a9ba-ece4-4e09-8bc7-de92eb38f029\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnzgv" Sep 17 07:27:38.545269 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.543908 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/60c84ae0-1da1-4845-8d12-d9656a247588-host-var-lib-cni-bin\") pod \"multus-zpwkj\" (UID: \"60c84ae0-1da1-4845-8d12-d9656a247588\") " pod="openshift-multus/multus-zpwkj" Sep 17 07:27:38.545269 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.543932 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/60c84ae0-1da1-4845-8d12-d9656a247588-host-var-lib-cni-multus\") pod \"multus-zpwkj\" (UID: \"60c84ae0-1da1-4845-8d12-d9656a247588\") " pod="openshift-multus/multus-zpwkj" Sep 17 07:27:38.545269 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.543947 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e9046265-84c2-44b5-9d4a-da21902077aa-run\") pod \"tuned-wxwzn\" (UID: \"e9046265-84c2-44b5-9d4a-da21902077aa\") " pod="openshift-cluster-node-tuning-operator/tuned-wxwzn" Sep 17 07:27:38.545269 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.543961 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6642bc78-77e5-44af-af76-a8f87d25c89a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lh5z8\" (UID: \"6642bc78-77e5-44af-af76-a8f87d25c89a\") " pod="openshift-multus/multus-additional-cni-plugins-lh5z8" Sep 17 07:27:38.545269 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.543982 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpnwq\" (UniqueName: \"kubernetes.io/projected/737e0243-b8eb-47e4-bf0f-c6b31e6ba787-kube-api-access-tpnwq\") pod \"network-metrics-daemon-lnpjx\" (UID: \"737e0243-b8eb-47e4-bf0f-c6b31e6ba787\") " pod="openshift-multus/network-metrics-daemon-lnpjx" Sep 17 07:27:38.545269 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.544010 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/bfe3853f-127c-4124-8df0-cdd6439052bb-konnectivity-ca\") pod \"konnectivity-agent-qbfvv\" (UID: \"bfe3853f-127c-4124-8df0-cdd6439052bb\") " pod="kube-system/konnectivity-agent-qbfvv" Sep 17 07:27:38.545269 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.544031 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/eeb1a9ba-ece4-4e09-8bc7-de92eb38f029-run-ovn\") pod \"ovnkube-node-hnzgv\" (UID: \"eeb1a9ba-ece4-4e09-8bc7-de92eb38f029\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnzgv" Sep 17 07:27:38.545269 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.544071 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/eeb1a9ba-ece4-4e09-8bc7-de92eb38f029-host-cni-netd\") pod \"ovnkube-node-hnzgv\" (UID: \"eeb1a9ba-ece4-4e09-8bc7-de92eb38f029\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnzgv" Sep 17 07:27:38.545269 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.544092 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e9046265-84c2-44b5-9d4a-da21902077aa-etc-sysctl-conf\") pod \"tuned-wxwzn\" (UID: \"e9046265-84c2-44b5-9d4a-da21902077aa\") " pod="openshift-cluster-node-tuning-operator/tuned-wxwzn" Sep 17 07:27:38.545269 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.544112 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e9046265-84c2-44b5-9d4a-da21902077aa-tmp\") pod \"tuned-wxwzn\" (UID: \"e9046265-84c2-44b5-9d4a-da21902077aa\") " pod="openshift-cluster-node-tuning-operator/tuned-wxwzn" Sep 17 07:27:38.545269 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.544150 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6642bc78-77e5-44af-af76-a8f87d25c89a-cnibin\") pod \"multus-additional-cni-plugins-lh5z8\" (UID: \"6642bc78-77e5-44af-af76-a8f87d25c89a\") " pod="openshift-multus/multus-additional-cni-plugins-lh5z8" Sep 17 07:27:38.545269 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.544173 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d2d890e8-ee86-4c3d-b7a7-81f5f57b3e43-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dtt5v\" (UID: \"d2d890e8-ee86-4c3d-b7a7-81f5f57b3e43\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dtt5v" Sep 17 07:27:38.545269 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.544194 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p66qq\" (UniqueName: \"kubernetes.io/projected/d2d890e8-ee86-4c3d-b7a7-81f5f57b3e43-kube-api-access-p66qq\") pod \"aws-ebs-csi-driver-node-dtt5v\" (UID: \"d2d890e8-ee86-4c3d-b7a7-81f5f57b3e43\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dtt5v" Sep 17 07:27:38.545269 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.544216 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnl6r\" (UniqueName: \"kubernetes.io/projected/c42682fc-ab24-4492-a27b-c03a53648e5b-kube-api-access-pnl6r\") pod \"iptables-alerter-c8gq4\" (UID: \"c42682fc-ab24-4492-a27b-c03a53648e5b\") " pod="openshift-network-operator/iptables-alerter-c8gq4" Sep 17 07:27:38.545269 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.544231 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 17 07:27:38.545269 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.544254 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/eeb1a9ba-ece4-4e09-8bc7-de92eb38f029-ovnkube-script-lib\") pod \"ovnkube-node-hnzgv\" (UID: \"eeb1a9ba-ece4-4e09-8bc7-de92eb38f029\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnzgv" Sep 17 07:27:38.545269 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.544277 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e9046265-84c2-44b5-9d4a-da21902077aa-etc-kubernetes\") pod \"tuned-wxwzn\" (UID: \"e9046265-84c2-44b5-9d4a-da21902077aa\") " pod="openshift-cluster-node-tuning-operator/tuned-wxwzn" Sep 17 07:27:38.549131 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.548950 2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 17 07:27:38.586323 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.586303 2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-p9prp" Sep 17 07:27:38.587968 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.587947 2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 17 07:27:38.596093 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.596073 2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-p9prp" Sep 17 07:27:38.631184 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.631165 2578 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 17 07:27:38.644527 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.644503 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a236d1bd-81b1-4c9e-a14f-997204ad1b98-original-pull-secret\") pod \"global-pull-secret-syncer-lqcw7\" (UID: \"a236d1bd-81b1-4c9e-a14f-997204ad1b98\") " pod="kube-system/global-pull-secret-syncer-lqcw7" Sep 17 07:27:38.644643 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.644537 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5vw8\" (UniqueName: \"kubernetes.io/projected/d9df1c13-2108-44d8-9137-730e35cf2165-kube-api-access-f5vw8\") pod \"node-resolver-xfggl\" (UID: \"d9df1c13-2108-44d8-9137-730e35cf2165\") " pod="openshift-dns/node-resolver-xfggl" Sep 17 07:27:38.644643 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.644562 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/bfe3853f-127c-4124-8df0-cdd6439052bb-agent-certs\") pod \"konnectivity-agent-qbfvv\" (UID: \"bfe3853f-127c-4124-8df0-cdd6439052bb\") " pod="kube-system/konnectivity-agent-qbfvv" Sep 17 07:27:38.644643 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.644584 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c42682fc-ab24-4492-a27b-c03a53648e5b-iptables-alerter-script\") pod \"iptables-alerter-c8gq4\" (UID: \"c42682fc-ab24-4492-a27b-c03a53648e5b\") " pod="openshift-network-operator/iptables-alerter-c8gq4" Sep 17 07:27:38.644643 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.644607 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c42682fc-ab24-4492-a27b-c03a53648e5b-host-slash\") pod \"iptables-alerter-c8gq4\" (UID: \"c42682fc-ab24-4492-a27b-c03a53648e5b\") " pod="openshift-network-operator/iptables-alerter-c8gq4" Sep 17 07:27:38.644643 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.644629 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/eeb1a9ba-ece4-4e09-8bc7-de92eb38f029-host-run-netns\") pod \"ovnkube-node-hnzgv\" (UID: \"eeb1a9ba-ece4-4e09-8bc7-de92eb38f029\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnzgv" Sep 17 07:27:38.644643 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:27:38.644640 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 17 07:27:38.644958 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.644675 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/eeb1a9ba-ece4-4e09-8bc7-de92eb38f029-host-cni-bin\") pod \"ovnkube-node-hnzgv\" (UID: \"eeb1a9ba-ece4-4e09-8bc7-de92eb38f029\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnzgv" Sep 17 07:27:38.644958 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:27:38.644714 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a236d1bd-81b1-4c9e-a14f-997204ad1b98-original-pull-secret podName:a236d1bd-81b1-4c9e-a14f-997204ad1b98 nodeName:}" failed. No retries permitted until 2026-09-17 07:27:39.144680507 +0000 UTC m=+3.025714857 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a236d1bd-81b1-4c9e-a14f-997204ad1b98-original-pull-secret") pod "global-pull-secret-syncer-lqcw7" (UID: "a236d1bd-81b1-4c9e-a14f-997204ad1b98") : object "kube-system"/"original-pull-secret" not registered Sep 17 07:27:38.644958 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.644728 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/eeb1a9ba-ece4-4e09-8bc7-de92eb38f029-host-cni-bin\") pod \"ovnkube-node-hnzgv\" (UID: \"eeb1a9ba-ece4-4e09-8bc7-de92eb38f029\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnzgv" Sep 17 07:27:38.644958 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.644742 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e9046265-84c2-44b5-9d4a-da21902077aa-host\") pod \"tuned-wxwzn\" (UID: \"e9046265-84c2-44b5-9d4a-da21902077aa\") " pod="openshift-cluster-node-tuning-operator/tuned-wxwzn" Sep 17 07:27:38.644958 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.644766 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/60c84ae0-1da1-4845-8d12-d9656a247588-system-cni-dir\") pod \"multus-zpwkj\" (UID: \"60c84ae0-1da1-4845-8d12-d9656a247588\") " pod="openshift-multus/multus-zpwkj" Sep 17 07:27:38.644958 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.644789 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/60c84ae0-1da1-4845-8d12-d9656a247588-cnibin\") pod \"multus-zpwkj\" (UID: \"60c84ae0-1da1-4845-8d12-d9656a247588\") " pod="openshift-multus/multus-zpwkj" Sep 17 07:27:38.644958 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.644791 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c42682fc-ab24-4492-a27b-c03a53648e5b-host-slash\") pod \"iptables-alerter-c8gq4\" (UID: \"c42682fc-ab24-4492-a27b-c03a53648e5b\") " pod="openshift-network-operator/iptables-alerter-c8gq4" Sep 17 07:27:38.644958 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.644812 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/60c84ae0-1da1-4845-8d12-d9656a247588-cni-binary-copy\") pod \"multus-zpwkj\" (UID: \"60c84ae0-1da1-4845-8d12-d9656a247588\") " pod="openshift-multus/multus-zpwkj" Sep 17 07:27:38.644958 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.644836 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/60c84ae0-1da1-4845-8d12-d9656a247588-hostroot\") pod \"multus-zpwkj\" (UID: \"60c84ae0-1da1-4845-8d12-d9656a247588\") " pod="openshift-multus/multus-zpwkj" Sep 17 07:27:38.644958 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.644876 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5z7j\" (UniqueName: \"kubernetes.io/projected/60c84ae0-1da1-4845-8d12-d9656a247588-kube-api-access-v5z7j\") pod \"multus-zpwkj\" (UID: \"60c84ae0-1da1-4845-8d12-d9656a247588\") " pod="openshift-multus/multus-zpwkj" Sep 17 07:27:38.644958 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.644901 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d2d890e8-ee86-4c3d-b7a7-81f5f57b3e43-sys-fs\") pod \"aws-ebs-csi-driver-node-dtt5v\" (UID: \"d2d890e8-ee86-4c3d-b7a7-81f5f57b3e43\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dtt5v" Sep 17 07:27:38.644958 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.644914 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/60c84ae0-1da1-4845-8d12-d9656a247588-cnibin\") pod \"multus-zpwkj\" (UID: \"60c84ae0-1da1-4845-8d12-d9656a247588\") " pod="openshift-multus/multus-zpwkj" Sep 17 07:27:38.644958 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.644942 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/60c84ae0-1da1-4845-8d12-d9656a247588-host-run-netns\") pod \"multus-zpwkj\" (UID: \"60c84ae0-1da1-4845-8d12-d9656a247588\") " pod="openshift-multus/multus-zpwkj" Sep 17 07:27:38.644958 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.644964 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/eeb1a9ba-ece4-4e09-8bc7-de92eb38f029-host-kubelet\") pod \"ovnkube-node-hnzgv\" (UID: \"eeb1a9ba-ece4-4e09-8bc7-de92eb38f029\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnzgv" Sep 17 07:27:38.645661 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.644989 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6642bc78-77e5-44af-af76-a8f87d25c89a-cni-binary-copy\") pod \"multus-additional-cni-plugins-lh5z8\" (UID: \"6642bc78-77e5-44af-af76-a8f87d25c89a\") " pod="openshift-multus/multus-additional-cni-plugins-lh5z8" Sep 17 07:27:38.645661 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.645013 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/60c84ae0-1da1-4845-8d12-d9656a247588-multus-cni-dir\") pod \"multus-zpwkj\" (UID: \"60c84ae0-1da1-4845-8d12-d9656a247588\") " pod="openshift-multus/multus-zpwkj" Sep 17 07:27:38.645661 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.645036 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/eeb1a9ba-ece4-4e09-8bc7-de92eb38f029-node-log\") pod \"ovnkube-node-hnzgv\" (UID: \"eeb1a9ba-ece4-4e09-8bc7-de92eb38f029\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnzgv" Sep 17 07:27:38.645661 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.645047 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/60c84ae0-1da1-4845-8d12-d9656a247588-system-cni-dir\") pod \"multus-zpwkj\" (UID: \"60c84ae0-1da1-4845-8d12-d9656a247588\") " pod="openshift-multus/multus-zpwkj" Sep 17 07:27:38.645661 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.645059 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/eeb1a9ba-ece4-4e09-8bc7-de92eb38f029-log-socket\") pod \"ovnkube-node-hnzgv\" (UID: \"eeb1a9ba-ece4-4e09-8bc7-de92eb38f029\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnzgv" Sep 17 07:27:38.645661 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.645083 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eeb1a9ba-ece4-4e09-8bc7-de92eb38f029-host-run-ovn-kubernetes\") pod \"ovnkube-node-hnzgv\" (UID: \"eeb1a9ba-ece4-4e09-8bc7-de92eb38f029\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnzgv" Sep 17 07:27:38.645661 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.645093 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/60c84ae0-1da1-4845-8d12-d9656a247588-host-run-netns\") pod \"multus-zpwkj\" (UID: \"60c84ae0-1da1-4845-8d12-d9656a247588\") " pod="openshift-multus/multus-zpwkj" Sep 17 07:27:38.645661 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.645110 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjmvw\" (UniqueName: \"kubernetes.io/projected/eeb1a9ba-ece4-4e09-8bc7-de92eb38f029-kube-api-access-kjmvw\") pod \"ovnkube-node-hnzgv\" (UID: \"eeb1a9ba-ece4-4e09-8bc7-de92eb38f029\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnzgv" Sep 17 07:27:38.645661 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.645186 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c42682fc-ab24-4492-a27b-c03a53648e5b-iptables-alerter-script\") pod \"iptables-alerter-c8gq4\" (UID: \"c42682fc-ab24-4492-a27b-c03a53648e5b\") " pod="openshift-network-operator/iptables-alerter-c8gq4" Sep 17 07:27:38.645661 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.645273 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/60c84ae0-1da1-4845-8d12-d9656a247588-hostroot\") pod \"multus-zpwkj\" (UID: \"60c84ae0-1da1-4845-8d12-d9656a247588\") " pod="openshift-multus/multus-zpwkj" Sep 17 07:27:38.645661 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.645346 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/60c84ae0-1da1-4845-8d12-d9656a247588-multus-cni-dir\") pod \"multus-zpwkj\" (UID: \"60c84ae0-1da1-4845-8d12-d9656a247588\") " pod="openshift-multus/multus-zpwkj" Sep 17 07:27:38.645661 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.645414 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d2d890e8-ee86-4c3d-b7a7-81f5f57b3e43-sys-fs\") pod \"aws-ebs-csi-driver-node-dtt5v\" (UID: \"d2d890e8-ee86-4c3d-b7a7-81f5f57b3e43\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dtt5v" Sep 17 07:27:38.645661 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.645465 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/eeb1a9ba-ece4-4e09-8bc7-de92eb38f029-node-log\") pod \"ovnkube-node-hnzgv\" (UID: \"eeb1a9ba-ece4-4e09-8bc7-de92eb38f029\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnzgv" Sep 17 07:27:38.645661 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.645471 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/eeb1a9ba-ece4-4e09-8bc7-de92eb38f029-log-socket\") pod \"ovnkube-node-hnzgv\" (UID: \"eeb1a9ba-ece4-4e09-8bc7-de92eb38f029\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnzgv" Sep 17 07:27:38.645661 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.645467 2578 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 17 07:27:38.645661 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.645613 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/60c84ae0-1da1-4845-8d12-d9656a247588-cni-binary-copy\") pod \"multus-zpwkj\" (UID: \"60c84ae0-1da1-4845-8d12-d9656a247588\") " pod="openshift-multus/multus-zpwkj" Sep 17 07:27:38.645661 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.645509 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eeb1a9ba-ece4-4e09-8bc7-de92eb38f029-host-run-ovn-kubernetes\") pod \"ovnkube-node-hnzgv\" (UID: \"eeb1a9ba-ece4-4e09-8bc7-de92eb38f029\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnzgv" Sep 17 07:27:38.645661 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.645525 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e9046265-84c2-44b5-9d4a-da21902077aa-sys\") pod \"tuned-wxwzn\" (UID: \"e9046265-84c2-44b5-9d4a-da21902077aa\") " pod="openshift-cluster-node-tuning-operator/tuned-wxwzn" Sep 17 07:27:38.646498 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.645656 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/eeb1a9ba-ece4-4e09-8bc7-de92eb38f029-host-run-netns\") pod \"ovnkube-node-hnzgv\" (UID: \"eeb1a9ba-ece4-4e09-8bc7-de92eb38f029\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnzgv" Sep 17 07:27:38.646498 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.645572 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e9046265-84c2-44b5-9d4a-da21902077aa-host\") pod \"tuned-wxwzn\" (UID: \"e9046265-84c2-44b5-9d4a-da21902077aa\") " pod="openshift-cluster-node-tuning-operator/tuned-wxwzn" Sep 17 07:27:38.646498 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.645503 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/eeb1a9ba-ece4-4e09-8bc7-de92eb38f029-host-kubelet\") pod \"ovnkube-node-hnzgv\" (UID: \"eeb1a9ba-ece4-4e09-8bc7-de92eb38f029\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnzgv" Sep 17 07:27:38.646498 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.645681 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6642bc78-77e5-44af-af76-a8f87d25c89a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lh5z8\" (UID: \"6642bc78-77e5-44af-af76-a8f87d25c89a\") " pod="openshift-multus/multus-additional-cni-plugins-lh5z8" Sep 17 07:27:38.646498 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.645549 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e9046265-84c2-44b5-9d4a-da21902077aa-sys\") pod \"tuned-wxwzn\" (UID: \"e9046265-84c2-44b5-9d4a-da21902077aa\") " pod="openshift-cluster-node-tuning-operator/tuned-wxwzn" Sep 17 07:27:38.646498 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.645716 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a236d1bd-81b1-4c9e-a14f-997204ad1b98-dbus\") pod \"global-pull-secret-syncer-lqcw7\" (UID: \"a236d1bd-81b1-4c9e-a14f-997204ad1b98\") " pod="kube-system/global-pull-secret-syncer-lqcw7" Sep 17 07:27:38.646498 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.645741 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/737e0243-b8eb-47e4-bf0f-c6b31e6ba787-metrics-certs\") pod \"network-metrics-daemon-lnpjx\" (UID: \"737e0243-b8eb-47e4-bf0f-c6b31e6ba787\") " pod="openshift-multus/network-metrics-daemon-lnpjx" Sep 17 07:27:38.646498 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.645773 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d2d890e8-ee86-4c3d-b7a7-81f5f57b3e43-socket-dir\") pod \"aws-ebs-csi-driver-node-dtt5v\" (UID: \"d2d890e8-ee86-4c3d-b7a7-81f5f57b3e43\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dtt5v" Sep 17 07:27:38.646498 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.645797 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d2d890e8-ee86-4c3d-b7a7-81f5f57b3e43-etc-selinux\") pod \"aws-ebs-csi-driver-node-dtt5v\" (UID: \"d2d890e8-ee86-4c3d-b7a7-81f5f57b3e43\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dtt5v" Sep 17 07:27:38.646498 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.645821 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eeb1a9ba-ece4-4e09-8bc7-de92eb38f029-var-lib-openvswitch\") pod \"ovnkube-node-hnzgv\" (UID: \"eeb1a9ba-ece4-4e09-8bc7-de92eb38f029\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnzgv" Sep 17 07:27:38.646498 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.645845 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a236d1bd-81b1-4c9e-a14f-997204ad1b98-dbus\") pod \"global-pull-secret-syncer-lqcw7\" (UID: \"a236d1bd-81b1-4c9e-a14f-997204ad1b98\") " pod="kube-system/global-pull-secret-syncer-lqcw7" Sep 17 07:27:38.646498 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.645847 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e9046265-84c2-44b5-9d4a-da21902077aa-var-lib-kubelet\") pod \"tuned-wxwzn\" (UID: \"e9046265-84c2-44b5-9d4a-da21902077aa\") " pod="openshift-cluster-node-tuning-operator/tuned-wxwzn" Sep 17 07:27:38.646498 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.645903 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e9046265-84c2-44b5-9d4a-da21902077aa-var-lib-kubelet\") pod \"tuned-wxwzn\" (UID: \"e9046265-84c2-44b5-9d4a-da21902077aa\") " pod="openshift-cluster-node-tuning-operator/tuned-wxwzn" Sep 17 07:27:38.646498 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.645913 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6642bc78-77e5-44af-af76-a8f87d25c89a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lh5z8\" (UID: \"6642bc78-77e5-44af-af76-a8f87d25c89a\") " pod="openshift-multus/multus-additional-cni-plugins-lh5z8" Sep 17 07:27:38.646498 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.645937 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eeb1a9ba-ece4-4e09-8bc7-de92eb38f029-var-lib-openvswitch\") pod \"ovnkube-node-hnzgv\" (UID: \"eeb1a9ba-ece4-4e09-8bc7-de92eb38f029\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnzgv" Sep 17 07:27:38.646498 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.645940 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6642bc78-77e5-44af-af76-a8f87d25c89a-system-cni-dir\") pod \"multus-additional-cni-plugins-lh5z8\" (UID: \"6642bc78-77e5-44af-af76-a8f87d25c89a\") " pod="openshift-multus/multus-additional-cni-plugins-lh5z8" Sep 17 07:27:38.646498 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.645969 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6642bc78-77e5-44af-af76-a8f87d25c89a-system-cni-dir\") pod \"multus-additional-cni-plugins-lh5z8\" (UID: \"6642bc78-77e5-44af-af76-a8f87d25c89a\") " pod="openshift-multus/multus-additional-cni-plugins-lh5z8" Sep 17 07:27:38.647343 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.645969 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d2d890e8-ee86-4c3d-b7a7-81f5f57b3e43-etc-selinux\") pod \"aws-ebs-csi-driver-node-dtt5v\" (UID: \"d2d890e8-ee86-4c3d-b7a7-81f5f57b3e43\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dtt5v" Sep 17 07:27:38.647343 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.645968 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2v6lb\" (UniqueName: \"kubernetes.io/projected/6642bc78-77e5-44af-af76-a8f87d25c89a-kube-api-access-2v6lb\") pod \"multus-additional-cni-plugins-lh5z8\" (UID: \"6642bc78-77e5-44af-af76-a8f87d25c89a\") " pod="openshift-multus/multus-additional-cni-plugins-lh5z8" Sep 17 07:27:38.647343 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:27:38.645884 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 07:27:38.647343 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.646013 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d2d890e8-ee86-4c3d-b7a7-81f5f57b3e43-socket-dir\") pod \"aws-ebs-csi-driver-node-dtt5v\" (UID: \"d2d890e8-ee86-4c3d-b7a7-81f5f57b3e43\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dtt5v" Sep 17 07:27:38.647343 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.646018 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6642bc78-77e5-44af-af76-a8f87d25c89a-cni-binary-copy\") pod \"multus-additional-cni-plugins-lh5z8\" (UID: \"6642bc78-77e5-44af-af76-a8f87d25c89a\") " pod="openshift-multus/multus-additional-cni-plugins-lh5z8" Sep 17 07:27:38.647343 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.646099 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f50cb832-b497-4a6a-bbe6-b758ec939688-host\") pod \"node-ca-fwdd5\" (UID: \"f50cb832-b497-4a6a-bbe6-b758ec939688\") " pod="openshift-image-registry/node-ca-fwdd5" Sep 17 07:27:38.647343 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.646108 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6642bc78-77e5-44af-af76-a8f87d25c89a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lh5z8\" (UID: \"6642bc78-77e5-44af-af76-a8f87d25c89a\") " pod="openshift-multus/multus-additional-cni-plugins-lh5z8" Sep 17 07:27:38.647343 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.646160 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6642bc78-77e5-44af-af76-a8f87d25c89a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lh5z8\" (UID: \"6642bc78-77e5-44af-af76-a8f87d25c89a\") " pod="openshift-multus/multus-additional-cni-plugins-lh5z8" Sep 17 07:27:38.647343 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:27:38.646200 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/737e0243-b8eb-47e4-bf0f-c6b31e6ba787-metrics-certs podName:737e0243-b8eb-47e4-bf0f-c6b31e6ba787 nodeName:}" failed. No retries permitted until 2026-09-17 07:27:39.146178808 +0000 UTC m=+3.027213161 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/737e0243-b8eb-47e4-bf0f-c6b31e6ba787-metrics-certs") pod "network-metrics-daemon-lnpjx" (UID: "737e0243-b8eb-47e4-bf0f-c6b31e6ba787") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 07:27:38.647343 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.646217 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f50cb832-b497-4a6a-bbe6-b758ec939688-host\") pod \"node-ca-fwdd5\" (UID: \"f50cb832-b497-4a6a-bbe6-b758ec939688\") " pod="openshift-image-registry/node-ca-fwdd5" Sep 17 07:27:38.647343 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.646225 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92f4v\" (UniqueName: \"kubernetes.io/projected/f50cb832-b497-4a6a-bbe6-b758ec939688-kube-api-access-92f4v\") pod \"node-ca-fwdd5\" (UID: \"f50cb832-b497-4a6a-bbe6-b758ec939688\") " pod="openshift-image-registry/node-ca-fwdd5" Sep 17 07:27:38.647343 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.646265 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/eeb1a9ba-ece4-4e09-8bc7-de92eb38f029-env-overrides\") pod \"ovnkube-node-hnzgv\" (UID: \"eeb1a9ba-ece4-4e09-8bc7-de92eb38f029\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnzgv" Sep 17 07:27:38.647343 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.646299 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e9046265-84c2-44b5-9d4a-da21902077aa-etc-modprobe-d\") pod \"tuned-wxwzn\" (UID: \"e9046265-84c2-44b5-9d4a-da21902077aa\") " pod="openshift-cluster-node-tuning-operator/tuned-wxwzn" Sep 17 07:27:38.647343 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.646323 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e9046265-84c2-44b5-9d4a-da21902077aa-etc-tuned\") pod \"tuned-wxwzn\" (UID: \"e9046265-84c2-44b5-9d4a-da21902077aa\") " pod="openshift-cluster-node-tuning-operator/tuned-wxwzn" Sep 17 07:27:38.647343 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.646347 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f50cb832-b497-4a6a-bbe6-b758ec939688-serviceca\") pod \"node-ca-fwdd5\" (UID: \"f50cb832-b497-4a6a-bbe6-b758ec939688\") " pod="openshift-image-registry/node-ca-fwdd5" Sep 17 07:27:38.647343 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.646371 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/60c84ae0-1da1-4845-8d12-d9656a247588-multus-conf-dir\") pod \"multus-zpwkj\" (UID: \"60c84ae0-1da1-4845-8d12-d9656a247588\") " pod="openshift-multus/multus-zpwkj" Sep 17 07:27:38.647343 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.646394 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eeb1a9ba-ece4-4e09-8bc7-de92eb38f029-run-openvswitch\") pod \"ovnkube-node-hnzgv\" (UID: \"eeb1a9ba-ece4-4e09-8bc7-de92eb38f029\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnzgv" Sep 17 07:27:38.648148 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.646408 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e9046265-84c2-44b5-9d4a-da21902077aa-etc-modprobe-d\") pod \"tuned-wxwzn\" (UID: \"e9046265-84c2-44b5-9d4a-da21902077aa\") " pod="openshift-cluster-node-tuning-operator/tuned-wxwzn" Sep 17 07:27:38.648148 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.646417 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/eeb1a9ba-ece4-4e09-8bc7-de92eb38f029-ovnkube-config\") pod \"ovnkube-node-hnzgv\" (UID: \"eeb1a9ba-ece4-4e09-8bc7-de92eb38f029\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnzgv" Sep 17 07:27:38.648148 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.646461 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e9046265-84c2-44b5-9d4a-da21902077aa-etc-sysctl-d\") pod \"tuned-wxwzn\" (UID: \"e9046265-84c2-44b5-9d4a-da21902077aa\") " pod="openshift-cluster-node-tuning-operator/tuned-wxwzn" Sep 17 07:27:38.648148 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.646474 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/60c84ae0-1da1-4845-8d12-d9656a247588-multus-conf-dir\") pod \"multus-zpwkj\" (UID: \"60c84ae0-1da1-4845-8d12-d9656a247588\") " pod="openshift-multus/multus-zpwkj" Sep 17 07:27:38.648148 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.646492 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/60c84ae0-1da1-4845-8d12-d9656a247588-multus-socket-dir-parent\") pod \"multus-zpwkj\" (UID: \"60c84ae0-1da1-4845-8d12-d9656a247588\") " pod="openshift-multus/multus-zpwkj" Sep 17 07:27:38.648148 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.646585 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e9046265-84c2-44b5-9d4a-da21902077aa-etc-sysctl-d\") pod \"tuned-wxwzn\" (UID: \"e9046265-84c2-44b5-9d4a-da21902077aa\") " pod="openshift-cluster-node-tuning-operator/tuned-wxwzn" Sep 17 07:27:38.648148 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.646636 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eeb1a9ba-ece4-4e09-8bc7-de92eb38f029-run-openvswitch\") pod \"ovnkube-node-hnzgv\" (UID: \"eeb1a9ba-ece4-4e09-8bc7-de92eb38f029\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnzgv" Sep 17 07:27:38.648148 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.646665 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/60c84ae0-1da1-4845-8d12-d9656a247588-host-run-multus-certs\") pod \"multus-zpwkj\" (UID: \"60c84ae0-1da1-4845-8d12-d9656a247588\") " pod="openshift-multus/multus-zpwkj" Sep 17 07:27:38.648148 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.646690 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/eeb1a9ba-ece4-4e09-8bc7-de92eb38f029-run-systemd\") pod \"ovnkube-node-hnzgv\" (UID: \"eeb1a9ba-ece4-4e09-8bc7-de92eb38f029\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnzgv" Sep 17 07:27:38.648148 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.646713 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/60c84ae0-1da1-4845-8d12-d9656a247588-host-var-lib-cni-bin\") pod \"multus-zpwkj\" (UID: \"60c84ae0-1da1-4845-8d12-d9656a247588\") " pod="openshift-multus/multus-zpwkj" Sep 17 07:27:38.648148 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.646737 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/eeb1a9ba-ece4-4e09-8bc7-de92eb38f029-env-overrides\") pod \"ovnkube-node-hnzgv\" (UID: \"eeb1a9ba-ece4-4e09-8bc7-de92eb38f029\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnzgv" Sep 17 07:27:38.648148 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.646737 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/60c84ae0-1da1-4845-8d12-d9656a247588-host-var-lib-cni-multus\") pod \"multus-zpwkj\" (UID: \"60c84ae0-1da1-4845-8d12-d9656a247588\") " pod="openshift-multus/multus-zpwkj" Sep 17 07:27:38.648148 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.646785 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e9046265-84c2-44b5-9d4a-da21902077aa-run\") pod \"tuned-wxwzn\" (UID: \"e9046265-84c2-44b5-9d4a-da21902077aa\") " pod="openshift-cluster-node-tuning-operator/tuned-wxwzn" Sep 17 07:27:38.648148 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.646794 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/eeb1a9ba-ece4-4e09-8bc7-de92eb38f029-run-systemd\") pod \"ovnkube-node-hnzgv\" (UID: \"eeb1a9ba-ece4-4e09-8bc7-de92eb38f029\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnzgv" Sep 17 07:27:38.648148 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.646812 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6642bc78-77e5-44af-af76-a8f87d25c89a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lh5z8\" (UID: \"6642bc78-77e5-44af-af76-a8f87d25c89a\") " pod="openshift-multus/multus-additional-cni-plugins-lh5z8" Sep 17 07:27:38.648148 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.646833 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/60c84ae0-1da1-4845-8d12-d9656a247588-host-run-multus-certs\") pod \"multus-zpwkj\" (UID: \"60c84ae0-1da1-4845-8d12-d9656a247588\") " pod="openshift-multus/multus-zpwkj" Sep 17 07:27:38.648148 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.646839 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpnwq\" (UniqueName: \"kubernetes.io/projected/737e0243-b8eb-47e4-bf0f-c6b31e6ba787-kube-api-access-tpnwq\") pod \"network-metrics-daemon-lnpjx\" (UID: \"737e0243-b8eb-47e4-bf0f-c6b31e6ba787\") " pod="openshift-multus/network-metrics-daemon-lnpjx" Sep 17 07:27:38.648148 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.646878 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f50cb832-b497-4a6a-bbe6-b758ec939688-serviceca\") pod \"node-ca-fwdd5\" (UID: \"f50cb832-b497-4a6a-bbe6-b758ec939688\") " pod="openshift-image-registry/node-ca-fwdd5" Sep 17 07:27:38.649022 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.646903 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/60c84ae0-1da1-4845-8d12-d9656a247588-host-var-lib-cni-bin\") pod \"multus-zpwkj\" (UID: \"60c84ae0-1da1-4845-8d12-d9656a247588\") " pod="openshift-multus/multus-zpwkj" Sep 17 07:27:38.649022 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.646916 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/bfe3853f-127c-4124-8df0-cdd6439052bb-konnectivity-ca\") pod \"konnectivity-agent-qbfvv\" (UID: \"bfe3853f-127c-4124-8df0-cdd6439052bb\") " pod="kube-system/konnectivity-agent-qbfvv" Sep 17 07:27:38.649022 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.646952 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/eeb1a9ba-ece4-4e09-8bc7-de92eb38f029-ovnkube-config\") pod \"ovnkube-node-hnzgv\" (UID: \"eeb1a9ba-ece4-4e09-8bc7-de92eb38f029\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnzgv" Sep 17 07:27:38.649022 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.646971 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/60c84ae0-1da1-4845-8d12-d9656a247588-host-var-lib-cni-multus\") pod \"multus-zpwkj\" (UID: \"60c84ae0-1da1-4845-8d12-d9656a247588\") " pod="openshift-multus/multus-zpwkj" Sep 17 07:27:38.649022 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.647006 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e9046265-84c2-44b5-9d4a-da21902077aa-run\") pod \"tuned-wxwzn\" (UID: \"e9046265-84c2-44b5-9d4a-da21902077aa\") " pod="openshift-cluster-node-tuning-operator/tuned-wxwzn" Sep 17 07:27:38.649022 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.647025 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/eeb1a9ba-ece4-4e09-8bc7-de92eb38f029-run-ovn\") pod \"ovnkube-node-hnzgv\" (UID: \"eeb1a9ba-ece4-4e09-8bc7-de92eb38f029\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnzgv" Sep 17 07:27:38.649022 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.647097 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/eeb1a9ba-ece4-4e09-8bc7-de92eb38f029-host-cni-netd\") pod \"ovnkube-node-hnzgv\" (UID: \"eeb1a9ba-ece4-4e09-8bc7-de92eb38f029\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnzgv" Sep 17 07:27:38.649022 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.647126 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/eeb1a9ba-ece4-4e09-8bc7-de92eb38f029-run-ovn\") pod \"ovnkube-node-hnzgv\" (UID: \"eeb1a9ba-ece4-4e09-8bc7-de92eb38f029\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnzgv" Sep 17 07:27:38.649022 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.647134 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e9046265-84c2-44b5-9d4a-da21902077aa-etc-sysctl-conf\") pod \"tuned-wxwzn\" (UID: \"e9046265-84c2-44b5-9d4a-da21902077aa\") " pod="openshift-cluster-node-tuning-operator/tuned-wxwzn" Sep 17 07:27:38.649022 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.647166 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e9046265-84c2-44b5-9d4a-da21902077aa-tmp\") pod \"tuned-wxwzn\" (UID: \"e9046265-84c2-44b5-9d4a-da21902077aa\") " pod="openshift-cluster-node-tuning-operator/tuned-wxwzn" Sep 17 07:27:38.649022 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.647171 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/eeb1a9ba-ece4-4e09-8bc7-de92eb38f029-host-cni-netd\") pod \"ovnkube-node-hnzgv\" (UID: \"eeb1a9ba-ece4-4e09-8bc7-de92eb38f029\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnzgv" Sep 17 07:27:38.649022 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.647190 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6642bc78-77e5-44af-af76-a8f87d25c89a-cnibin\") pod \"multus-additional-cni-plugins-lh5z8\" (UID: \"6642bc78-77e5-44af-af76-a8f87d25c89a\") " pod="openshift-multus/multus-additional-cni-plugins-lh5z8" Sep 17 07:27:38.649022 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.647217 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/60c84ae0-1da1-4845-8d12-d9656a247588-multus-socket-dir-parent\") pod \"multus-zpwkj\" (UID: \"60c84ae0-1da1-4845-8d12-d9656a247588\") " pod="openshift-multus/multus-zpwkj" Sep 17 07:27:38.649022 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.647254 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d2d890e8-ee86-4c3d-b7a7-81f5f57b3e43-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dtt5v\" (UID: \"d2d890e8-ee86-4c3d-b7a7-81f5f57b3e43\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dtt5v" Sep 17 07:27:38.649022 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.647280 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p66qq\" (UniqueName: \"kubernetes.io/projected/d2d890e8-ee86-4c3d-b7a7-81f5f57b3e43-kube-api-access-p66qq\") pod \"aws-ebs-csi-driver-node-dtt5v\" (UID: \"d2d890e8-ee86-4c3d-b7a7-81f5f57b3e43\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dtt5v" Sep 17 07:27:38.649022 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.647306 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnl6r\" (UniqueName: \"kubernetes.io/projected/c42682fc-ab24-4492-a27b-c03a53648e5b-kube-api-access-pnl6r\") pod \"iptables-alerter-c8gq4\" (UID: \"c42682fc-ab24-4492-a27b-c03a53648e5b\") " pod="openshift-network-operator/iptables-alerter-c8gq4" Sep 17 07:27:38.649022 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.647300 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6642bc78-77e5-44af-af76-a8f87d25c89a-cnibin\") pod \"multus-additional-cni-plugins-lh5z8\" (UID: \"6642bc78-77e5-44af-af76-a8f87d25c89a\") " pod="openshift-multus/multus-additional-cni-plugins-lh5z8" Sep 17 07:27:38.649819 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.647326 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e9046265-84c2-44b5-9d4a-da21902077aa-etc-sysctl-conf\") pod \"tuned-wxwzn\" (UID: \"e9046265-84c2-44b5-9d4a-da21902077aa\") " pod="openshift-cluster-node-tuning-operator/tuned-wxwzn" Sep 17 07:27:38.649819 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.647370 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d2d890e8-ee86-4c3d-b7a7-81f5f57b3e43-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dtt5v\" (UID: \"d2d890e8-ee86-4c3d-b7a7-81f5f57b3e43\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dtt5v" Sep 17 07:27:38.649819 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.647417 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/eeb1a9ba-ece4-4e09-8bc7-de92eb38f029-ovnkube-script-lib\") pod \"ovnkube-node-hnzgv\" (UID: \"eeb1a9ba-ece4-4e09-8bc7-de92eb38f029\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnzgv" Sep 17 07:27:38.649819 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.647421 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6642bc78-77e5-44af-af76-a8f87d25c89a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lh5z8\" (UID: \"6642bc78-77e5-44af-af76-a8f87d25c89a\") " pod="openshift-multus/multus-additional-cni-plugins-lh5z8" Sep 17 07:27:38.649819 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.647453 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/bfe3853f-127c-4124-8df0-cdd6439052bb-konnectivity-ca\") pod \"konnectivity-agent-qbfvv\" (UID: \"bfe3853f-127c-4124-8df0-cdd6439052bb\") " pod="kube-system/konnectivity-agent-qbfvv" Sep 17 07:27:38.649819 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.647464 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e9046265-84c2-44b5-9d4a-da21902077aa-etc-kubernetes\") pod \"tuned-wxwzn\" (UID: \"e9046265-84c2-44b5-9d4a-da21902077aa\") " pod="openshift-cluster-node-tuning-operator/tuned-wxwzn" Sep 17 07:27:38.649819 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.647498 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a236d1bd-81b1-4c9e-a14f-997204ad1b98-kubelet-config\") pod \"global-pull-secret-syncer-lqcw7\" (UID: \"a236d1bd-81b1-4c9e-a14f-997204ad1b98\") " pod="kube-system/global-pull-secret-syncer-lqcw7" Sep 17 07:27:38.649819 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.647508 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e9046265-84c2-44b5-9d4a-da21902077aa-etc-kubernetes\") pod \"tuned-wxwzn\" (UID: \"e9046265-84c2-44b5-9d4a-da21902077aa\") " pod="openshift-cluster-node-tuning-operator/tuned-wxwzn" Sep 17 07:27:38.649819 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.647524 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d2d890e8-ee86-4c3d-b7a7-81f5f57b3e43-device-dir\") pod \"aws-ebs-csi-driver-node-dtt5v\" (UID: \"d2d890e8-ee86-4c3d-b7a7-81f5f57b3e43\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dtt5v" Sep 17 07:27:38.649819 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.647549 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/60c84ae0-1da1-4845-8d12-d9656a247588-host-run-k8s-cni-cncf-io\") pod \"multus-zpwkj\" (UID: \"60c84ae0-1da1-4845-8d12-d9656a247588\") " pod="openshift-multus/multus-zpwkj" Sep 17 07:27:38.649819 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.647552 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a236d1bd-81b1-4c9e-a14f-997204ad1b98-kubelet-config\") pod \"global-pull-secret-syncer-lqcw7\" (UID: \"a236d1bd-81b1-4c9e-a14f-997204ad1b98\") " pod="kube-system/global-pull-secret-syncer-lqcw7" Sep 17 07:27:38.649819 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.647573 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/60c84ae0-1da1-4845-8d12-d9656a247588-host-var-lib-kubelet\") pod \"multus-zpwkj\" (UID: \"60c84ae0-1da1-4845-8d12-d9656a247588\") " pod="openshift-multus/multus-zpwkj" Sep 17 07:27:38.649819 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.647596 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d2d890e8-ee86-4c3d-b7a7-81f5f57b3e43-device-dir\") pod \"aws-ebs-csi-driver-node-dtt5v\" (UID: \"d2d890e8-ee86-4c3d-b7a7-81f5f57b3e43\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dtt5v" Sep 17 07:27:38.649819 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.647598 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/eeb1a9ba-ece4-4e09-8bc7-de92eb38f029-systemd-units\") pod \"ovnkube-node-hnzgv\" (UID: \"eeb1a9ba-ece4-4e09-8bc7-de92eb38f029\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnzgv" Sep 17 07:27:38.649819 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.647631 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/eeb1a9ba-ece4-4e09-8bc7-de92eb38f029-ovn-node-metrics-cert\") pod \"ovnkube-node-hnzgv\" (UID: \"eeb1a9ba-ece4-4e09-8bc7-de92eb38f029\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnzgv" Sep 17 07:27:38.649819 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.647634 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/eeb1a9ba-ece4-4e09-8bc7-de92eb38f029-systemd-units\") pod \"ovnkube-node-hnzgv\" (UID: \"eeb1a9ba-ece4-4e09-8bc7-de92eb38f029\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnzgv" Sep 17 07:27:38.649819 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.647631 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/60c84ae0-1da1-4845-8d12-d9656a247588-host-var-lib-kubelet\") pod \"multus-zpwkj\" (UID: \"60c84ae0-1da1-4845-8d12-d9656a247588\") " pod="openshift-multus/multus-zpwkj" Sep 17 07:27:38.651365 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.647605 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/60c84ae0-1da1-4845-8d12-d9656a247588-host-run-k8s-cni-cncf-io\") pod \"multus-zpwkj\" (UID: \"60c84ae0-1da1-4845-8d12-d9656a247588\") " pod="openshift-multus/multus-zpwkj" Sep 17 07:27:38.651365 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.647669 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e9046265-84c2-44b5-9d4a-da21902077aa-lib-modules\") pod \"tuned-wxwzn\" (UID: \"e9046265-84c2-44b5-9d4a-da21902077aa\") " pod="openshift-cluster-node-tuning-operator/tuned-wxwzn" Sep 17 07:27:38.651365 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.647698 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7qqf\" (UniqueName: \"kubernetes.io/projected/e9046265-84c2-44b5-9d4a-da21902077aa-kube-api-access-j7qqf\") pod \"tuned-wxwzn\" (UID: \"e9046265-84c2-44b5-9d4a-da21902077aa\") " pod="openshift-cluster-node-tuning-operator/tuned-wxwzn" Sep 17 07:27:38.651365 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.647721 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d9df1c13-2108-44d8-9137-730e35cf2165-hosts-file\") pod \"node-resolver-xfggl\" (UID: \"d9df1c13-2108-44d8-9137-730e35cf2165\") " pod="openshift-dns/node-resolver-xfggl" Sep 17 07:27:38.651365 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.647745 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9df1c13-2108-44d8-9137-730e35cf2165-tmp-dir\") pod \"node-resolver-xfggl\" (UID: \"d9df1c13-2108-44d8-9137-730e35cf2165\") " pod="openshift-dns/node-resolver-xfggl" Sep 17 07:27:38.651365 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.647810 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d9df1c13-2108-44d8-9137-730e35cf2165-hosts-file\") pod \"node-resolver-xfggl\" (UID: \"d9df1c13-2108-44d8-9137-730e35cf2165\") " pod="openshift-dns/node-resolver-xfggl" Sep 17 07:27:38.651365 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.647838 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e9046265-84c2-44b5-9d4a-da21902077aa-lib-modules\") pod \"tuned-wxwzn\" (UID: \"e9046265-84c2-44b5-9d4a-da21902077aa\") " pod="openshift-cluster-node-tuning-operator/tuned-wxwzn" Sep 17 07:27:38.651365 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.647951 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/eeb1a9ba-ece4-4e09-8bc7-de92eb38f029-ovnkube-script-lib\") pod \"ovnkube-node-hnzgv\" (UID: \"eeb1a9ba-ece4-4e09-8bc7-de92eb38f029\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnzgv" Sep 17 07:27:38.651365 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.647999 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eeb1a9ba-ece4-4e09-8bc7-de92eb38f029-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hnzgv\" (UID: \"eeb1a9ba-ece4-4e09-8bc7-de92eb38f029\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnzgv" Sep 17 07:27:38.651365 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.648028 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e9046265-84c2-44b5-9d4a-da21902077aa-etc-sysconfig\") pod \"tuned-wxwzn\" (UID: \"e9046265-84c2-44b5-9d4a-da21902077aa\") " pod="openshift-cluster-node-tuning-operator/tuned-wxwzn" Sep 17 07:27:38.651365 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.648043 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9df1c13-2108-44d8-9137-730e35cf2165-tmp-dir\") pod \"node-resolver-xfggl\" (UID: \"d9df1c13-2108-44d8-9137-730e35cf2165\") " pod="openshift-dns/node-resolver-xfggl" Sep 17 07:27:38.651365 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.648055 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnz4n\" (UniqueName: \"kubernetes.io/projected/82555a36-653a-4106-a31e-b4da6445b838-kube-api-access-gnz4n\") pod \"network-check-target-bvxfq\" (UID: \"82555a36-653a-4106-a31e-b4da6445b838\") " pod="openshift-network-diagnostics/network-check-target-bvxfq" Sep 17 07:27:38.651365 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.648077 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eeb1a9ba-ece4-4e09-8bc7-de92eb38f029-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hnzgv\" (UID: \"eeb1a9ba-ece4-4e09-8bc7-de92eb38f029\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnzgv" Sep 17 07:27:38.651365 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.648081 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d2d890e8-ee86-4c3d-b7a7-81f5f57b3e43-registration-dir\") pod \"aws-ebs-csi-driver-node-dtt5v\" (UID: \"d2d890e8-ee86-4c3d-b7a7-81f5f57b3e43\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dtt5v" Sep 17 07:27:38.651365 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.648122 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/60c84ae0-1da1-4845-8d12-d9656a247588-multus-daemon-config\") pod \"multus-zpwkj\" (UID: \"60c84ae0-1da1-4845-8d12-d9656a247588\") " pod="openshift-multus/multus-zpwkj" Sep 17 07:27:38.651365 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.648128 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e9046265-84c2-44b5-9d4a-da21902077aa-etc-sysconfig\") pod \"tuned-wxwzn\" (UID: \"e9046265-84c2-44b5-9d4a-da21902077aa\") " pod="openshift-cluster-node-tuning-operator/tuned-wxwzn" Sep 17 07:27:38.651365 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.648134 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d2d890e8-ee86-4c3d-b7a7-81f5f57b3e43-registration-dir\") pod \"aws-ebs-csi-driver-node-dtt5v\" (UID: \"d2d890e8-ee86-4c3d-b7a7-81f5f57b3e43\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dtt5v" Sep 17 07:27:38.652109 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.648147 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/60c84ae0-1da1-4845-8d12-d9656a247588-etc-kubernetes\") pod \"multus-zpwkj\" (UID: \"60c84ae0-1da1-4845-8d12-d9656a247588\") " pod="openshift-multus/multus-zpwkj" Sep 17 07:27:38.652109 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.648170 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/eeb1a9ba-ece4-4e09-8bc7-de92eb38f029-host-slash\") pod \"ovnkube-node-hnzgv\" (UID: \"eeb1a9ba-ece4-4e09-8bc7-de92eb38f029\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnzgv" Sep 17 07:27:38.652109 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.648182 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/60c84ae0-1da1-4845-8d12-d9656a247588-etc-kubernetes\") pod \"multus-zpwkj\" (UID: \"60c84ae0-1da1-4845-8d12-d9656a247588\") " pod="openshift-multus/multus-zpwkj" Sep 17 07:27:38.652109 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.648193 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eeb1a9ba-ece4-4e09-8bc7-de92eb38f029-etc-openvswitch\") pod \"ovnkube-node-hnzgv\" (UID: \"eeb1a9ba-ece4-4e09-8bc7-de92eb38f029\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnzgv" Sep 17 07:27:38.652109 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.648227 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/eeb1a9ba-ece4-4e09-8bc7-de92eb38f029-host-slash\") pod \"ovnkube-node-hnzgv\" (UID: \"eeb1a9ba-ece4-4e09-8bc7-de92eb38f029\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnzgv" Sep 17 07:27:38.652109 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.648241 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eeb1a9ba-ece4-4e09-8bc7-de92eb38f029-etc-openvswitch\") pod \"ovnkube-node-hnzgv\" (UID: \"eeb1a9ba-ece4-4e09-8bc7-de92eb38f029\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnzgv" Sep 17 07:27:38.652109 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.648254 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e9046265-84c2-44b5-9d4a-da21902077aa-etc-systemd\") pod \"tuned-wxwzn\" (UID: \"e9046265-84c2-44b5-9d4a-da21902077aa\") " pod="openshift-cluster-node-tuning-operator/tuned-wxwzn" Sep 17 07:27:38.652109 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.648318 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e9046265-84c2-44b5-9d4a-da21902077aa-etc-systemd\") pod \"tuned-wxwzn\" (UID: \"e9046265-84c2-44b5-9d4a-da21902077aa\") " pod="openshift-cluster-node-tuning-operator/tuned-wxwzn" Sep 17 07:27:38.652109 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.648557 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/60c84ae0-1da1-4845-8d12-d9656a247588-multus-daemon-config\") pod \"multus-zpwkj\" (UID: \"60c84ae0-1da1-4845-8d12-d9656a247588\") " pod="openshift-multus/multus-zpwkj" Sep 17 07:27:38.652109 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.650710 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/eeb1a9ba-ece4-4e09-8bc7-de92eb38f029-ovn-node-metrics-cert\") pod \"ovnkube-node-hnzgv\" (UID: \"eeb1a9ba-ece4-4e09-8bc7-de92eb38f029\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnzgv" Sep 17 07:27:38.652109 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.650714 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e9046265-84c2-44b5-9d4a-da21902077aa-tmp\") pod \"tuned-wxwzn\" (UID: \"e9046265-84c2-44b5-9d4a-da21902077aa\") " pod="openshift-cluster-node-tuning-operator/tuned-wxwzn" Sep 17 07:27:38.652109 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.650948 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/bfe3853f-127c-4124-8df0-cdd6439052bb-agent-certs\") pod \"konnectivity-agent-qbfvv\" (UID: \"bfe3853f-127c-4124-8df0-cdd6439052bb\") " pod="kube-system/konnectivity-agent-qbfvv" Sep 17 07:27:38.652109 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.651097 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e9046265-84c2-44b5-9d4a-da21902077aa-etc-tuned\") pod \"tuned-wxwzn\" (UID: \"e9046265-84c2-44b5-9d4a-da21902077aa\") " pod="openshift-cluster-node-tuning-operator/tuned-wxwzn" Sep 17 07:27:38.658895 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.658872 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5vw8\" (UniqueName: \"kubernetes.io/projected/d9df1c13-2108-44d8-9137-730e35cf2165-kube-api-access-f5vw8\") pod \"node-resolver-xfggl\" (UID: \"d9df1c13-2108-44d8-9137-730e35cf2165\") " pod="openshift-dns/node-resolver-xfggl" Sep 17 07:27:38.659321 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.659297 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjmvw\" (UniqueName: \"kubernetes.io/projected/eeb1a9ba-ece4-4e09-8bc7-de92eb38f029-kube-api-access-kjmvw\") pod \"ovnkube-node-hnzgv\" (UID: \"eeb1a9ba-ece4-4e09-8bc7-de92eb38f029\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnzgv" Sep 17 07:27:38.659583 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.659559 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5z7j\" (UniqueName: \"kubernetes.io/projected/60c84ae0-1da1-4845-8d12-d9656a247588-kube-api-access-v5z7j\") pod \"multus-zpwkj\" (UID: \"60c84ae0-1da1-4845-8d12-d9656a247588\") " pod="openshift-multus/multus-zpwkj" Sep 17 07:27:38.660090 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.660066 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2v6lb\" (UniqueName: \"kubernetes.io/projected/6642bc78-77e5-44af-af76-a8f87d25c89a-kube-api-access-2v6lb\") pod \"multus-additional-cni-plugins-lh5z8\" (UID: \"6642bc78-77e5-44af-af76-a8f87d25c89a\") " pod="openshift-multus/multus-additional-cni-plugins-lh5z8" Sep 17 07:27:38.661076 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:27:38.660795 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 17 07:27:38.661076 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:27:38.660820 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 17 07:27:38.661076 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:27:38.660834 2578 projected.go:196] Error preparing data for projected volume kube-api-access-gnz4n for pod openshift-network-diagnostics/network-check-target-bvxfq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 07:27:38.661076 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:27:38.660915 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/82555a36-653a-4106-a31e-b4da6445b838-kube-api-access-gnz4n podName:82555a36-653a-4106-a31e-b4da6445b838 nodeName:}" failed. No retries permitted until 2026-09-17 07:27:39.160897732 +0000 UTC m=+3.041932087 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-gnz4n" (UniqueName: "kubernetes.io/projected/82555a36-653a-4106-a31e-b4da6445b838-kube-api-access-gnz4n") pod "network-check-target-bvxfq" (UID: "82555a36-653a-4106-a31e-b4da6445b838") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 07:27:38.661076 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.661039 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-92f4v\" (UniqueName: \"kubernetes.io/projected/f50cb832-b497-4a6a-bbe6-b758ec939688-kube-api-access-92f4v\") pod \"node-ca-fwdd5\" (UID: \"f50cb832-b497-4a6a-bbe6-b758ec939688\") " pod="openshift-image-registry/node-ca-fwdd5" Sep 17 07:27:38.661682 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.661328 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpnwq\" (UniqueName: \"kubernetes.io/projected/737e0243-b8eb-47e4-bf0f-c6b31e6ba787-kube-api-access-tpnwq\") pod \"network-metrics-daemon-lnpjx\" (UID: \"737e0243-b8eb-47e4-bf0f-c6b31e6ba787\") " pod="openshift-multus/network-metrics-daemon-lnpjx" Sep 17 07:27:38.663654 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.663623 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p66qq\" (UniqueName: \"kubernetes.io/projected/d2d890e8-ee86-4c3d-b7a7-81f5f57b3e43-kube-api-access-p66qq\") pod \"aws-ebs-csi-driver-node-dtt5v\" (UID: \"d2d890e8-ee86-4c3d-b7a7-81f5f57b3e43\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dtt5v" Sep 17 07:27:38.663746 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.663727 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnl6r\" (UniqueName: \"kubernetes.io/projected/c42682fc-ab24-4492-a27b-c03a53648e5b-kube-api-access-pnl6r\") pod \"iptables-alerter-c8gq4\" (UID: \"c42682fc-ab24-4492-a27b-c03a53648e5b\") " pod="openshift-network-operator/iptables-alerter-c8gq4" Sep 17 07:27:38.664028 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.664007 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7qqf\" (UniqueName: \"kubernetes.io/projected/e9046265-84c2-44b5-9d4a-da21902077aa-kube-api-access-j7qqf\") pod \"tuned-wxwzn\" (UID: \"e9046265-84c2-44b5-9d4a-da21902077aa\") " pod="openshift-cluster-node-tuning-operator/tuned-wxwzn" Sep 17 07:27:38.828071 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.827991 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-zpwkj" Sep 17 07:27:38.836612 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.836588 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-hnzgv" Sep 17 07:27:38.848831 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.848811 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-c8gq4" Sep 17 07:27:38.855463 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.855440 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xfggl" Sep 17 07:27:38.863709 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.863691 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-wxwzn" Sep 17 07:27:38.871993 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.871975 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-fwdd5" Sep 17 07:27:38.879126 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.879104 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-lh5z8" Sep 17 07:27:38.886593 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.886572 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-qbfvv" Sep 17 07:27:38.891160 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:38.891141 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dtt5v" Sep 17 07:27:39.152098 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:39.152056 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a236d1bd-81b1-4c9e-a14f-997204ad1b98-original-pull-secret\") pod \"global-pull-secret-syncer-lqcw7\" (UID: \"a236d1bd-81b1-4c9e-a14f-997204ad1b98\") " pod="kube-system/global-pull-secret-syncer-lqcw7" Sep 17 07:27:39.152249 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:39.152116 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/737e0243-b8eb-47e4-bf0f-c6b31e6ba787-metrics-certs\") pod \"network-metrics-daemon-lnpjx\" (UID: \"737e0243-b8eb-47e4-bf0f-c6b31e6ba787\") " pod="openshift-multus/network-metrics-daemon-lnpjx" Sep 17 07:27:39.152249 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:27:39.152203 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 07:27:39.152329 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:27:39.152262 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/737e0243-b8eb-47e4-bf0f-c6b31e6ba787-metrics-certs podName:737e0243-b8eb-47e4-bf0f-c6b31e6ba787 nodeName:}" failed. No retries permitted until 2026-09-17 07:27:40.152247445 +0000 UTC m=+4.033281810 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/737e0243-b8eb-47e4-bf0f-c6b31e6ba787-metrics-certs") pod "network-metrics-daemon-lnpjx" (UID: "737e0243-b8eb-47e4-bf0f-c6b31e6ba787") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 07:27:39.152329 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:27:39.152203 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 17 07:27:39.152329 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:27:39.152293 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a236d1bd-81b1-4c9e-a14f-997204ad1b98-original-pull-secret podName:a236d1bd-81b1-4c9e-a14f-997204ad1b98 nodeName:}" failed. No retries permitted until 2026-09-17 07:27:40.15228736 +0000 UTC m=+4.033321697 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a236d1bd-81b1-4c9e-a14f-997204ad1b98-original-pull-secret") pod "global-pull-secret-syncer-lqcw7" (UID: "a236d1bd-81b1-4c9e-a14f-997204ad1b98") : object "kube-system"/"original-pull-secret" not registered Sep 17 07:27:39.252720 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:39.252686 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnz4n\" (UniqueName: \"kubernetes.io/projected/82555a36-653a-4106-a31e-b4da6445b838-kube-api-access-gnz4n\") pod \"network-check-target-bvxfq\" (UID: \"82555a36-653a-4106-a31e-b4da6445b838\") " pod="openshift-network-diagnostics/network-check-target-bvxfq" Sep 17 07:27:39.252890 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:27:39.252811 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 17 07:27:39.252890 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:27:39.252829 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 17 07:27:39.252890 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:27:39.252841 2578 projected.go:196] Error preparing data for projected volume kube-api-access-gnz4n for pod openshift-network-diagnostics/network-check-target-bvxfq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 07:27:39.253045 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:27:39.252918 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/82555a36-653a-4106-a31e-b4da6445b838-kube-api-access-gnz4n podName:82555a36-653a-4106-a31e-b4da6445b838 nodeName:}" failed. No retries permitted until 2026-09-17 07:27:40.252900601 +0000 UTC m=+4.133934938 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-gnz4n" (UniqueName: "kubernetes.io/projected/82555a36-653a-4106-a31e-b4da6445b838-kube-api-access-gnz4n") pod "network-check-target-bvxfq" (UID: "82555a36-653a-4106-a31e-b4da6445b838") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 07:27:39.543281 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:39.543252 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60c84ae0_1da1_4845_8d12_d9656a247588.slice/crio-99625c52fdb55aac6be727121140fec0bd4455bb1d4e64e5b3f1862044393b92 WatchSource:0}: Error finding container 99625c52fdb55aac6be727121140fec0bd4455bb1d4e64e5b3f1862044393b92: Status 404 returned error can't find the container with id 99625c52fdb55aac6be727121140fec0bd4455bb1d4e64e5b3f1862044393b92 Sep 17 07:27:39.544588 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:39.544565 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6642bc78_77e5_44af_af76_a8f87d25c89a.slice/crio-f2d65b689056147e6612a4f94040c83f24929647bfcbc6a9e742c5bdaeb7444b WatchSource:0}: Error finding container f2d65b689056147e6612a4f94040c83f24929647bfcbc6a9e742c5bdaeb7444b: Status 404 returned error can't find the container with id f2d65b689056147e6612a4f94040c83f24929647bfcbc6a9e742c5bdaeb7444b Sep 17 07:27:39.545258 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:39.545189 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeeb1a9ba_ece4_4e09_8bc7_de92eb38f029.slice/crio-839515f2dc9cfcac60584a910cb8de7b12e9e93365a922325c20ad62597f8f61 WatchSource:0}: Error finding container 839515f2dc9cfcac60584a910cb8de7b12e9e93365a922325c20ad62597f8f61: Status 404 returned error can't find the container with id 839515f2dc9cfcac60584a910cb8de7b12e9e93365a922325c20ad62597f8f61 Sep 17 07:27:39.548514 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:39.548488 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfe3853f_127c_4124_8df0_cdd6439052bb.slice/crio-5a273175467b2a8b47ec8c10e23052a92795980e6ef9d1df9c282c73109b326a WatchSource:0}: Error finding container 5a273175467b2a8b47ec8c10e23052a92795980e6ef9d1df9c282c73109b326a: Status 404 returned error can't find the container with id 5a273175467b2a8b47ec8c10e23052a92795980e6ef9d1df9c282c73109b326a Sep 17 07:27:39.550386 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:39.550365 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc42682fc_ab24_4492_a27b_c03a53648e5b.slice/crio-94a8ec66b4f5573bac034740db0ff6c598d4dcee95c7f8cb003d0a56b6b59617 WatchSource:0}: Error finding container 94a8ec66b4f5573bac034740db0ff6c598d4dcee95c7f8cb003d0a56b6b59617: Status 404 returned error can't find the container with id 94a8ec66b4f5573bac034740db0ff6c598d4dcee95c7f8cb003d0a56b6b59617 Sep 17 07:27:39.551495 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:39.551476 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9df1c13_2108_44d8_9137_730e35cf2165.slice/crio-c89c05644202f5d7028f901896258c868acb30bf00ebefd9f1e798ad740c51bc WatchSource:0}: Error finding container c89c05644202f5d7028f901896258c868acb30bf00ebefd9f1e798ad740c51bc: Status 404 returned error can't find the container with id c89c05644202f5d7028f901896258c868acb30bf00ebefd9f1e798ad740c51bc Sep 17 07:27:39.553036 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:39.552935 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2d890e8_ee86_4c3d_b7a7_81f5f57b3e43.slice/crio-8070f221d1ad54b512d7af5d85e483936f1351872efbee37d9c57c6db8ba5f33 WatchSource:0}: Error finding container 8070f221d1ad54b512d7af5d85e483936f1351872efbee37d9c57c6db8ba5f33: Status 404 returned error can't find the container with id 8070f221d1ad54b512d7af5d85e483936f1351872efbee37d9c57c6db8ba5f33 Sep 17 07:27:39.553975 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:39.553952 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9046265_84c2_44b5_9d4a_da21902077aa.slice/crio-0b297f4b350f53ccf5d1967e39525779b30120af2e1b7e352c56c18eb2db544a WatchSource:0}: Error finding container 0b297f4b350f53ccf5d1967e39525779b30120af2e1b7e352c56c18eb2db544a: Status 404 returned error can't find the container with id 0b297f4b350f53ccf5d1967e39525779b30120af2e1b7e352c56c18eb2db544a Sep 17 07:27:39.555178 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:27:39.554847 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf50cb832_b497_4a6a_bbe6_b758ec939688.slice/crio-64386c669dc79a117fd899c7dedbec9d5f7301a4b768bb4ea7e561bc0e560d22 WatchSource:0}: Error finding container 64386c669dc79a117fd899c7dedbec9d5f7301a4b768bb4ea7e561bc0e560d22: Status 404 returned error can't find the container with id 64386c669dc79a117fd899c7dedbec9d5f7301a4b768bb4ea7e561bc0e560d22 Sep 17 07:27:39.598120 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:39.598083 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 07:22:38 +0000 UTC" deadline="2028-05-16 13:25:23.762213316 +0000 UTC" Sep 17 07:27:39.598120 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:39.598120 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14573h57m44.164097569s" Sep 17 07:27:39.623227 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:39.623205 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lnpjx" Sep 17 07:27:39.623322 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:27:39.623306 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lnpjx" podUID="737e0243-b8eb-47e4-bf0f-c6b31e6ba787" Sep 17 07:27:39.633818 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:39.633789 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-fwdd5" event={"ID":"f50cb832-b497-4a6a-bbe6-b758ec939688","Type":"ContainerStarted","Data":"64386c669dc79a117fd899c7dedbec9d5f7301a4b768bb4ea7e561bc0e560d22"} Sep 17 07:27:39.634994 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:39.634970 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-wxwzn" event={"ID":"e9046265-84c2-44b5-9d4a-da21902077aa","Type":"ContainerStarted","Data":"0b297f4b350f53ccf5d1967e39525779b30120af2e1b7e352c56c18eb2db544a"} Sep 17 07:27:39.636113 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:39.636094 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xfggl" event={"ID":"d9df1c13-2108-44d8-9137-730e35cf2165","Type":"ContainerStarted","Data":"c89c05644202f5d7028f901896258c868acb30bf00ebefd9f1e798ad740c51bc"} Sep 17 07:27:39.637184 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:39.637152 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hnzgv" event={"ID":"eeb1a9ba-ece4-4e09-8bc7-de92eb38f029","Type":"ContainerStarted","Data":"839515f2dc9cfcac60584a910cb8de7b12e9e93365a922325c20ad62597f8f61"} Sep 17 07:27:39.639745 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:39.639725 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dtt5v" event={"ID":"d2d890e8-ee86-4c3d-b7a7-81f5f57b3e43","Type":"ContainerStarted","Data":"8070f221d1ad54b512d7af5d85e483936f1351872efbee37d9c57c6db8ba5f33"} Sep 17 07:27:39.640896 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:39.640841 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-c8gq4" event={"ID":"c42682fc-ab24-4492-a27b-c03a53648e5b","Type":"ContainerStarted","Data":"94a8ec66b4f5573bac034740db0ff6c598d4dcee95c7f8cb003d0a56b6b59617"} Sep 17 07:27:39.642037 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:39.642019 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-qbfvv" event={"ID":"bfe3853f-127c-4124-8df0-cdd6439052bb","Type":"ContainerStarted","Data":"5a273175467b2a8b47ec8c10e23052a92795980e6ef9d1df9c282c73109b326a"} Sep 17 07:27:39.643132 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:39.643112 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lh5z8" event={"ID":"6642bc78-77e5-44af-af76-a8f87d25c89a","Type":"ContainerStarted","Data":"f2d65b689056147e6612a4f94040c83f24929647bfcbc6a9e742c5bdaeb7444b"} Sep 17 07:27:39.644156 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:39.644132 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-zpwkj" event={"ID":"60c84ae0-1da1-4845-8d12-d9656a247588","Type":"ContainerStarted","Data":"99625c52fdb55aac6be727121140fec0bd4455bb1d4e64e5b3f1862044393b92"} Sep 17 07:27:39.645898 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:39.645876 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-100.ec2.internal" event={"ID":"9f09f680a0274abd6edf499bc1ec7a29","Type":"ContainerStarted","Data":"330ec4da6ce39aa3d97879ab26a1d632ebfa7fe1f77d1d78a12acb197b5a6afa"} Sep 17 07:27:39.665267 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:39.665223 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-100.ec2.internal" podStartSLOduration=2.665210669 podStartE2EDuration="2.665210669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 07:27:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 07:27:39.665010159 +0000 UTC m=+3.546044518" watchObservedRunningTime="2026-09-17 07:27:39.665210669 +0000 UTC m=+3.546245029" Sep 17 07:27:40.157986 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:40.157956 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/737e0243-b8eb-47e4-bf0f-c6b31e6ba787-metrics-certs\") pod \"network-metrics-daemon-lnpjx\" (UID: \"737e0243-b8eb-47e4-bf0f-c6b31e6ba787\") " pod="openshift-multus/network-metrics-daemon-lnpjx" Sep 17 07:27:40.158155 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:40.158114 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a236d1bd-81b1-4c9e-a14f-997204ad1b98-original-pull-secret\") pod \"global-pull-secret-syncer-lqcw7\" (UID: \"a236d1bd-81b1-4c9e-a14f-997204ad1b98\") " pod="kube-system/global-pull-secret-syncer-lqcw7" Sep 17 07:27:40.158260 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:27:40.158242 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 17 07:27:40.158325 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:27:40.158304 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a236d1bd-81b1-4c9e-a14f-997204ad1b98-original-pull-secret podName:a236d1bd-81b1-4c9e-a14f-997204ad1b98 nodeName:}" failed. No retries permitted until 2026-09-17 07:27:42.158285797 +0000 UTC m=+6.039320140 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a236d1bd-81b1-4c9e-a14f-997204ad1b98-original-pull-secret") pod "global-pull-secret-syncer-lqcw7" (UID: "a236d1bd-81b1-4c9e-a14f-997204ad1b98") : object "kube-system"/"original-pull-secret" not registered Sep 17 07:27:40.158676 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:27:40.158656 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 07:27:40.158763 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:27:40.158706 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/737e0243-b8eb-47e4-bf0f-c6b31e6ba787-metrics-certs podName:737e0243-b8eb-47e4-bf0f-c6b31e6ba787 nodeName:}" failed. No retries permitted until 2026-09-17 07:27:42.158692123 +0000 UTC m=+6.039726467 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/737e0243-b8eb-47e4-bf0f-c6b31e6ba787-metrics-certs") pod "network-metrics-daemon-lnpjx" (UID: "737e0243-b8eb-47e4-bf0f-c6b31e6ba787") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 07:27:40.258463 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:40.258431 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnz4n\" (UniqueName: \"kubernetes.io/projected/82555a36-653a-4106-a31e-b4da6445b838-kube-api-access-gnz4n\") pod \"network-check-target-bvxfq\" (UID: \"82555a36-653a-4106-a31e-b4da6445b838\") " pod="openshift-network-diagnostics/network-check-target-bvxfq" Sep 17 07:27:40.258659 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:27:40.258643 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 17 07:27:40.258744 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:27:40.258667 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 17 07:27:40.258744 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:27:40.258680 2578 projected.go:196] Error preparing data for projected volume kube-api-access-gnz4n for pod openshift-network-diagnostics/network-check-target-bvxfq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 07:27:40.258744 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:27:40.258731 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/82555a36-653a-4106-a31e-b4da6445b838-kube-api-access-gnz4n podName:82555a36-653a-4106-a31e-b4da6445b838 nodeName:}" failed. No retries permitted until 2026-09-17 07:27:42.258715306 +0000 UTC m=+6.139749649 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-gnz4n" (UniqueName: "kubernetes.io/projected/82555a36-653a-4106-a31e-b4da6445b838-kube-api-access-gnz4n") pod "network-check-target-bvxfq" (UID: "82555a36-653a-4106-a31e-b4da6445b838") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 07:27:40.598446 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:40.598356 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 07:22:38 +0000 UTC" deadline="2028-06-18 19:14:53.15357394 +0000 UTC" Sep 17 07:27:40.598446 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:40.598402 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15371h47m12.5551774s" Sep 17 07:27:40.624998 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:40.624970 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lqcw7" Sep 17 07:27:40.625163 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:27:40.625141 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lqcw7" podUID="a236d1bd-81b1-4c9e-a14f-997204ad1b98" Sep 17 07:27:40.625240 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:40.625176 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bvxfq" Sep 17 07:27:40.625328 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:27:40.625305 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bvxfq" podUID="82555a36-653a-4106-a31e-b4da6445b838" Sep 17 07:27:40.661675 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:40.661644 2578 generic.go:356] "Generic (PLEG): container finished" podID="4f0d3305f8ce3ecc2367db4b48860b3b" containerID="018d65f6ef01bb6e7591fd505de2cbd0871fbdfda80895fd2332aae45e6f24bb" exitCode=0 Sep 17 07:27:40.662633 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:40.662607 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-100.ec2.internal" event={"ID":"4f0d3305f8ce3ecc2367db4b48860b3b","Type":"ContainerDied","Data":"018d65f6ef01bb6e7591fd505de2cbd0871fbdfda80895fd2332aae45e6f24bb"} Sep 17 07:27:41.623985 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:41.623498 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lnpjx" Sep 17 07:27:41.623985 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:27:41.623636 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lnpjx" podUID="737e0243-b8eb-47e4-bf0f-c6b31e6ba787" Sep 17 07:27:41.689855 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:41.689145 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-100.ec2.internal" event={"ID":"4f0d3305f8ce3ecc2367db4b48860b3b","Type":"ContainerStarted","Data":"b8f0f38cf39bf857a9b59c36a84d5895eb6c557e5084f8e5047483729ffe002a"} Sep 17 07:27:41.709722 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:41.708605 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-100.ec2.internal" podStartSLOduration=4.708584222 podStartE2EDuration="4.708584222s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 07:27:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 07:27:41.708364502 +0000 UTC m=+5.589398862" watchObservedRunningTime="2026-09-17 07:27:41.708584222 +0000 UTC m=+5.589618580" Sep 17 07:27:42.173623 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:42.173374 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a236d1bd-81b1-4c9e-a14f-997204ad1b98-original-pull-secret\") pod \"global-pull-secret-syncer-lqcw7\" (UID: \"a236d1bd-81b1-4c9e-a14f-997204ad1b98\") " pod="kube-system/global-pull-secret-syncer-lqcw7" Sep 17 07:27:42.173819 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:42.173653 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/737e0243-b8eb-47e4-bf0f-c6b31e6ba787-metrics-certs\") pod \"network-metrics-daemon-lnpjx\" (UID: \"737e0243-b8eb-47e4-bf0f-c6b31e6ba787\") " pod="openshift-multus/network-metrics-daemon-lnpjx" Sep 17 07:27:42.173819 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:27:42.173545 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 17 07:27:42.173950 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:27:42.173854 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a236d1bd-81b1-4c9e-a14f-997204ad1b98-original-pull-secret podName:a236d1bd-81b1-4c9e-a14f-997204ad1b98 nodeName:}" failed. No retries permitted until 2026-09-17 07:27:46.173837062 +0000 UTC m=+10.054871399 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a236d1bd-81b1-4c9e-a14f-997204ad1b98-original-pull-secret") pod "global-pull-secret-syncer-lqcw7" (UID: "a236d1bd-81b1-4c9e-a14f-997204ad1b98") : object "kube-system"/"original-pull-secret" not registered Sep 17 07:27:42.173950 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:27:42.173798 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 07:27:42.173950 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:27:42.173921 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/737e0243-b8eb-47e4-bf0f-c6b31e6ba787-metrics-certs podName:737e0243-b8eb-47e4-bf0f-c6b31e6ba787 nodeName:}" failed. No retries permitted until 2026-09-17 07:27:46.173910349 +0000 UTC m=+10.054944692 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/737e0243-b8eb-47e4-bf0f-c6b31e6ba787-metrics-certs") pod "network-metrics-daemon-lnpjx" (UID: "737e0243-b8eb-47e4-bf0f-c6b31e6ba787") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 07:27:42.274188 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:42.274130 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnz4n\" (UniqueName: \"kubernetes.io/projected/82555a36-653a-4106-a31e-b4da6445b838-kube-api-access-gnz4n\") pod \"network-check-target-bvxfq\" (UID: \"82555a36-653a-4106-a31e-b4da6445b838\") " pod="openshift-network-diagnostics/network-check-target-bvxfq" Sep 17 07:27:42.274371 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:27:42.274344 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 17 07:27:42.274371 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:27:42.274363 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 17 07:27:42.274491 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:27:42.274375 2578 projected.go:196] Error preparing data for projected volume kube-api-access-gnz4n for pod openshift-network-diagnostics/network-check-target-bvxfq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 07:27:42.274491 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:27:42.274465 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/82555a36-653a-4106-a31e-b4da6445b838-kube-api-access-gnz4n podName:82555a36-653a-4106-a31e-b4da6445b838 nodeName:}" failed. No retries permitted until 2026-09-17 07:27:46.27444638 +0000 UTC m=+10.155480723 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-gnz4n" (UniqueName: "kubernetes.io/projected/82555a36-653a-4106-a31e-b4da6445b838-kube-api-access-gnz4n") pod "network-check-target-bvxfq" (UID: "82555a36-653a-4106-a31e-b4da6445b838") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 07:27:42.624511 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:42.623969 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lqcw7" Sep 17 07:27:42.624511 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:42.623981 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bvxfq" Sep 17 07:27:42.624511 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:27:42.624099 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lqcw7" podUID="a236d1bd-81b1-4c9e-a14f-997204ad1b98" Sep 17 07:27:42.624511 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:27:42.624216 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bvxfq" podUID="82555a36-653a-4106-a31e-b4da6445b838" Sep 17 07:27:43.624007 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:43.623977 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lnpjx" Sep 17 07:27:43.624169 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:27:43.624127 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lnpjx" podUID="737e0243-b8eb-47e4-bf0f-c6b31e6ba787" Sep 17 07:27:44.624001 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:44.623968 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bvxfq" Sep 17 07:27:44.624479 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:27:44.624099 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bvxfq" podUID="82555a36-653a-4106-a31e-b4da6445b838" Sep 17 07:27:44.624479 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:44.624187 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lqcw7" Sep 17 07:27:44.624479 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:27:44.624292 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lqcw7" podUID="a236d1bd-81b1-4c9e-a14f-997204ad1b98" Sep 17 07:27:45.623869 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:45.623616 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lnpjx" Sep 17 07:27:45.623869 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:27:45.623757 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lnpjx" podUID="737e0243-b8eb-47e4-bf0f-c6b31e6ba787" Sep 17 07:27:46.207391 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:46.207357 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a236d1bd-81b1-4c9e-a14f-997204ad1b98-original-pull-secret\") pod \"global-pull-secret-syncer-lqcw7\" (UID: \"a236d1bd-81b1-4c9e-a14f-997204ad1b98\") " pod="kube-system/global-pull-secret-syncer-lqcw7" Sep 17 07:27:46.207580 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:46.207412 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/737e0243-b8eb-47e4-bf0f-c6b31e6ba787-metrics-certs\") pod \"network-metrics-daemon-lnpjx\" (UID: \"737e0243-b8eb-47e4-bf0f-c6b31e6ba787\") " pod="openshift-multus/network-metrics-daemon-lnpjx" Sep 17 07:27:46.207580 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:27:46.207508 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 17 07:27:46.207580 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:27:46.207518 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 07:27:46.207580 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:27:46.207565 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a236d1bd-81b1-4c9e-a14f-997204ad1b98-original-pull-secret podName:a236d1bd-81b1-4c9e-a14f-997204ad1b98 nodeName:}" failed. No retries permitted until 2026-09-17 07:27:54.207549652 +0000 UTC m=+18.088583994 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a236d1bd-81b1-4c9e-a14f-997204ad1b98-original-pull-secret") pod "global-pull-secret-syncer-lqcw7" (UID: "a236d1bd-81b1-4c9e-a14f-997204ad1b98") : object "kube-system"/"original-pull-secret" not registered Sep 17 07:27:46.207580 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:27:46.207580 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/737e0243-b8eb-47e4-bf0f-c6b31e6ba787-metrics-certs podName:737e0243-b8eb-47e4-bf0f-c6b31e6ba787 nodeName:}" failed. No retries permitted until 2026-09-17 07:27:54.20757292 +0000 UTC m=+18.088607256 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/737e0243-b8eb-47e4-bf0f-c6b31e6ba787-metrics-certs") pod "network-metrics-daemon-lnpjx" (UID: "737e0243-b8eb-47e4-bf0f-c6b31e6ba787") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 07:27:46.308533 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:46.308448 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnz4n\" (UniqueName: \"kubernetes.io/projected/82555a36-653a-4106-a31e-b4da6445b838-kube-api-access-gnz4n\") pod \"network-check-target-bvxfq\" (UID: \"82555a36-653a-4106-a31e-b4da6445b838\") " pod="openshift-network-diagnostics/network-check-target-bvxfq" Sep 17 07:27:46.308700 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:27:46.308626 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 17 07:27:46.308700 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:27:46.308652 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 17 07:27:46.308700 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:27:46.308665 2578 projected.go:196] Error preparing data for projected volume kube-api-access-gnz4n for pod openshift-network-diagnostics/network-check-target-bvxfq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 07:27:46.308890 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:27:46.308716 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/82555a36-653a-4106-a31e-b4da6445b838-kube-api-access-gnz4n podName:82555a36-653a-4106-a31e-b4da6445b838 nodeName:}" failed. No retries permitted until 2026-09-17 07:27:54.308701379 +0000 UTC m=+18.189735716 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-gnz4n" (UniqueName: "kubernetes.io/projected/82555a36-653a-4106-a31e-b4da6445b838-kube-api-access-gnz4n") pod "network-check-target-bvxfq" (UID: "82555a36-653a-4106-a31e-b4da6445b838") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 07:27:46.625039 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:46.624963 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lqcw7" Sep 17 07:27:46.625465 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:27:46.625073 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lqcw7" podUID="a236d1bd-81b1-4c9e-a14f-997204ad1b98" Sep 17 07:27:46.625465 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:46.625420 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bvxfq" Sep 17 07:27:46.625573 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:27:46.625509 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bvxfq" podUID="82555a36-653a-4106-a31e-b4da6445b838" Sep 17 07:27:47.624018 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:47.623753 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lnpjx" Sep 17 07:27:47.624018 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:27:47.623910 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lnpjx" podUID="737e0243-b8eb-47e4-bf0f-c6b31e6ba787" Sep 17 07:27:48.624158 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:48.624126 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lqcw7" Sep 17 07:27:48.624605 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:48.624136 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bvxfq" Sep 17 07:27:48.624605 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:27:48.624238 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lqcw7" podUID="a236d1bd-81b1-4c9e-a14f-997204ad1b98" Sep 17 07:27:48.624605 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:27:48.624343 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bvxfq" podUID="82555a36-653a-4106-a31e-b4da6445b838" Sep 17 07:27:49.624224 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:49.624198 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lnpjx" Sep 17 07:27:49.624567 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:27:49.624292 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lnpjx" podUID="737e0243-b8eb-47e4-bf0f-c6b31e6ba787" Sep 17 07:27:50.624141 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:50.624099 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lqcw7" Sep 17 07:27:50.624322 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:50.624149 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bvxfq" Sep 17 07:27:50.624322 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:27:50.624229 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lqcw7" podUID="a236d1bd-81b1-4c9e-a14f-997204ad1b98" Sep 17 07:27:50.624735 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:27:50.624354 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bvxfq" podUID="82555a36-653a-4106-a31e-b4da6445b838" Sep 17 07:27:51.623232 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:51.623202 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lnpjx" Sep 17 07:27:51.623415 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:27:51.623320 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lnpjx" podUID="737e0243-b8eb-47e4-bf0f-c6b31e6ba787" Sep 17 07:27:52.624058 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:52.624023 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lqcw7" Sep 17 07:27:52.624462 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:52.624069 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bvxfq" Sep 17 07:27:52.624462 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:27:52.624151 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lqcw7" podUID="a236d1bd-81b1-4c9e-a14f-997204ad1b98" Sep 17 07:27:52.624462 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:27:52.624305 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bvxfq" podUID="82555a36-653a-4106-a31e-b4da6445b838" Sep 17 07:27:53.624123 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:53.624091 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lnpjx" Sep 17 07:27:53.624581 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:27:53.624203 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lnpjx" podUID="737e0243-b8eb-47e4-bf0f-c6b31e6ba787" Sep 17 07:27:54.263993 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:54.263951 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a236d1bd-81b1-4c9e-a14f-997204ad1b98-original-pull-secret\") pod \"global-pull-secret-syncer-lqcw7\" (UID: \"a236d1bd-81b1-4c9e-a14f-997204ad1b98\") " pod="kube-system/global-pull-secret-syncer-lqcw7" Sep 17 07:27:54.264205 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:54.264019 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/737e0243-b8eb-47e4-bf0f-c6b31e6ba787-metrics-certs\") pod \"network-metrics-daemon-lnpjx\" (UID: \"737e0243-b8eb-47e4-bf0f-c6b31e6ba787\") " pod="openshift-multus/network-metrics-daemon-lnpjx" Sep 17 07:27:54.264205 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:27:54.264118 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 17 07:27:54.264205 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:27:54.264203 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a236d1bd-81b1-4c9e-a14f-997204ad1b98-original-pull-secret podName:a236d1bd-81b1-4c9e-a14f-997204ad1b98 nodeName:}" failed. No retries permitted until 2026-09-17 07:28:10.264187657 +0000 UTC m=+34.145221998 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a236d1bd-81b1-4c9e-a14f-997204ad1b98-original-pull-secret") pod "global-pull-secret-syncer-lqcw7" (UID: "a236d1bd-81b1-4c9e-a14f-997204ad1b98") : object "kube-system"/"original-pull-secret" not registered Sep 17 07:27:54.264370 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:27:54.264126 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 07:27:54.264370 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:27:54.264272 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/737e0243-b8eb-47e4-bf0f-c6b31e6ba787-metrics-certs podName:737e0243-b8eb-47e4-bf0f-c6b31e6ba787 nodeName:}" failed. No retries permitted until 2026-09-17 07:28:10.264254288 +0000 UTC m=+34.145288649 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/737e0243-b8eb-47e4-bf0f-c6b31e6ba787-metrics-certs") pod "network-metrics-daemon-lnpjx" (UID: "737e0243-b8eb-47e4-bf0f-c6b31e6ba787") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 07:27:54.364520 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:54.364489 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnz4n\" (UniqueName: \"kubernetes.io/projected/82555a36-653a-4106-a31e-b4da6445b838-kube-api-access-gnz4n\") pod \"network-check-target-bvxfq\" (UID: \"82555a36-653a-4106-a31e-b4da6445b838\") " pod="openshift-network-diagnostics/network-check-target-bvxfq" Sep 17 07:27:54.364690 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:27:54.364654 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 17 07:27:54.364690 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:27:54.364678 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 17 07:27:54.364690 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:27:54.364691 2578 projected.go:196] Error preparing data for projected volume kube-api-access-gnz4n for pod openshift-network-diagnostics/network-check-target-bvxfq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 07:27:54.364891 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:27:54.364744 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/82555a36-653a-4106-a31e-b4da6445b838-kube-api-access-gnz4n podName:82555a36-653a-4106-a31e-b4da6445b838 nodeName:}" failed. No retries permitted until 2026-09-17 07:28:10.364730792 +0000 UTC m=+34.245765134 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-gnz4n" (UniqueName: "kubernetes.io/projected/82555a36-653a-4106-a31e-b4da6445b838-kube-api-access-gnz4n") pod "network-check-target-bvxfq" (UID: "82555a36-653a-4106-a31e-b4da6445b838") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 07:27:54.623662 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:54.623590 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bvxfq" Sep 17 07:27:54.623821 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:27:54.623700 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bvxfq" podUID="82555a36-653a-4106-a31e-b4da6445b838" Sep 17 07:27:54.623821 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:54.623756 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lqcw7" Sep 17 07:27:54.623944 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:27:54.623839 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lqcw7" podUID="a236d1bd-81b1-4c9e-a14f-997204ad1b98" Sep 17 07:27:55.624079 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:55.624037 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lnpjx" Sep 17 07:27:55.624546 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:27:55.624180 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lnpjx" podUID="737e0243-b8eb-47e4-bf0f-c6b31e6ba787" Sep 17 07:27:56.624305 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:56.624274 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lqcw7" Sep 17 07:27:56.624749 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:27:56.624380 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lqcw7" podUID="a236d1bd-81b1-4c9e-a14f-997204ad1b98" Sep 17 07:27:56.624749 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:56.624462 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bvxfq" Sep 17 07:27:56.624749 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:27:56.624555 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bvxfq" podUID="82555a36-653a-4106-a31e-b4da6445b838" Sep 17 07:27:57.623995 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:57.623813 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lnpjx" Sep 17 07:27:57.624105 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:27:57.624069 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lnpjx" podUID="737e0243-b8eb-47e4-bf0f-c6b31e6ba787" Sep 17 07:27:57.722745 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:57.722718 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-zpwkj" event={"ID":"60c84ae0-1da1-4845-8d12-d9656a247588","Type":"ContainerStarted","Data":"347286a41443c1ddec3c2409fc18ac16f296a4f0c98a4541df933d6176f2f30d"} Sep 17 07:27:57.725443 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:57.725416 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-fwdd5" event={"ID":"f50cb832-b497-4a6a-bbe6-b758ec939688","Type":"ContainerStarted","Data":"c48dcf203957513b733c0e4acaa61a54250042ad551634c9c0e13c6cd61703fc"} Sep 17 07:27:57.727211 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:57.727192 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-wxwzn" event={"ID":"e9046265-84c2-44b5-9d4a-da21902077aa","Type":"ContainerStarted","Data":"2f2b95de0f09f5cc9c03f91cdc1dd71325a680ee9b7d7b02a327cca579ac66aa"} Sep 17 07:27:57.728585 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:57.728554 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xfggl" event={"ID":"d9df1c13-2108-44d8-9137-730e35cf2165","Type":"ContainerStarted","Data":"d1e350eeb6289fb83e55683b5c8790921f7de2e9beb791a7c89b74483621cee5"} Sep 17 07:27:57.730844 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:57.730819 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hnzgv" event={"ID":"eeb1a9ba-ece4-4e09-8bc7-de92eb38f029","Type":"ContainerStarted","Data":"d9f160243d65267ac92049ce55b41933273deaed493fb05370b600d6fd0c7544"} Sep 17 07:27:57.730950 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:57.730902 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hnzgv" event={"ID":"eeb1a9ba-ece4-4e09-8bc7-de92eb38f029","Type":"ContainerStarted","Data":"170a30e92eb2a6881aa6bcb49a751e6571e8c3f11fdb4543ce1dbc885ab465be"} Sep 17 07:27:57.730950 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:57.730945 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hnzgv" event={"ID":"eeb1a9ba-ece4-4e09-8bc7-de92eb38f029","Type":"ContainerStarted","Data":"6ad5dd1bfcc4c5924bfcaf7f35a472a24892b747532494c6dda0a1459e2db474"} Sep 17 07:27:57.731053 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:57.730975 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hnzgv" event={"ID":"eeb1a9ba-ece4-4e09-8bc7-de92eb38f029","Type":"ContainerStarted","Data":"18f1e074c3b0c75796b0dd1a9ac86cfd427fc74d909f451ba31d32cb9e185838"} Sep 17 07:27:57.732101 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:57.732083 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dtt5v" event={"ID":"d2d890e8-ee86-4c3d-b7a7-81f5f57b3e43","Type":"ContainerStarted","Data":"c6849bae1aaf8a703a5e7b7b0737e50b6785a30f788fb71e86ed854a06b2b4cf"} Sep 17 07:27:57.733361 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:57.733342 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-qbfvv" event={"ID":"bfe3853f-127c-4124-8df0-cdd6439052bb","Type":"ContainerStarted","Data":"4e0a69d8a1b22b2490f30842c59a8edf7f5b8e205c79558a84985ad238b9cb30"} Sep 17 07:27:57.734665 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:57.734643 2578 generic.go:356] "Generic (PLEG): container finished" podID="6642bc78-77e5-44af-af76-a8f87d25c89a" containerID="a69f886fecfafddd79a3eac7292b94c5b1f04144e8faaabacb545607c232ae6a" exitCode=0 Sep 17 07:27:57.734734 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:57.734673 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lh5z8" event={"ID":"6642bc78-77e5-44af-af76-a8f87d25c89a","Type":"ContainerDied","Data":"a69f886fecfafddd79a3eac7292b94c5b1f04144e8faaabacb545607c232ae6a"} Sep 17 07:27:57.747996 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:57.747930 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-zpwkj" podStartSLOduration=3.883708784 podStartE2EDuration="21.747919671s" totalImagesPullingTime="17.864210887s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 07:27:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 07:27:57.747578389 +0000 UTC m=+21.628612747" watchObservedRunningTime="2026-09-17 07:27:57.747919671 +0000 UTC m=+21.628954030" Sep 17 07:27:57.764415 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:57.764378 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-xfggl" podStartSLOduration=4.070900816 podStartE2EDuration="21.764368331s" totalImagesPullingTime="17.693467515s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 07:27:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 07:27:57.764023872 +0000 UTC m=+21.645058231" watchObservedRunningTime="2026-09-17 07:27:57.764368331 +0000 UTC m=+21.645402689" Sep 17 07:27:57.787615 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:57.787582 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-wxwzn" podStartSLOduration=4.09495138 podStartE2EDuration="21.787570548s" totalImagesPullingTime="17.692619168s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 07:27:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 07:27:57.786671663 +0000 UTC m=+21.667706026" watchObservedRunningTime="2026-09-17 07:27:57.787570548 +0000 UTC m=+21.668604906" Sep 17 07:27:57.843844 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:57.843791 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-fwdd5" podStartSLOduration=4.153880594 podStartE2EDuration="21.843771572s" totalImagesPullingTime="17.689890978s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 07:27:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 07:27:57.808451224 +0000 UTC m=+21.689485583" watchObservedRunningTime="2026-09-17 07:27:57.843771572 +0000 UTC m=+21.724805950" Sep 17 07:27:57.864475 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:57.864391 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-qbfvv" podStartSLOduration=6.71452108 podStartE2EDuration="20.864373304s" totalImagesPullingTime="14.149852224s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 07:27:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 07:27:57.864103369 +0000 UTC m=+21.745137728" watchObservedRunningTime="2026-09-17 07:27:57.864373304 +0000 UTC m=+21.745407665" Sep 17 07:27:58.624087 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:58.624059 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lqcw7" Sep 17 07:27:58.624087 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:58.624083 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bvxfq" Sep 17 07:27:58.624430 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:27:58.624174 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lqcw7" podUID="a236d1bd-81b1-4c9e-a14f-997204ad1b98" Sep 17 07:27:58.624430 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:27:58.624296 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bvxfq" podUID="82555a36-653a-4106-a31e-b4da6445b838" Sep 17 07:27:58.738389 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:58.738355 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-c8gq4" event={"ID":"c42682fc-ab24-4492-a27b-c03a53648e5b","Type":"ContainerStarted","Data":"0a17b3ab063ad450e0bcea4f8367a0bd29fb7963d9221eb45a84e8677a4dfe35"} Sep 17 07:27:58.741341 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:58.741314 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hnzgv" event={"ID":"eeb1a9ba-ece4-4e09-8bc7-de92eb38f029","Type":"ContainerStarted","Data":"9bc76097dd13044d43d2e47feaf4e3a0a7c917298d86485862df8ddb0ccbd043"} Sep 17 07:27:58.741464 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:58.741382 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hnzgv" event={"ID":"eeb1a9ba-ece4-4e09-8bc7-de92eb38f029","Type":"ContainerStarted","Data":"fcd7e06114e9490158e14f7cd1182ce726461f6d649dafce331b283a5d867937"} Sep 17 07:27:58.913372 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:58.913350 2578 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 17 07:27:59.599345 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:59.599245 2578 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-17T07:27:58.91336812Z","UUID":"c0fb29f0-9147-4616-a3a2-a34525eb89c4","Handler":null,"Name":"","Endpoint":""} Sep 17 07:27:59.601539 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:59.601235 2578 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 17 07:27:59.601539 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:59.601277 2578 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 17 07:27:59.623882 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:59.623804 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lnpjx" Sep 17 07:27:59.624030 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:27:59.623986 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lnpjx" podUID="737e0243-b8eb-47e4-bf0f-c6b31e6ba787" Sep 17 07:27:59.644135 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:59.644099 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-qbfvv" Sep 17 07:27:59.644658 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:59.644636 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-qbfvv" Sep 17 07:27:59.671731 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:59.671682 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-c8gq4" podStartSLOduration=4.977061051 podStartE2EDuration="22.67166845s" totalImagesPullingTime="17.694607399s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 07:27:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 07:27:58.755913817 +0000 UTC m=+22.636948176" watchObservedRunningTime="2026-09-17 07:27:59.67166845 +0000 UTC m=+23.552702811" Sep 17 07:27:59.746282 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:59.746247 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dtt5v" event={"ID":"d2d890e8-ee86-4c3d-b7a7-81f5f57b3e43","Type":"ContainerStarted","Data":"05f60cccd27f202bc10e6192116a65c238f247fb3f417c4c01107f16af2b1474"} Sep 17 07:27:59.746832 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:59.746780 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-qbfvv" Sep 17 07:27:59.747241 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:27:59.747219 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-qbfvv" Sep 17 07:28:00.623936 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:00.623855 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bvxfq" Sep 17 07:28:00.624070 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:28:00.623962 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bvxfq" podUID="82555a36-653a-4106-a31e-b4da6445b838" Sep 17 07:28:00.624070 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:00.623854 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lqcw7" Sep 17 07:28:00.624070 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:28:00.624027 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lqcw7" podUID="a236d1bd-81b1-4c9e-a14f-997204ad1b98" Sep 17 07:28:00.752349 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:00.752317 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hnzgv" event={"ID":"eeb1a9ba-ece4-4e09-8bc7-de92eb38f029","Type":"ContainerStarted","Data":"545df16e52f42c2e57b730c03afc478772fcd07502623666c91a7890f4071afb"} Sep 17 07:28:00.754558 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:00.754533 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dtt5v" event={"ID":"d2d890e8-ee86-4c3d-b7a7-81f5f57b3e43","Type":"ContainerStarted","Data":"4c8b18e320eb11a7cb95dcdefc5b6f015a24128b1274624ba69fc65a152f6cfd"} Sep 17 07:28:01.623904 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:01.623871 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lnpjx" Sep 17 07:28:01.624119 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:28:01.623986 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lnpjx" podUID="737e0243-b8eb-47e4-bf0f-c6b31e6ba787" Sep 17 07:28:02.624083 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:02.623834 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lqcw7" Sep 17 07:28:02.624480 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:02.623916 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bvxfq" Sep 17 07:28:02.624480 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:28:02.624125 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lqcw7" podUID="a236d1bd-81b1-4c9e-a14f-997204ad1b98" Sep 17 07:28:02.624480 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:28:02.624193 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bvxfq" podUID="82555a36-653a-4106-a31e-b4da6445b838" Sep 17 07:28:02.760694 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:02.760663 2578 generic.go:356] "Generic (PLEG): container finished" podID="6642bc78-77e5-44af-af76-a8f87d25c89a" containerID="9e5aa305838d3fd7c2a21ad6f325e4ab888da5b2668b12e9f0770379ec433b27" exitCode=0 Sep 17 07:28:02.760897 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:02.760745 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lh5z8" event={"ID":"6642bc78-77e5-44af-af76-a8f87d25c89a","Type":"ContainerDied","Data":"9e5aa305838d3fd7c2a21ad6f325e4ab888da5b2668b12e9f0770379ec433b27"} Sep 17 07:28:02.764531 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:02.764509 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hnzgv" event={"ID":"eeb1a9ba-ece4-4e09-8bc7-de92eb38f029","Type":"ContainerStarted","Data":"fe8a80ba679f5b763f9181b79d53e4996322953c0db7a2b5067438dba17cb9b9"} Sep 17 07:28:02.764783 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:02.764759 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hnzgv" Sep 17 07:28:02.764893 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:02.764816 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hnzgv" Sep 17 07:28:02.764893 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:02.764855 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hnzgv" Sep 17 07:28:02.779534 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:02.779513 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hnzgv" Sep 17 07:28:02.779646 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:02.779638 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hnzgv" Sep 17 07:28:02.786978 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:02.786937 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dtt5v" podStartSLOduration=5.459719067 podStartE2EDuration="25.786924896s" totalImagesPullingTime="20.327205829s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 07:27:37 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 07:28:00.786883074 +0000 UTC m=+24.667917461" watchObservedRunningTime="2026-09-17 07:28:02.786924896 +0000 UTC m=+26.667959254" Sep 17 07:28:02.820207 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:02.820164 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-hnzgv" podStartSLOduration=9.08052221 podStartE2EDuration="26.8201518s" totalImagesPullingTime="17.73962959s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 07:27:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 07:28:02.81966135 +0000 UTC m=+26.700695710" watchObservedRunningTime="2026-09-17 07:28:02.8201518 +0000 UTC m=+26.701186158" Sep 17 07:28:03.623552 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:03.623514 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lnpjx" Sep 17 07:28:03.623716 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:28:03.623666 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lnpjx" podUID="737e0243-b8eb-47e4-bf0f-c6b31e6ba787" Sep 17 07:28:03.768993 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:03.768964 2578 generic.go:356] "Generic (PLEG): container finished" podID="6642bc78-77e5-44af-af76-a8f87d25c89a" containerID="9e5b907f1f5c6b3e1cb357acdc9ab1c5b381802162064e95faa3914b5f24bc4a" exitCode=0 Sep 17 07:28:03.769508 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:03.769037 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lh5z8" event={"ID":"6642bc78-77e5-44af-af76-a8f87d25c89a","Type":"ContainerDied","Data":"9e5b907f1f5c6b3e1cb357acdc9ab1c5b381802162064e95faa3914b5f24bc4a"} Sep 17 07:28:04.199707 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:04.199672 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-lqcw7"] Sep 17 07:28:04.199903 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:04.199824 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lqcw7" Sep 17 07:28:04.199970 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:28:04.199950 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lqcw7" podUID="a236d1bd-81b1-4c9e-a14f-997204ad1b98" Sep 17 07:28:04.201949 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:04.201919 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-bvxfq"] Sep 17 07:28:04.202042 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:04.202000 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bvxfq" Sep 17 07:28:04.202093 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:28:04.202063 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bvxfq" podUID="82555a36-653a-4106-a31e-b4da6445b838" Sep 17 07:28:04.205176 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:04.205151 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-lnpjx"] Sep 17 07:28:04.205247 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:04.205235 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lnpjx" Sep 17 07:28:04.205319 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:28:04.205302 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lnpjx" podUID="737e0243-b8eb-47e4-bf0f-c6b31e6ba787" Sep 17 07:28:04.773138 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:04.773114 2578 generic.go:356] "Generic (PLEG): container finished" podID="6642bc78-77e5-44af-af76-a8f87d25c89a" containerID="603ea5e584f2eac6770794c0ebc534a772680433d60ebc19d0171fa73e8f90ae" exitCode=0 Sep 17 07:28:04.773413 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:04.773201 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lh5z8" event={"ID":"6642bc78-77e5-44af-af76-a8f87d25c89a","Type":"ContainerDied","Data":"603ea5e584f2eac6770794c0ebc534a772680433d60ebc19d0171fa73e8f90ae"} Sep 17 07:28:05.623512 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:05.623436 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lnpjx" Sep 17 07:28:05.623512 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:05.623468 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lqcw7" Sep 17 07:28:05.623736 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:28:05.623551 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lnpjx" podUID="737e0243-b8eb-47e4-bf0f-c6b31e6ba787" Sep 17 07:28:05.623736 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:28:05.623692 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lqcw7" podUID="a236d1bd-81b1-4c9e-a14f-997204ad1b98" Sep 17 07:28:06.627897 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:06.627657 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bvxfq" Sep 17 07:28:06.628307 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:28:06.627948 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bvxfq" podUID="82555a36-653a-4106-a31e-b4da6445b838" Sep 17 07:28:07.624065 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:07.624033 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lqcw7" Sep 17 07:28:07.624203 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:07.624044 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lnpjx" Sep 17 07:28:07.624203 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:28:07.624130 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lqcw7" podUID="a236d1bd-81b1-4c9e-a14f-997204ad1b98" Sep 17 07:28:07.624283 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:28:07.624217 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lnpjx" podUID="737e0243-b8eb-47e4-bf0f-c6b31e6ba787" Sep 17 07:28:08.623499 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:08.623426 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bvxfq" Sep 17 07:28:08.623930 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:28:08.623540 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bvxfq" podUID="82555a36-653a-4106-a31e-b4da6445b838" Sep 17 07:28:09.624060 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:09.624037 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lnpjx" Sep 17 07:28:09.624394 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:28:09.624169 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lnpjx" podUID="737e0243-b8eb-47e4-bf0f-c6b31e6ba787" Sep 17 07:28:09.624394 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:09.624207 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lqcw7" Sep 17 07:28:09.624394 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:28:09.624292 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lqcw7" podUID="a236d1bd-81b1-4c9e-a14f-997204ad1b98" Sep 17 07:28:09.788178 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:09.788121 2578 generic.go:356] "Generic (PLEG): container finished" podID="6642bc78-77e5-44af-af76-a8f87d25c89a" containerID="68061b5a32e441d5fd9c4733ff49fed52504672f4de4c3f4c74a5d203fe37ce7" exitCode=0 Sep 17 07:28:09.788178 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:09.788160 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lh5z8" event={"ID":"6642bc78-77e5-44af-af76-a8f87d25c89a","Type":"ContainerDied","Data":"68061b5a32e441d5fd9c4733ff49fed52504672f4de4c3f4c74a5d203fe37ce7"} Sep 17 07:28:09.938275 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:09.938254 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-100.ec2.internal" event="NodeReady" Sep 17 07:28:09.938368 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:09.938359 2578 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 17 07:28:09.982450 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:09.982423 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-648fcb787c-kxf78"] Sep 17 07:28:09.988587 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:09.988571 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-648fcb787c-kxf78" Sep 17 07:28:09.991363 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:09.991342 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 17 07:28:09.991363 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:09.991353 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 17 07:28:09.991538 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:09.991402 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-vtldk\"" Sep 17 07:28:09.991538 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:09.991412 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 17 07:28:10.002632 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:10.002608 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 17 07:28:10.003460 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:10.003441 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-648fcb787c-kxf78"] Sep 17 07:28:10.010386 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:10.010365 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-nfjvh"] Sep 17 07:28:10.024486 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:10.024459 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-kcfcf"] Sep 17 07:28:10.024588 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:10.024573 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-nfjvh" Sep 17 07:28:10.028515 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:10.028498 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-bnzll\"" Sep 17 07:28:10.028801 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:10.028781 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 17 07:28:10.028801 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:10.028795 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 17 07:28:10.034558 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:10.034540 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-kcfcf"] Sep 17 07:28:10.034670 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:10.034575 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-nfjvh"] Sep 17 07:28:10.034670 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:10.034647 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-kcfcf" Sep 17 07:28:10.037669 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:10.037651 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-jvgrq\"" Sep 17 07:28:10.037791 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:10.037695 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 17 07:28:10.037791 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:10.037721 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 17 07:28:10.037791 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:10.037769 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 17 07:28:10.078722 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:10.078701 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfwmj\" (UniqueName: \"kubernetes.io/projected/44eefc80-34c2-47a2-ab48-e7e685f7dfe3-kube-api-access-dfwmj\") pod \"image-registry-648fcb787c-kxf78\" (UID: \"44eefc80-34c2-47a2-ab48-e7e685f7dfe3\") " pod="openshift-image-registry/image-registry-648fcb787c-kxf78" Sep 17 07:28:10.078832 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:10.078763 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/44eefc80-34c2-47a2-ab48-e7e685f7dfe3-image-registry-private-configuration\") pod \"image-registry-648fcb787c-kxf78\" (UID: \"44eefc80-34c2-47a2-ab48-e7e685f7dfe3\") " pod="openshift-image-registry/image-registry-648fcb787c-kxf78" Sep 17 07:28:10.078832 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:10.078793 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/44eefc80-34c2-47a2-ab48-e7e685f7dfe3-registry-tls\") pod \"image-registry-648fcb787c-kxf78\" (UID: \"44eefc80-34c2-47a2-ab48-e7e685f7dfe3\") " pod="openshift-image-registry/image-registry-648fcb787c-kxf78" Sep 17 07:28:10.078832 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:10.078815 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/44eefc80-34c2-47a2-ab48-e7e685f7dfe3-ca-trust-extracted\") pod \"image-registry-648fcb787c-kxf78\" (UID: \"44eefc80-34c2-47a2-ab48-e7e685f7dfe3\") " pod="openshift-image-registry/image-registry-648fcb787c-kxf78" Sep 17 07:28:10.078949 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:10.078840 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/44eefc80-34c2-47a2-ab48-e7e685f7dfe3-registry-certificates\") pod \"image-registry-648fcb787c-kxf78\" (UID: \"44eefc80-34c2-47a2-ab48-e7e685f7dfe3\") " pod="openshift-image-registry/image-registry-648fcb787c-kxf78" Sep 17 07:28:10.078949 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:10.078856 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/44eefc80-34c2-47a2-ab48-e7e685f7dfe3-installation-pull-secrets\") pod \"image-registry-648fcb787c-kxf78\" (UID: \"44eefc80-34c2-47a2-ab48-e7e685f7dfe3\") " pod="openshift-image-registry/image-registry-648fcb787c-kxf78" Sep 17 07:28:10.078949 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:10.078922 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/44eefc80-34c2-47a2-ab48-e7e685f7dfe3-trusted-ca\") pod \"image-registry-648fcb787c-kxf78\" (UID: \"44eefc80-34c2-47a2-ab48-e7e685f7dfe3\") " pod="openshift-image-registry/image-registry-648fcb787c-kxf78" Sep 17 07:28:10.078949 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:10.078945 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/44eefc80-34c2-47a2-ab48-e7e685f7dfe3-bound-sa-token\") pod \"image-registry-648fcb787c-kxf78\" (UID: \"44eefc80-34c2-47a2-ab48-e7e685f7dfe3\") " pod="openshift-image-registry/image-registry-648fcb787c-kxf78" Sep 17 07:28:10.180129 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:10.180104 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfwmj\" (UniqueName: \"kubernetes.io/projected/44eefc80-34c2-47a2-ab48-e7e685f7dfe3-kube-api-access-dfwmj\") pod \"image-registry-648fcb787c-kxf78\" (UID: \"44eefc80-34c2-47a2-ab48-e7e685f7dfe3\") " pod="openshift-image-registry/image-registry-648fcb787c-kxf78" Sep 17 07:28:10.180243 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:10.180136 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fdc3a585-47be-40de-bfa4-9503c7e28701-cert\") pod \"ingress-canary-kcfcf\" (UID: \"fdc3a585-47be-40de-bfa4-9503c7e28701\") " pod="openshift-ingress-canary/ingress-canary-kcfcf" Sep 17 07:28:10.180243 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:10.180176 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/44eefc80-34c2-47a2-ab48-e7e685f7dfe3-image-registry-private-configuration\") pod \"image-registry-648fcb787c-kxf78\" (UID: \"44eefc80-34c2-47a2-ab48-e7e685f7dfe3\") " pod="openshift-image-registry/image-registry-648fcb787c-kxf78" Sep 17 07:28:10.180243 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:10.180195 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/44eefc80-34c2-47a2-ab48-e7e685f7dfe3-registry-tls\") pod \"image-registry-648fcb787c-kxf78\" (UID: \"44eefc80-34c2-47a2-ab48-e7e685f7dfe3\") " pod="openshift-image-registry/image-registry-648fcb787c-kxf78" Sep 17 07:28:10.180243 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:10.180208 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fcfd678-3207-4e47-a6d4-3d1416568d1c-tmp-dir\") pod \"dns-default-nfjvh\" (UID: \"2fcfd678-3207-4e47-a6d4-3d1416568d1c\") " pod="openshift-dns/dns-default-nfjvh" Sep 17 07:28:10.180243 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:10.180224 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nckpn\" (UniqueName: \"kubernetes.io/projected/2fcfd678-3207-4e47-a6d4-3d1416568d1c-kube-api-access-nckpn\") pod \"dns-default-nfjvh\" (UID: \"2fcfd678-3207-4e47-a6d4-3d1416568d1c\") " pod="openshift-dns/dns-default-nfjvh" Sep 17 07:28:10.180243 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:10.180241 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/44eefc80-34c2-47a2-ab48-e7e685f7dfe3-ca-trust-extracted\") pod \"image-registry-648fcb787c-kxf78\" (UID: \"44eefc80-34c2-47a2-ab48-e7e685f7dfe3\") " pod="openshift-image-registry/image-registry-648fcb787c-kxf78" Sep 17 07:28:10.180497 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:28:10.180336 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 17 07:28:10.180497 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:28:10.180355 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-648fcb787c-kxf78: secret "image-registry-tls" not found Sep 17 07:28:10.180497 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:10.180365 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/44eefc80-34c2-47a2-ab48-e7e685f7dfe3-registry-certificates\") pod \"image-registry-648fcb787c-kxf78\" (UID: \"44eefc80-34c2-47a2-ab48-e7e685f7dfe3\") " pod="openshift-image-registry/image-registry-648fcb787c-kxf78" Sep 17 07:28:10.180497 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:10.180402 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2fcfd678-3207-4e47-a6d4-3d1416568d1c-metrics-tls\") pod \"dns-default-nfjvh\" (UID: \"2fcfd678-3207-4e47-a6d4-3d1416568d1c\") " pod="openshift-dns/dns-default-nfjvh" Sep 17 07:28:10.180497 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:28:10.180411 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/44eefc80-34c2-47a2-ab48-e7e685f7dfe3-registry-tls podName:44eefc80-34c2-47a2-ab48-e7e685f7dfe3 nodeName:}" failed. No retries permitted until 2026-09-17 07:28:10.680394068 +0000 UTC m=+34.561428407 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/44eefc80-34c2-47a2-ab48-e7e685f7dfe3-registry-tls") pod "image-registry-648fcb787c-kxf78" (UID: "44eefc80-34c2-47a2-ab48-e7e685f7dfe3") : secret "image-registry-tls" not found Sep 17 07:28:10.180497 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:10.180459 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/44eefc80-34c2-47a2-ab48-e7e685f7dfe3-installation-pull-secrets\") pod \"image-registry-648fcb787c-kxf78\" (UID: \"44eefc80-34c2-47a2-ab48-e7e685f7dfe3\") " pod="openshift-image-registry/image-registry-648fcb787c-kxf78" Sep 17 07:28:10.180497 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:10.180492 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhlwn\" (UniqueName: \"kubernetes.io/projected/fdc3a585-47be-40de-bfa4-9503c7e28701-kube-api-access-fhlwn\") pod \"ingress-canary-kcfcf\" (UID: \"fdc3a585-47be-40de-bfa4-9503c7e28701\") " pod="openshift-ingress-canary/ingress-canary-kcfcf" Sep 17 07:28:10.180827 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:10.180521 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/44eefc80-34c2-47a2-ab48-e7e685f7dfe3-trusted-ca\") pod \"image-registry-648fcb787c-kxf78\" (UID: \"44eefc80-34c2-47a2-ab48-e7e685f7dfe3\") " pod="openshift-image-registry/image-registry-648fcb787c-kxf78" Sep 17 07:28:10.180827 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:10.180543 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/44eefc80-34c2-47a2-ab48-e7e685f7dfe3-bound-sa-token\") pod \"image-registry-648fcb787c-kxf78\" (UID: \"44eefc80-34c2-47a2-ab48-e7e685f7dfe3\") " pod="openshift-image-registry/image-registry-648fcb787c-kxf78" Sep 17 07:28:10.180827 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:10.180569 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2fcfd678-3207-4e47-a6d4-3d1416568d1c-config-volume\") pod \"dns-default-nfjvh\" (UID: \"2fcfd678-3207-4e47-a6d4-3d1416568d1c\") " pod="openshift-dns/dns-default-nfjvh" Sep 17 07:28:10.180827 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:10.180634 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/44eefc80-34c2-47a2-ab48-e7e685f7dfe3-ca-trust-extracted\") pod \"image-registry-648fcb787c-kxf78\" (UID: \"44eefc80-34c2-47a2-ab48-e7e685f7dfe3\") " pod="openshift-image-registry/image-registry-648fcb787c-kxf78" Sep 17 07:28:10.181246 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:10.181220 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/44eefc80-34c2-47a2-ab48-e7e685f7dfe3-registry-certificates\") pod \"image-registry-648fcb787c-kxf78\" (UID: \"44eefc80-34c2-47a2-ab48-e7e685f7dfe3\") " pod="openshift-image-registry/image-registry-648fcb787c-kxf78" Sep 17 07:28:10.181598 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:10.181580 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/44eefc80-34c2-47a2-ab48-e7e685f7dfe3-trusted-ca\") pod \"image-registry-648fcb787c-kxf78\" (UID: \"44eefc80-34c2-47a2-ab48-e7e685f7dfe3\") " pod="openshift-image-registry/image-registry-648fcb787c-kxf78" Sep 17 07:28:10.184283 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:10.184186 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/44eefc80-34c2-47a2-ab48-e7e685f7dfe3-installation-pull-secrets\") pod \"image-registry-648fcb787c-kxf78\" (UID: \"44eefc80-34c2-47a2-ab48-e7e685f7dfe3\") " pod="openshift-image-registry/image-registry-648fcb787c-kxf78" Sep 17 07:28:10.184347 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:10.184206 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/44eefc80-34c2-47a2-ab48-e7e685f7dfe3-image-registry-private-configuration\") pod \"image-registry-648fcb787c-kxf78\" (UID: \"44eefc80-34c2-47a2-ab48-e7e685f7dfe3\") " pod="openshift-image-registry/image-registry-648fcb787c-kxf78" Sep 17 07:28:10.190604 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:10.190580 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfwmj\" (UniqueName: \"kubernetes.io/projected/44eefc80-34c2-47a2-ab48-e7e685f7dfe3-kube-api-access-dfwmj\") pod \"image-registry-648fcb787c-kxf78\" (UID: \"44eefc80-34c2-47a2-ab48-e7e685f7dfe3\") " pod="openshift-image-registry/image-registry-648fcb787c-kxf78" Sep 17 07:28:10.190675 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:10.190581 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/44eefc80-34c2-47a2-ab48-e7e685f7dfe3-bound-sa-token\") pod \"image-registry-648fcb787c-kxf78\" (UID: \"44eefc80-34c2-47a2-ab48-e7e685f7dfe3\") " pod="openshift-image-registry/image-registry-648fcb787c-kxf78" Sep 17 07:28:10.281822 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:10.281792 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2fcfd678-3207-4e47-a6d4-3d1416568d1c-config-volume\") pod \"dns-default-nfjvh\" (UID: \"2fcfd678-3207-4e47-a6d4-3d1416568d1c\") " pod="openshift-dns/dns-default-nfjvh" Sep 17 07:28:10.281965 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:10.281833 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fdc3a585-47be-40de-bfa4-9503c7e28701-cert\") pod \"ingress-canary-kcfcf\" (UID: \"fdc3a585-47be-40de-bfa4-9503c7e28701\") " pod="openshift-ingress-canary/ingress-canary-kcfcf" Sep 17 07:28:10.281965 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:10.281881 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a236d1bd-81b1-4c9e-a14f-997204ad1b98-original-pull-secret\") pod \"global-pull-secret-syncer-lqcw7\" (UID: \"a236d1bd-81b1-4c9e-a14f-997204ad1b98\") " pod="kube-system/global-pull-secret-syncer-lqcw7" Sep 17 07:28:10.281965 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:10.281911 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fcfd678-3207-4e47-a6d4-3d1416568d1c-tmp-dir\") pod \"dns-default-nfjvh\" (UID: \"2fcfd678-3207-4e47-a6d4-3d1416568d1c\") " pod="openshift-dns/dns-default-nfjvh" Sep 17 07:28:10.281965 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:10.281927 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nckpn\" (UniqueName: \"kubernetes.io/projected/2fcfd678-3207-4e47-a6d4-3d1416568d1c-kube-api-access-nckpn\") pod \"dns-default-nfjvh\" (UID: \"2fcfd678-3207-4e47-a6d4-3d1416568d1c\") " pod="openshift-dns/dns-default-nfjvh" Sep 17 07:28:10.281965 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:10.281948 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/737e0243-b8eb-47e4-bf0f-c6b31e6ba787-metrics-certs\") pod \"network-metrics-daemon-lnpjx\" (UID: \"737e0243-b8eb-47e4-bf0f-c6b31e6ba787\") " pod="openshift-multus/network-metrics-daemon-lnpjx" Sep 17 07:28:10.281965 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:10.281963 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2fcfd678-3207-4e47-a6d4-3d1416568d1c-metrics-tls\") pod \"dns-default-nfjvh\" (UID: \"2fcfd678-3207-4e47-a6d4-3d1416568d1c\") " pod="openshift-dns/dns-default-nfjvh" Sep 17 07:28:10.282245 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:10.281989 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhlwn\" (UniqueName: \"kubernetes.io/projected/fdc3a585-47be-40de-bfa4-9503c7e28701-kube-api-access-fhlwn\") pod \"ingress-canary-kcfcf\" (UID: \"fdc3a585-47be-40de-bfa4-9503c7e28701\") " pod="openshift-ingress-canary/ingress-canary-kcfcf" Sep 17 07:28:10.282245 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:28:10.282027 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 17 07:28:10.282245 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:28:10.282062 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 07:28:10.282245 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:28:10.282104 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a236d1bd-81b1-4c9e-a14f-997204ad1b98-original-pull-secret podName:a236d1bd-81b1-4c9e-a14f-997204ad1b98 nodeName:}" failed. No retries permitted until 2026-09-17 07:28:42.282084321 +0000 UTC m=+66.163118677 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a236d1bd-81b1-4c9e-a14f-997204ad1b98-original-pull-secret") pod "global-pull-secret-syncer-lqcw7" (UID: "a236d1bd-81b1-4c9e-a14f-997204ad1b98") : object "kube-system"/"original-pull-secret" not registered Sep 17 07:28:10.282245 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:28:10.282123 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/737e0243-b8eb-47e4-bf0f-c6b31e6ba787-metrics-certs podName:737e0243-b8eb-47e4-bf0f-c6b31e6ba787 nodeName:}" failed. No retries permitted until 2026-09-17 07:28:42.282113062 +0000 UTC m=+66.163147399 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/737e0243-b8eb-47e4-bf0f-c6b31e6ba787-metrics-certs") pod "network-metrics-daemon-lnpjx" (UID: "737e0243-b8eb-47e4-bf0f-c6b31e6ba787") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 07:28:10.282425 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:10.282348 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fcfd678-3207-4e47-a6d4-3d1416568d1c-tmp-dir\") pod \"dns-default-nfjvh\" (UID: \"2fcfd678-3207-4e47-a6d4-3d1416568d1c\") " pod="openshift-dns/dns-default-nfjvh" Sep 17 07:28:10.282475 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:28:10.282440 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 17 07:28:10.282475 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:10.282456 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2fcfd678-3207-4e47-a6d4-3d1416568d1c-config-volume\") pod \"dns-default-nfjvh\" (UID: \"2fcfd678-3207-4e47-a6d4-3d1416568d1c\") " pod="openshift-dns/dns-default-nfjvh" Sep 17 07:28:10.282537 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:28:10.282478 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fcfd678-3207-4e47-a6d4-3d1416568d1c-metrics-tls podName:2fcfd678-3207-4e47-a6d4-3d1416568d1c nodeName:}" failed. No retries permitted until 2026-09-17 07:28:10.782463809 +0000 UTC m=+34.663498163 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2fcfd678-3207-4e47-a6d4-3d1416568d1c-metrics-tls") pod "dns-default-nfjvh" (UID: "2fcfd678-3207-4e47-a6d4-3d1416568d1c") : secret "dns-default-metrics-tls" not found Sep 17 07:28:10.284174 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:10.284153 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fdc3a585-47be-40de-bfa4-9503c7e28701-cert\") pod \"ingress-canary-kcfcf\" (UID: \"fdc3a585-47be-40de-bfa4-9503c7e28701\") " pod="openshift-ingress-canary/ingress-canary-kcfcf" Sep 17 07:28:10.293323 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:10.293275 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nckpn\" (UniqueName: \"kubernetes.io/projected/2fcfd678-3207-4e47-a6d4-3d1416568d1c-kube-api-access-nckpn\") pod \"dns-default-nfjvh\" (UID: \"2fcfd678-3207-4e47-a6d4-3d1416568d1c\") " pod="openshift-dns/dns-default-nfjvh" Sep 17 07:28:10.294951 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:10.294936 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhlwn\" (UniqueName: \"kubernetes.io/projected/fdc3a585-47be-40de-bfa4-9503c7e28701-kube-api-access-fhlwn\") pod \"ingress-canary-kcfcf\" (UID: \"fdc3a585-47be-40de-bfa4-9503c7e28701\") " pod="openshift-ingress-canary/ingress-canary-kcfcf" Sep 17 07:28:10.342564 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:10.342545 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-kcfcf" Sep 17 07:28:10.383198 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:10.383180 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnz4n\" (UniqueName: \"kubernetes.io/projected/82555a36-653a-4106-a31e-b4da6445b838-kube-api-access-gnz4n\") pod \"network-check-target-bvxfq\" (UID: \"82555a36-653a-4106-a31e-b4da6445b838\") " pod="openshift-network-diagnostics/network-check-target-bvxfq" Sep 17 07:28:10.383305 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:28:10.383292 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 17 07:28:10.383345 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:28:10.383309 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 17 07:28:10.383345 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:28:10.383318 2578 projected.go:196] Error preparing data for projected volume kube-api-access-gnz4n for pod openshift-network-diagnostics/network-check-target-bvxfq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 07:28:10.383407 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:28:10.383361 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/82555a36-653a-4106-a31e-b4da6445b838-kube-api-access-gnz4n podName:82555a36-653a-4106-a31e-b4da6445b838 nodeName:}" failed. No retries permitted until 2026-09-17 07:28:42.383350293 +0000 UTC m=+66.264384635 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-gnz4n" (UniqueName: "kubernetes.io/projected/82555a36-653a-4106-a31e-b4da6445b838-kube-api-access-gnz4n") pod "network-check-target-bvxfq" (UID: "82555a36-653a-4106-a31e-b4da6445b838") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 07:28:10.623599 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:10.623514 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bvxfq" Sep 17 07:28:10.627375 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:10.627341 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 17 07:28:10.627375 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:10.627362 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-vqjjh\"" Sep 17 07:28:10.627877 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:10.627394 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 17 07:28:10.663688 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:10.663661 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-kcfcf"] Sep 17 07:28:10.666698 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:28:10.666676 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdc3a585_47be_40de_bfa4_9503c7e28701.slice/crio-85c5f4836c8bba543df22e309663d4adccbbcbf9d53c0d2a697952b3ed5dd579 WatchSource:0}: Error finding container 85c5f4836c8bba543df22e309663d4adccbbcbf9d53c0d2a697952b3ed5dd579: Status 404 returned error can't find the container with id 85c5f4836c8bba543df22e309663d4adccbbcbf9d53c0d2a697952b3ed5dd579 Sep 17 07:28:10.685444 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:10.685426 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/44eefc80-34c2-47a2-ab48-e7e685f7dfe3-registry-tls\") pod \"image-registry-648fcb787c-kxf78\" (UID: \"44eefc80-34c2-47a2-ab48-e7e685f7dfe3\") " pod="openshift-image-registry/image-registry-648fcb787c-kxf78" Sep 17 07:28:10.685582 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:28:10.685567 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 17 07:28:10.685626 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:28:10.685585 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-648fcb787c-kxf78: secret "image-registry-tls" not found Sep 17 07:28:10.685658 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:28:10.685636 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/44eefc80-34c2-47a2-ab48-e7e685f7dfe3-registry-tls podName:44eefc80-34c2-47a2-ab48-e7e685f7dfe3 nodeName:}" failed. No retries permitted until 2026-09-17 07:28:11.68562196 +0000 UTC m=+35.566656296 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/44eefc80-34c2-47a2-ab48-e7e685f7dfe3-registry-tls") pod "image-registry-648fcb787c-kxf78" (UID: "44eefc80-34c2-47a2-ab48-e7e685f7dfe3") : secret "image-registry-tls" not found Sep 17 07:28:10.785922 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:10.785894 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2fcfd678-3207-4e47-a6d4-3d1416568d1c-metrics-tls\") pod \"dns-default-nfjvh\" (UID: \"2fcfd678-3207-4e47-a6d4-3d1416568d1c\") " pod="openshift-dns/dns-default-nfjvh" Sep 17 07:28:10.786057 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:28:10.786038 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 17 07:28:10.786105 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:28:10.786096 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fcfd678-3207-4e47-a6d4-3d1416568d1c-metrics-tls podName:2fcfd678-3207-4e47-a6d4-3d1416568d1c nodeName:}" failed. No retries permitted until 2026-09-17 07:28:11.786082863 +0000 UTC m=+35.667117199 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2fcfd678-3207-4e47-a6d4-3d1416568d1c-metrics-tls") pod "dns-default-nfjvh" (UID: "2fcfd678-3207-4e47-a6d4-3d1416568d1c") : secret "dns-default-metrics-tls" not found Sep 17 07:28:10.791176 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:10.791150 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-kcfcf" event={"ID":"fdc3a585-47be-40de-bfa4-9503c7e28701","Type":"ContainerStarted","Data":"85c5f4836c8bba543df22e309663d4adccbbcbf9d53c0d2a697952b3ed5dd579"} Sep 17 07:28:10.794282 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:10.794261 2578 generic.go:356] "Generic (PLEG): container finished" podID="6642bc78-77e5-44af-af76-a8f87d25c89a" containerID="4decaa6df42c4b60bdadcdf7d5c1a0b40c64698e190354d82ac9a281587a85c5" exitCode=0 Sep 17 07:28:10.794375 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:10.794295 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lh5z8" event={"ID":"6642bc78-77e5-44af-af76-a8f87d25c89a","Type":"ContainerDied","Data":"4decaa6df42c4b60bdadcdf7d5c1a0b40c64698e190354d82ac9a281587a85c5"} Sep 17 07:28:11.624096 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:11.624067 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lnpjx" Sep 17 07:28:11.624261 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:11.624240 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lqcw7" Sep 17 07:28:11.627108 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:11.627079 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 17 07:28:11.628010 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:11.627978 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-rjccm\"" Sep 17 07:28:11.628010 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:11.627995 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 17 07:28:11.692068 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:11.692042 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/44eefc80-34c2-47a2-ab48-e7e685f7dfe3-registry-tls\") pod \"image-registry-648fcb787c-kxf78\" (UID: \"44eefc80-34c2-47a2-ab48-e7e685f7dfe3\") " pod="openshift-image-registry/image-registry-648fcb787c-kxf78" Sep 17 07:28:11.692187 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:28:11.692144 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 17 07:28:11.692187 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:28:11.692156 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-648fcb787c-kxf78: secret "image-registry-tls" not found Sep 17 07:28:11.692263 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:28:11.692207 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/44eefc80-34c2-47a2-ab48-e7e685f7dfe3-registry-tls podName:44eefc80-34c2-47a2-ab48-e7e685f7dfe3 nodeName:}" failed. No retries permitted until 2026-09-17 07:28:13.692189514 +0000 UTC m=+37.573223871 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/44eefc80-34c2-47a2-ab48-e7e685f7dfe3-registry-tls") pod "image-registry-648fcb787c-kxf78" (UID: "44eefc80-34c2-47a2-ab48-e7e685f7dfe3") : secret "image-registry-tls" not found Sep 17 07:28:11.792962 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:11.792926 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2fcfd678-3207-4e47-a6d4-3d1416568d1c-metrics-tls\") pod \"dns-default-nfjvh\" (UID: \"2fcfd678-3207-4e47-a6d4-3d1416568d1c\") " pod="openshift-dns/dns-default-nfjvh" Sep 17 07:28:11.793121 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:28:11.793086 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 17 07:28:11.793182 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:28:11.793159 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fcfd678-3207-4e47-a6d4-3d1416568d1c-metrics-tls podName:2fcfd678-3207-4e47-a6d4-3d1416568d1c nodeName:}" failed. No retries permitted until 2026-09-17 07:28:13.793138937 +0000 UTC m=+37.674173294 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2fcfd678-3207-4e47-a6d4-3d1416568d1c-metrics-tls") pod "dns-default-nfjvh" (UID: "2fcfd678-3207-4e47-a6d4-3d1416568d1c") : secret "dns-default-metrics-tls" not found Sep 17 07:28:11.801147 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:11.801120 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lh5z8" event={"ID":"6642bc78-77e5-44af-af76-a8f87d25c89a","Type":"ContainerStarted","Data":"8cb89d80d890ce01bb28df9337d3844635bde00041a862595fe95e490c923b19"} Sep 17 07:28:11.841405 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:11.841353 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-lh5z8" podStartSLOduration=6.237197151 podStartE2EDuration="34.841334246s" totalImagesPullingTime="28.604137095s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 07:27:37 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 07:28:11.841113881 +0000 UTC m=+35.722148265" watchObservedRunningTime="2026-09-17 07:28:11.841334246 +0000 UTC m=+35.722368608" Sep 17 07:28:13.708017 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:13.707983 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/44eefc80-34c2-47a2-ab48-e7e685f7dfe3-registry-tls\") pod \"image-registry-648fcb787c-kxf78\" (UID: \"44eefc80-34c2-47a2-ab48-e7e685f7dfe3\") " pod="openshift-image-registry/image-registry-648fcb787c-kxf78" Sep 17 07:28:13.708358 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:28:13.708095 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 17 07:28:13.708358 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:28:13.708108 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-648fcb787c-kxf78: secret "image-registry-tls" not found Sep 17 07:28:13.708358 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:28:13.708156 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/44eefc80-34c2-47a2-ab48-e7e685f7dfe3-registry-tls podName:44eefc80-34c2-47a2-ab48-e7e685f7dfe3 nodeName:}" failed. No retries permitted until 2026-09-17 07:28:17.708143254 +0000 UTC m=+41.589177604 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/44eefc80-34c2-47a2-ab48-e7e685f7dfe3-registry-tls") pod "image-registry-648fcb787c-kxf78" (UID: "44eefc80-34c2-47a2-ab48-e7e685f7dfe3") : secret "image-registry-tls" not found Sep 17 07:28:13.807542 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:13.807515 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-kcfcf" event={"ID":"fdc3a585-47be-40de-bfa4-9503c7e28701","Type":"ContainerStarted","Data":"994df6d66a1595c0370ee793f2d67e72362925612f8efedec2f6cf3dd2ad6d20"} Sep 17 07:28:13.808902 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:13.808886 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2fcfd678-3207-4e47-a6d4-3d1416568d1c-metrics-tls\") pod \"dns-default-nfjvh\" (UID: \"2fcfd678-3207-4e47-a6d4-3d1416568d1c\") " pod="openshift-dns/dns-default-nfjvh" Sep 17 07:28:13.809005 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:28:13.808992 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 17 07:28:13.809059 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:28:13.809039 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fcfd678-3207-4e47-a6d4-3d1416568d1c-metrics-tls podName:2fcfd678-3207-4e47-a6d4-3d1416568d1c nodeName:}" failed. No retries permitted until 2026-09-17 07:28:17.809023877 +0000 UTC m=+41.690058214 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2fcfd678-3207-4e47-a6d4-3d1416568d1c-metrics-tls") pod "dns-default-nfjvh" (UID: "2fcfd678-3207-4e47-a6d4-3d1416568d1c") : secret "dns-default-metrics-tls" not found Sep 17 07:28:13.829842 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:13.829800 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-kcfcf" podStartSLOduration=2.324988137 podStartE2EDuration="4.829786894s" totalImagesPullingTime="2.504798757s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 07:28:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 07:28:13.828342025 +0000 UTC m=+37.709376383" watchObservedRunningTime="2026-09-17 07:28:13.829786894 +0000 UTC m=+37.710821252" Sep 17 07:28:17.732974 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:17.732937 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/44eefc80-34c2-47a2-ab48-e7e685f7dfe3-registry-tls\") pod \"image-registry-648fcb787c-kxf78\" (UID: \"44eefc80-34c2-47a2-ab48-e7e685f7dfe3\") " pod="openshift-image-registry/image-registry-648fcb787c-kxf78" Sep 17 07:28:17.733410 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:28:17.733109 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 17 07:28:17.733410 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:28:17.733133 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-648fcb787c-kxf78: secret "image-registry-tls" not found Sep 17 07:28:17.733410 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:28:17.733200 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/44eefc80-34c2-47a2-ab48-e7e685f7dfe3-registry-tls podName:44eefc80-34c2-47a2-ab48-e7e685f7dfe3 nodeName:}" failed. No retries permitted until 2026-09-17 07:28:25.733180589 +0000 UTC m=+49.614214940 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/44eefc80-34c2-47a2-ab48-e7e685f7dfe3-registry-tls") pod "image-registry-648fcb787c-kxf78" (UID: "44eefc80-34c2-47a2-ab48-e7e685f7dfe3") : secret "image-registry-tls" not found Sep 17 07:28:17.833394 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:17.833358 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2fcfd678-3207-4e47-a6d4-3d1416568d1c-metrics-tls\") pod \"dns-default-nfjvh\" (UID: \"2fcfd678-3207-4e47-a6d4-3d1416568d1c\") " pod="openshift-dns/dns-default-nfjvh" Sep 17 07:28:17.833531 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:28:17.833508 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 17 07:28:17.833588 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:28:17.833561 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fcfd678-3207-4e47-a6d4-3d1416568d1c-metrics-tls podName:2fcfd678-3207-4e47-a6d4-3d1416568d1c nodeName:}" failed. No retries permitted until 2026-09-17 07:28:25.833547131 +0000 UTC m=+49.714581468 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2fcfd678-3207-4e47-a6d4-3d1416568d1c-metrics-tls") pod "dns-default-nfjvh" (UID: "2fcfd678-3207-4e47-a6d4-3d1416568d1c") : secret "dns-default-metrics-tls" not found Sep 17 07:28:25.791981 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:25.791928 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/44eefc80-34c2-47a2-ab48-e7e685f7dfe3-registry-tls\") pod \"image-registry-648fcb787c-kxf78\" (UID: \"44eefc80-34c2-47a2-ab48-e7e685f7dfe3\") " pod="openshift-image-registry/image-registry-648fcb787c-kxf78" Sep 17 07:28:25.792439 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:28:25.792079 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 17 07:28:25.792439 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:28:25.792098 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-648fcb787c-kxf78: secret "image-registry-tls" not found Sep 17 07:28:25.792439 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:28:25.792150 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/44eefc80-34c2-47a2-ab48-e7e685f7dfe3-registry-tls podName:44eefc80-34c2-47a2-ab48-e7e685f7dfe3 nodeName:}" failed. No retries permitted until 2026-09-17 07:28:41.792134406 +0000 UTC m=+65.673168744 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/44eefc80-34c2-47a2-ab48-e7e685f7dfe3-registry-tls") pod "image-registry-648fcb787c-kxf78" (UID: "44eefc80-34c2-47a2-ab48-e7e685f7dfe3") : secret "image-registry-tls" not found Sep 17 07:28:25.892274 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:25.892244 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2fcfd678-3207-4e47-a6d4-3d1416568d1c-metrics-tls\") pod \"dns-default-nfjvh\" (UID: \"2fcfd678-3207-4e47-a6d4-3d1416568d1c\") " pod="openshift-dns/dns-default-nfjvh" Sep 17 07:28:25.892425 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:28:25.892391 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 17 07:28:25.892467 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:28:25.892447 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fcfd678-3207-4e47-a6d4-3d1416568d1c-metrics-tls podName:2fcfd678-3207-4e47-a6d4-3d1416568d1c nodeName:}" failed. No retries permitted until 2026-09-17 07:28:41.892431892 +0000 UTC m=+65.773466250 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2fcfd678-3207-4e47-a6d4-3d1416568d1c-metrics-tls") pod "dns-default-nfjvh" (UID: "2fcfd678-3207-4e47-a6d4-3d1416568d1c") : secret "dns-default-metrics-tls" not found Sep 17 07:28:34.783267 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:34.783241 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hnzgv" Sep 17 07:28:40.108025 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:40.107991 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-764969ff56-mgc25"] Sep 17 07:28:40.141895 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:40.141870 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-764969ff56-mgc25"] Sep 17 07:28:40.142032 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:40.141971 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-764969ff56-mgc25" Sep 17 07:28:40.144496 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:40.144472 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 17 07:28:40.144496 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:40.144488 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Sep 17 07:28:40.144671 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:40.144567 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Sep 17 07:28:40.144671 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:40.144572 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Sep 17 07:28:40.144768 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:40.144737 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Sep 17 07:28:40.145036 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:40.145019 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 17 07:28:40.145160 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:40.145141 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-qpvdz\"" Sep 17 07:28:40.296303 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:40.296261 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/67157086-7b61-49c2-8862-e594a9881821-default-certificate\") pod \"router-default-764969ff56-mgc25\" (UID: \"67157086-7b61-49c2-8862-e594a9881821\") " pod="openshift-ingress/router-default-764969ff56-mgc25" Sep 17 07:28:40.296481 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:40.296315 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67157086-7b61-49c2-8862-e594a9881821-service-ca-bundle\") pod \"router-default-764969ff56-mgc25\" (UID: \"67157086-7b61-49c2-8862-e594a9881821\") " pod="openshift-ingress/router-default-764969ff56-mgc25" Sep 17 07:28:40.296481 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:40.296334 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67157086-7b61-49c2-8862-e594a9881821-metrics-certs\") pod \"router-default-764969ff56-mgc25\" (UID: \"67157086-7b61-49c2-8862-e594a9881821\") " pod="openshift-ingress/router-default-764969ff56-mgc25" Sep 17 07:28:40.296481 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:40.296390 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x77kr\" (UniqueName: \"kubernetes.io/projected/67157086-7b61-49c2-8862-e594a9881821-kube-api-access-x77kr\") pod \"router-default-764969ff56-mgc25\" (UID: \"67157086-7b61-49c2-8862-e594a9881821\") " pod="openshift-ingress/router-default-764969ff56-mgc25" Sep 17 07:28:40.296598 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:40.296478 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/67157086-7b61-49c2-8862-e594a9881821-stats-auth\") pod \"router-default-764969ff56-mgc25\" (UID: \"67157086-7b61-49c2-8862-e594a9881821\") " pod="openshift-ingress/router-default-764969ff56-mgc25" Sep 17 07:28:40.397334 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:40.397301 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67157086-7b61-49c2-8862-e594a9881821-metrics-certs\") pod \"router-default-764969ff56-mgc25\" (UID: \"67157086-7b61-49c2-8862-e594a9881821\") " pod="openshift-ingress/router-default-764969ff56-mgc25" Sep 17 07:28:40.397483 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:40.397343 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x77kr\" (UniqueName: \"kubernetes.io/projected/67157086-7b61-49c2-8862-e594a9881821-kube-api-access-x77kr\") pod \"router-default-764969ff56-mgc25\" (UID: \"67157086-7b61-49c2-8862-e594a9881821\") " pod="openshift-ingress/router-default-764969ff56-mgc25" Sep 17 07:28:40.397483 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:40.397364 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/67157086-7b61-49c2-8862-e594a9881821-stats-auth\") pod \"router-default-764969ff56-mgc25\" (UID: \"67157086-7b61-49c2-8862-e594a9881821\") " pod="openshift-ingress/router-default-764969ff56-mgc25" Sep 17 07:28:40.397483 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:28:40.397452 2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 17 07:28:40.397633 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:28:40.397514 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67157086-7b61-49c2-8862-e594a9881821-metrics-certs podName:67157086-7b61-49c2-8862-e594a9881821 nodeName:}" failed. No retries permitted until 2026-09-17 07:28:40.897498051 +0000 UTC m=+64.778532416 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/67157086-7b61-49c2-8862-e594a9881821-metrics-certs") pod "router-default-764969ff56-mgc25" (UID: "67157086-7b61-49c2-8862-e594a9881821") : secret "router-metrics-certs-default" not found Sep 17 07:28:40.397633 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:40.397548 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/67157086-7b61-49c2-8862-e594a9881821-default-certificate\") pod \"router-default-764969ff56-mgc25\" (UID: \"67157086-7b61-49c2-8862-e594a9881821\") " pod="openshift-ingress/router-default-764969ff56-mgc25" Sep 17 07:28:40.397633 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:40.397582 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67157086-7b61-49c2-8862-e594a9881821-service-ca-bundle\") pod \"router-default-764969ff56-mgc25\" (UID: \"67157086-7b61-49c2-8862-e594a9881821\") " pod="openshift-ingress/router-default-764969ff56-mgc25" Sep 17 07:28:40.397773 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:28:40.397665 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/67157086-7b61-49c2-8862-e594a9881821-service-ca-bundle podName:67157086-7b61-49c2-8862-e594a9881821 nodeName:}" failed. No retries permitted until 2026-09-17 07:28:40.897652527 +0000 UTC m=+64.778686864 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/67157086-7b61-49c2-8862-e594a9881821-service-ca-bundle") pod "router-default-764969ff56-mgc25" (UID: "67157086-7b61-49c2-8862-e594a9881821") : configmap references non-existent config key: service-ca.crt Sep 17 07:28:40.400469 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:40.400449 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/67157086-7b61-49c2-8862-e594a9881821-stats-auth\") pod \"router-default-764969ff56-mgc25\" (UID: \"67157086-7b61-49c2-8862-e594a9881821\") " pod="openshift-ingress/router-default-764969ff56-mgc25" Sep 17 07:28:40.400614 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:40.400581 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/67157086-7b61-49c2-8862-e594a9881821-default-certificate\") pod \"router-default-764969ff56-mgc25\" (UID: \"67157086-7b61-49c2-8862-e594a9881821\") " pod="openshift-ingress/router-default-764969ff56-mgc25" Sep 17 07:28:40.406371 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:40.406352 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x77kr\" (UniqueName: \"kubernetes.io/projected/67157086-7b61-49c2-8862-e594a9881821-kube-api-access-x77kr\") pod \"router-default-764969ff56-mgc25\" (UID: \"67157086-7b61-49c2-8862-e594a9881821\") " pod="openshift-ingress/router-default-764969ff56-mgc25" Sep 17 07:28:40.900456 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:40.900432 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67157086-7b61-49c2-8862-e594a9881821-service-ca-bundle\") pod \"router-default-764969ff56-mgc25\" (UID: \"67157086-7b61-49c2-8862-e594a9881821\") " pod="openshift-ingress/router-default-764969ff56-mgc25" Sep 17 07:28:40.900601 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:40.900462 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67157086-7b61-49c2-8862-e594a9881821-metrics-certs\") pod \"router-default-764969ff56-mgc25\" (UID: \"67157086-7b61-49c2-8862-e594a9881821\") " pod="openshift-ingress/router-default-764969ff56-mgc25" Sep 17 07:28:40.900601 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:28:40.900552 2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 17 07:28:40.900601 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:28:40.900578 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/67157086-7b61-49c2-8862-e594a9881821-service-ca-bundle podName:67157086-7b61-49c2-8862-e594a9881821 nodeName:}" failed. No retries permitted until 2026-09-17 07:28:41.900562027 +0000 UTC m=+65.781596382 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/67157086-7b61-49c2-8862-e594a9881821-service-ca-bundle") pod "router-default-764969ff56-mgc25" (UID: "67157086-7b61-49c2-8862-e594a9881821") : configmap references non-existent config key: service-ca.crt Sep 17 07:28:40.900601 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:28:40.900599 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67157086-7b61-49c2-8862-e594a9881821-metrics-certs podName:67157086-7b61-49c2-8862-e594a9881821 nodeName:}" failed. No retries permitted until 2026-09-17 07:28:41.900591662 +0000 UTC m=+65.781625999 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/67157086-7b61-49c2-8862-e594a9881821-metrics-certs") pod "router-default-764969ff56-mgc25" (UID: "67157086-7b61-49c2-8862-e594a9881821") : secret "router-metrics-certs-default" not found Sep 17 07:28:41.806405 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:41.806359 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/44eefc80-34c2-47a2-ab48-e7e685f7dfe3-registry-tls\") pod \"image-registry-648fcb787c-kxf78\" (UID: \"44eefc80-34c2-47a2-ab48-e7e685f7dfe3\") " pod="openshift-image-registry/image-registry-648fcb787c-kxf78" Sep 17 07:28:41.806882 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:28:41.806499 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 17 07:28:41.806882 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:28:41.806516 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-648fcb787c-kxf78: secret "image-registry-tls" not found Sep 17 07:28:41.806882 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:28:41.806577 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/44eefc80-34c2-47a2-ab48-e7e685f7dfe3-registry-tls podName:44eefc80-34c2-47a2-ab48-e7e685f7dfe3 nodeName:}" failed. No retries permitted until 2026-09-17 07:29:13.806562206 +0000 UTC m=+97.687596568 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/44eefc80-34c2-47a2-ab48-e7e685f7dfe3-registry-tls") pod "image-registry-648fcb787c-kxf78" (UID: "44eefc80-34c2-47a2-ab48-e7e685f7dfe3") : secret "image-registry-tls" not found Sep 17 07:28:41.907399 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:41.907370 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2fcfd678-3207-4e47-a6d4-3d1416568d1c-metrics-tls\") pod \"dns-default-nfjvh\" (UID: \"2fcfd678-3207-4e47-a6d4-3d1416568d1c\") " pod="openshift-dns/dns-default-nfjvh" Sep 17 07:28:41.907399 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:41.907397 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67157086-7b61-49c2-8862-e594a9881821-service-ca-bundle\") pod \"router-default-764969ff56-mgc25\" (UID: \"67157086-7b61-49c2-8862-e594a9881821\") " pod="openshift-ingress/router-default-764969ff56-mgc25" Sep 17 07:28:41.907568 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:41.907420 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67157086-7b61-49c2-8862-e594a9881821-metrics-certs\") pod \"router-default-764969ff56-mgc25\" (UID: \"67157086-7b61-49c2-8862-e594a9881821\") " pod="openshift-ingress/router-default-764969ff56-mgc25" Sep 17 07:28:41.907568 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:28:41.907498 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 17 07:28:41.907568 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:28:41.907547 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/67157086-7b61-49c2-8862-e594a9881821-service-ca-bundle podName:67157086-7b61-49c2-8862-e594a9881821 nodeName:}" failed. No retries permitted until 2026-09-17 07:28:43.907526346 +0000 UTC m=+67.788560700 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/67157086-7b61-49c2-8862-e594a9881821-service-ca-bundle") pod "router-default-764969ff56-mgc25" (UID: "67157086-7b61-49c2-8862-e594a9881821") : configmap references non-existent config key: service-ca.crt Sep 17 07:28:41.907680 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:28:41.907571 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fcfd678-3207-4e47-a6d4-3d1416568d1c-metrics-tls podName:2fcfd678-3207-4e47-a6d4-3d1416568d1c nodeName:}" failed. No retries permitted until 2026-09-17 07:29:13.907561918 +0000 UTC m=+97.788596255 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2fcfd678-3207-4e47-a6d4-3d1416568d1c-metrics-tls") pod "dns-default-nfjvh" (UID: "2fcfd678-3207-4e47-a6d4-3d1416568d1c") : secret "dns-default-metrics-tls" not found Sep 17 07:28:41.907680 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:28:41.907576 2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 17 07:28:41.907680 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:28:41.907614 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67157086-7b61-49c2-8862-e594a9881821-metrics-certs podName:67157086-7b61-49c2-8862-e594a9881821 nodeName:}" failed. No retries permitted until 2026-09-17 07:28:43.907605193 +0000 UTC m=+67.788639537 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/67157086-7b61-49c2-8862-e594a9881821-metrics-certs") pod "router-default-764969ff56-mgc25" (UID: "67157086-7b61-49c2-8862-e594a9881821") : secret "router-metrics-certs-default" not found Sep 17 07:28:42.310164 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:42.310127 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a236d1bd-81b1-4c9e-a14f-997204ad1b98-original-pull-secret\") pod \"global-pull-secret-syncer-lqcw7\" (UID: \"a236d1bd-81b1-4c9e-a14f-997204ad1b98\") " pod="kube-system/global-pull-secret-syncer-lqcw7" Sep 17 07:28:42.310330 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:42.310182 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/737e0243-b8eb-47e4-bf0f-c6b31e6ba787-metrics-certs\") pod \"network-metrics-daemon-lnpjx\" (UID: \"737e0243-b8eb-47e4-bf0f-c6b31e6ba787\") " pod="openshift-multus/network-metrics-daemon-lnpjx" Sep 17 07:28:42.314326 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:42.314306 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 17 07:28:42.314373 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:42.314306 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 17 07:28:42.320608 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:28:42.320589 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 17 07:28:42.320658 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:28:42.320644 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/737e0243-b8eb-47e4-bf0f-c6b31e6ba787-metrics-certs podName:737e0243-b8eb-47e4-bf0f-c6b31e6ba787 nodeName:}" failed. No retries permitted until 2026-09-17 07:29:46.32063033 +0000 UTC m=+130.201664666 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/737e0243-b8eb-47e4-bf0f-c6b31e6ba787-metrics-certs") pod "network-metrics-daemon-lnpjx" (UID: "737e0243-b8eb-47e4-bf0f-c6b31e6ba787") : secret "metrics-daemon-secret" not found Sep 17 07:28:42.322667 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:42.322652 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a236d1bd-81b1-4c9e-a14f-997204ad1b98-original-pull-secret\") pod \"global-pull-secret-syncer-lqcw7\" (UID: \"a236d1bd-81b1-4c9e-a14f-997204ad1b98\") " pod="kube-system/global-pull-secret-syncer-lqcw7" Sep 17 07:28:42.411032 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:42.411000 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnz4n\" (UniqueName: \"kubernetes.io/projected/82555a36-653a-4106-a31e-b4da6445b838-kube-api-access-gnz4n\") pod \"network-check-target-bvxfq\" (UID: \"82555a36-653a-4106-a31e-b4da6445b838\") " pod="openshift-network-diagnostics/network-check-target-bvxfq" Sep 17 07:28:42.414620 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:42.414606 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 17 07:28:42.424064 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:42.424045 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 17 07:28:42.434675 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:42.434648 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnz4n\" (UniqueName: \"kubernetes.io/projected/82555a36-653a-4106-a31e-b4da6445b838-kube-api-access-gnz4n\") pod \"network-check-target-bvxfq\" (UID: \"82555a36-653a-4106-a31e-b4da6445b838\") " pod="openshift-network-diagnostics/network-check-target-bvxfq" Sep 17 07:28:42.437829 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:42.437811 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-vqjjh\"" Sep 17 07:28:42.446224 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:42.446211 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bvxfq" Sep 17 07:28:42.541595 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:42.541568 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lqcw7" Sep 17 07:28:42.712089 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:42.712058 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-bvxfq"] Sep 17 07:28:42.715588 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:28:42.715565 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82555a36_653a_4106_a31e_b4da6445b838.slice/crio-5e45d118878572b437d9804bcc2e93197d0de392e3dff82beb7a549f6df89241 WatchSource:0}: Error finding container 5e45d118878572b437d9804bcc2e93197d0de392e3dff82beb7a549f6df89241: Status 404 returned error can't find the container with id 5e45d118878572b437d9804bcc2e93197d0de392e3dff82beb7a549f6df89241 Sep 17 07:28:42.807118 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:42.807085 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-lqcw7"] Sep 17 07:28:42.809878 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:28:42.809836 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda236d1bd_81b1_4c9e_a14f_997204ad1b98.slice/crio-08f54ab232b47ae762b7fae2014b8291798ecab3966b81e08ba1c3eb5296a1f3 WatchSource:0}: Error finding container 08f54ab232b47ae762b7fae2014b8291798ecab3966b81e08ba1c3eb5296a1f3: Status 404 returned error can't find the container with id 08f54ab232b47ae762b7fae2014b8291798ecab3966b81e08ba1c3eb5296a1f3 Sep 17 07:28:42.874318 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:42.874253 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-lqcw7" event={"ID":"a236d1bd-81b1-4c9e-a14f-997204ad1b98","Type":"ContainerStarted","Data":"08f54ab232b47ae762b7fae2014b8291798ecab3966b81e08ba1c3eb5296a1f3"} Sep 17 07:28:42.875402 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:42.875381 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-bvxfq" event={"ID":"82555a36-653a-4106-a31e-b4da6445b838","Type":"ContainerStarted","Data":"5e45d118878572b437d9804bcc2e93197d0de392e3dff82beb7a549f6df89241"} Sep 17 07:28:43.920699 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:43.920668 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67157086-7b61-49c2-8862-e594a9881821-service-ca-bundle\") pod \"router-default-764969ff56-mgc25\" (UID: \"67157086-7b61-49c2-8862-e594a9881821\") " pod="openshift-ingress/router-default-764969ff56-mgc25" Sep 17 07:28:43.921072 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:43.920704 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67157086-7b61-49c2-8862-e594a9881821-metrics-certs\") pod \"router-default-764969ff56-mgc25\" (UID: \"67157086-7b61-49c2-8862-e594a9881821\") " pod="openshift-ingress/router-default-764969ff56-mgc25" Sep 17 07:28:43.921072 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:28:43.920808 2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 17 07:28:43.921072 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:28:43.920832 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/67157086-7b61-49c2-8862-e594a9881821-service-ca-bundle podName:67157086-7b61-49c2-8862-e594a9881821 nodeName:}" failed. No retries permitted until 2026-09-17 07:28:47.920814041 +0000 UTC m=+71.801848395 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/67157086-7b61-49c2-8862-e594a9881821-service-ca-bundle") pod "router-default-764969ff56-mgc25" (UID: "67157086-7b61-49c2-8862-e594a9881821") : configmap references non-existent config key: service-ca.crt Sep 17 07:28:43.921072 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:28:43.920852 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67157086-7b61-49c2-8862-e594a9881821-metrics-certs podName:67157086-7b61-49c2-8862-e594a9881821 nodeName:}" failed. No retries permitted until 2026-09-17 07:28:47.920843626 +0000 UTC m=+71.801877966 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/67157086-7b61-49c2-8862-e594a9881821-metrics-certs") pod "router-default-764969ff56-mgc25" (UID: "67157086-7b61-49c2-8862-e594a9881821") : secret "router-metrics-certs-default" not found Sep 17 07:28:44.011738 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:44.011704 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-zkvmd"] Sep 17 07:28:44.013822 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:44.013796 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-zkvmd" Sep 17 07:28:44.017956 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:44.017929 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 17 07:28:44.018230 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:44.018207 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 17 07:28:44.018299 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:44.018253 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 17 07:28:44.019339 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:44.019312 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 17 07:28:44.019436 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:44.019391 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-pkg8r\"" Sep 17 07:28:44.021215 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:44.021181 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxbbj\" (UniqueName: \"kubernetes.io/projected/ed949cd2-5046-4ef7-870c-57671c2843f7-kube-api-access-kxbbj\") pod \"insights-runtime-extractor-zkvmd\" (UID: \"ed949cd2-5046-4ef7-870c-57671c2843f7\") " pod="openshift-insights/insights-runtime-extractor-zkvmd" Sep 17 07:28:44.021318 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:44.021224 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ed949cd2-5046-4ef7-870c-57671c2843f7-data-volume\") pod \"insights-runtime-extractor-zkvmd\" (UID: \"ed949cd2-5046-4ef7-870c-57671c2843f7\") " pod="openshift-insights/insights-runtime-extractor-zkvmd" Sep 17 07:28:44.021318 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:44.021250 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ed949cd2-5046-4ef7-870c-57671c2843f7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zkvmd\" (UID: \"ed949cd2-5046-4ef7-870c-57671c2843f7\") " pod="openshift-insights/insights-runtime-extractor-zkvmd" Sep 17 07:28:44.021318 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:44.021315 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ed949cd2-5046-4ef7-870c-57671c2843f7-crio-socket\") pod \"insights-runtime-extractor-zkvmd\" (UID: \"ed949cd2-5046-4ef7-870c-57671c2843f7\") " pod="openshift-insights/insights-runtime-extractor-zkvmd" Sep 17 07:28:44.021482 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:44.021359 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ed949cd2-5046-4ef7-870c-57671c2843f7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zkvmd\" (UID: \"ed949cd2-5046-4ef7-870c-57671c2843f7\") " pod="openshift-insights/insights-runtime-extractor-zkvmd" Sep 17 07:28:44.027813 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:44.027751 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-zkvmd"] Sep 17 07:28:44.122055 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:44.121664 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxbbj\" (UniqueName: \"kubernetes.io/projected/ed949cd2-5046-4ef7-870c-57671c2843f7-kube-api-access-kxbbj\") pod \"insights-runtime-extractor-zkvmd\" (UID: \"ed949cd2-5046-4ef7-870c-57671c2843f7\") " pod="openshift-insights/insights-runtime-extractor-zkvmd" Sep 17 07:28:44.122055 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:44.121712 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ed949cd2-5046-4ef7-870c-57671c2843f7-data-volume\") pod \"insights-runtime-extractor-zkvmd\" (UID: \"ed949cd2-5046-4ef7-870c-57671c2843f7\") " pod="openshift-insights/insights-runtime-extractor-zkvmd" Sep 17 07:28:44.122055 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:44.121744 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ed949cd2-5046-4ef7-870c-57671c2843f7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zkvmd\" (UID: \"ed949cd2-5046-4ef7-870c-57671c2843f7\") " pod="openshift-insights/insights-runtime-extractor-zkvmd" Sep 17 07:28:44.122055 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:44.121798 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ed949cd2-5046-4ef7-870c-57671c2843f7-crio-socket\") pod \"insights-runtime-extractor-zkvmd\" (UID: \"ed949cd2-5046-4ef7-870c-57671c2843f7\") " pod="openshift-insights/insights-runtime-extractor-zkvmd" Sep 17 07:28:44.122055 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:44.121836 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ed949cd2-5046-4ef7-870c-57671c2843f7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zkvmd\" (UID: \"ed949cd2-5046-4ef7-870c-57671c2843f7\") " pod="openshift-insights/insights-runtime-extractor-zkvmd" Sep 17 07:28:44.122055 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:28:44.121888 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 17 07:28:44.122055 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:28:44.121962 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ed949cd2-5046-4ef7-870c-57671c2843f7-kube-rbac-proxy-cm podName:ed949cd2-5046-4ef7-870c-57671c2843f7 nodeName:}" failed. No retries permitted until 2026-09-17 07:28:44.621941245 +0000 UTC m=+68.502975601 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ed949cd2-5046-4ef7-870c-57671c2843f7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zkvmd" (UID: "ed949cd2-5046-4ef7-870c-57671c2843f7") : configmap "kube-rbac-proxy" not found Sep 17 07:28:44.122055 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:28:44.121973 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 17 07:28:44.122055 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:28:44.122021 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed949cd2-5046-4ef7-870c-57671c2843f7-insights-runtime-extractor-tls podName:ed949cd2-5046-4ef7-870c-57671c2843f7 nodeName:}" failed. No retries permitted until 2026-09-17 07:28:44.6220056 +0000 UTC m=+68.503039937 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ed949cd2-5046-4ef7-870c-57671c2843f7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zkvmd" (UID: "ed949cd2-5046-4ef7-870c-57671c2843f7") : secret "insights-runtime-extractor-tls" not found Sep 17 07:28:44.122055 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:44.122018 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ed949cd2-5046-4ef7-870c-57671c2843f7-crio-socket\") pod \"insights-runtime-extractor-zkvmd\" (UID: \"ed949cd2-5046-4ef7-870c-57671c2843f7\") " pod="openshift-insights/insights-runtime-extractor-zkvmd" Sep 17 07:28:44.122661 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:44.122118 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ed949cd2-5046-4ef7-870c-57671c2843f7-data-volume\") pod \"insights-runtime-extractor-zkvmd\" (UID: \"ed949cd2-5046-4ef7-870c-57671c2843f7\") " pod="openshift-insights/insights-runtime-extractor-zkvmd" Sep 17 07:28:44.132384 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:44.132360 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxbbj\" (UniqueName: \"kubernetes.io/projected/ed949cd2-5046-4ef7-870c-57671c2843f7-kube-api-access-kxbbj\") pod \"insights-runtime-extractor-zkvmd\" (UID: \"ed949cd2-5046-4ef7-870c-57671c2843f7\") " pod="openshift-insights/insights-runtime-extractor-zkvmd" Sep 17 07:28:44.625474 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:44.625436 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ed949cd2-5046-4ef7-870c-57671c2843f7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zkvmd\" (UID: \"ed949cd2-5046-4ef7-870c-57671c2843f7\") " pod="openshift-insights/insights-runtime-extractor-zkvmd" Sep 17 07:28:44.625642 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:44.625541 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ed949cd2-5046-4ef7-870c-57671c2843f7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zkvmd\" (UID: \"ed949cd2-5046-4ef7-870c-57671c2843f7\") " pod="openshift-insights/insights-runtime-extractor-zkvmd" Sep 17 07:28:44.625714 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:28:44.625699 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 17 07:28:44.625786 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:28:44.625774 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed949cd2-5046-4ef7-870c-57671c2843f7-insights-runtime-extractor-tls podName:ed949cd2-5046-4ef7-870c-57671c2843f7 nodeName:}" failed. No retries permitted until 2026-09-17 07:28:45.625754332 +0000 UTC m=+69.506788669 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ed949cd2-5046-4ef7-870c-57671c2843f7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zkvmd" (UID: "ed949cd2-5046-4ef7-870c-57671c2843f7") : secret "insights-runtime-extractor-tls" not found Sep 17 07:28:44.625850 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:28:44.625816 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 17 07:28:44.625923 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:28:44.625875 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ed949cd2-5046-4ef7-870c-57671c2843f7-kube-rbac-proxy-cm podName:ed949cd2-5046-4ef7-870c-57671c2843f7 nodeName:}" failed. No retries permitted until 2026-09-17 07:28:45.625847581 +0000 UTC m=+69.506881932 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ed949cd2-5046-4ef7-870c-57671c2843f7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zkvmd" (UID: "ed949cd2-5046-4ef7-870c-57671c2843f7") : configmap "kube-rbac-proxy" not found Sep 17 07:28:45.632132 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:45.632098 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ed949cd2-5046-4ef7-870c-57671c2843f7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zkvmd\" (UID: \"ed949cd2-5046-4ef7-870c-57671c2843f7\") " pod="openshift-insights/insights-runtime-extractor-zkvmd" Sep 17 07:28:45.632590 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:45.632193 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ed949cd2-5046-4ef7-870c-57671c2843f7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zkvmd\" (UID: \"ed949cd2-5046-4ef7-870c-57671c2843f7\") " pod="openshift-insights/insights-runtime-extractor-zkvmd" Sep 17 07:28:45.632590 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:28:45.632321 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 17 07:28:45.632590 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:28:45.632383 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ed949cd2-5046-4ef7-870c-57671c2843f7-kube-rbac-proxy-cm podName:ed949cd2-5046-4ef7-870c-57671c2843f7 nodeName:}" failed. No retries permitted until 2026-09-17 07:28:47.632363843 +0000 UTC m=+71.513398198 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ed949cd2-5046-4ef7-870c-57671c2843f7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zkvmd" (UID: "ed949cd2-5046-4ef7-870c-57671c2843f7") : configmap "kube-rbac-proxy" not found Sep 17 07:28:45.632777 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:28:45.632752 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 17 07:28:45.632845 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:28:45.632831 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed949cd2-5046-4ef7-870c-57671c2843f7-insights-runtime-extractor-tls podName:ed949cd2-5046-4ef7-870c-57671c2843f7 nodeName:}" failed. No retries permitted until 2026-09-17 07:28:47.632811884 +0000 UTC m=+71.513846237 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ed949cd2-5046-4ef7-870c-57671c2843f7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zkvmd" (UID: "ed949cd2-5046-4ef7-870c-57671c2843f7") : secret "insights-runtime-extractor-tls" not found Sep 17 07:28:45.940344 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:45.940314 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xfggl_d9df1c13-2108-44d8-9137-730e35cf2165/dns-node-resolver/0.log" Sep 17 07:28:46.540711 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:46.540683 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-fwdd5_f50cb832-b497-4a6a-bbe6-b758ec939688/node-ca/0.log" Sep 17 07:28:47.548911 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:47.548889 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-kcfcf_fdc3a585-47be-40de-bfa4-9503c7e28701/serve-healthcheck-canary/0.log" Sep 17 07:28:47.646963 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:47.646939 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ed949cd2-5046-4ef7-870c-57671c2843f7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zkvmd\" (UID: \"ed949cd2-5046-4ef7-870c-57671c2843f7\") " pod="openshift-insights/insights-runtime-extractor-zkvmd" Sep 17 07:28:47.647077 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:47.647005 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ed949cd2-5046-4ef7-870c-57671c2843f7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zkvmd\" (UID: \"ed949cd2-5046-4ef7-870c-57671c2843f7\") " pod="openshift-insights/insights-runtime-extractor-zkvmd" Sep 17 07:28:47.647126 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:28:47.647070 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 17 07:28:47.647126 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:28:47.647111 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 17 07:28:47.647193 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:28:47.647151 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ed949cd2-5046-4ef7-870c-57671c2843f7-kube-rbac-proxy-cm podName:ed949cd2-5046-4ef7-870c-57671c2843f7 nodeName:}" failed. No retries permitted until 2026-09-17 07:28:51.647130463 +0000 UTC m=+75.528164800 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ed949cd2-5046-4ef7-870c-57671c2843f7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zkvmd" (UID: "ed949cd2-5046-4ef7-870c-57671c2843f7") : configmap "kube-rbac-proxy" not found Sep 17 07:28:47.647193 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:28:47.647171 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed949cd2-5046-4ef7-870c-57671c2843f7-insights-runtime-extractor-tls podName:ed949cd2-5046-4ef7-870c-57671c2843f7 nodeName:}" failed. No retries permitted until 2026-09-17 07:28:51.647161199 +0000 UTC m=+75.528195538 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ed949cd2-5046-4ef7-870c-57671c2843f7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zkvmd" (UID: "ed949cd2-5046-4ef7-870c-57671c2843f7") : secret "insights-runtime-extractor-tls" not found Sep 17 07:28:47.890220 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:47.890134 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-lqcw7" event={"ID":"a236d1bd-81b1-4c9e-a14f-997204ad1b98","Type":"ContainerStarted","Data":"65a127609ac6e02d658b0a059084371a7422414cf97bc071f84b7a35956f391d"} Sep 17 07:28:47.891685 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:47.891658 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-bvxfq" event={"ID":"82555a36-653a-4106-a31e-b4da6445b838","Type":"ContainerStarted","Data":"7f12061bf9938f910fb6145e03f64848ed5938e2a528dd105ee6f9c9bab69846"} Sep 17 07:28:47.891819 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:47.891713 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-bvxfq" Sep 17 07:28:47.910380 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:47.910334 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-lqcw7" podStartSLOduration=66.17724404 podStartE2EDuration="1m10.910318052s" totalImagesPullingTime="4.733074012s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 07:27:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 07:28:47.90855959 +0000 UTC m=+71.789593948" watchObservedRunningTime="2026-09-17 07:28:47.910318052 +0000 UTC m=+71.791352412" Sep 17 07:28:47.927754 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:47.927718 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-bvxfq" podStartSLOduration=66.112113706 podStartE2EDuration="1m10.927705327s" totalImagesPullingTime="4.815591621s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 07:27:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 07:28:47.926481996 +0000 UTC m=+71.807516379" watchObservedRunningTime="2026-09-17 07:28:47.927705327 +0000 UTC m=+71.808739717" Sep 17 07:28:47.948531 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:47.948504 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67157086-7b61-49c2-8862-e594a9881821-service-ca-bundle\") pod \"router-default-764969ff56-mgc25\" (UID: \"67157086-7b61-49c2-8862-e594a9881821\") " pod="openshift-ingress/router-default-764969ff56-mgc25" Sep 17 07:28:47.948628 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:28:47.948611 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/67157086-7b61-49c2-8862-e594a9881821-service-ca-bundle podName:67157086-7b61-49c2-8862-e594a9881821 nodeName:}" failed. No retries permitted until 2026-09-17 07:28:55.948594448 +0000 UTC m=+79.829628801 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/67157086-7b61-49c2-8862-e594a9881821-service-ca-bundle") pod "router-default-764969ff56-mgc25" (UID: "67157086-7b61-49c2-8862-e594a9881821") : configmap references non-existent config key: service-ca.crt Sep 17 07:28:47.948700 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:47.948647 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67157086-7b61-49c2-8862-e594a9881821-metrics-certs\") pod \"router-default-764969ff56-mgc25\" (UID: \"67157086-7b61-49c2-8862-e594a9881821\") " pod="openshift-ingress/router-default-764969ff56-mgc25" Sep 17 07:28:47.948791 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:28:47.948774 2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 17 07:28:47.948843 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:28:47.948821 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67157086-7b61-49c2-8862-e594a9881821-metrics-certs podName:67157086-7b61-49c2-8862-e594a9881821 nodeName:}" failed. No retries permitted until 2026-09-17 07:28:55.94880667 +0000 UTC m=+79.829841007 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/67157086-7b61-49c2-8862-e594a9881821-metrics-certs") pod "router-default-764969ff56-mgc25" (UID: "67157086-7b61-49c2-8862-e594a9881821") : secret "router-metrics-certs-default" not found Sep 17 07:28:51.675845 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:51.675812 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ed949cd2-5046-4ef7-870c-57671c2843f7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zkvmd\" (UID: \"ed949cd2-5046-4ef7-870c-57671c2843f7\") " pod="openshift-insights/insights-runtime-extractor-zkvmd" Sep 17 07:28:51.676232 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:51.675901 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ed949cd2-5046-4ef7-870c-57671c2843f7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zkvmd\" (UID: \"ed949cd2-5046-4ef7-870c-57671c2843f7\") " pod="openshift-insights/insights-runtime-extractor-zkvmd" Sep 17 07:28:51.676232 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:28:51.675928 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 17 07:28:51.676232 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:28:51.675998 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ed949cd2-5046-4ef7-870c-57671c2843f7-kube-rbac-proxy-cm podName:ed949cd2-5046-4ef7-870c-57671c2843f7 nodeName:}" failed. No retries permitted until 2026-09-17 07:28:59.675983641 +0000 UTC m=+83.557017978 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ed949cd2-5046-4ef7-870c-57671c2843f7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zkvmd" (UID: "ed949cd2-5046-4ef7-870c-57671c2843f7") : configmap "kube-rbac-proxy" not found Sep 17 07:28:51.676232 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:28:51.676000 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 17 07:28:51.676232 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:28:51.676045 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed949cd2-5046-4ef7-870c-57671c2843f7-insights-runtime-extractor-tls podName:ed949cd2-5046-4ef7-870c-57671c2843f7 nodeName:}" failed. No retries permitted until 2026-09-17 07:28:59.67603451 +0000 UTC m=+83.557068846 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ed949cd2-5046-4ef7-870c-57671c2843f7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zkvmd" (UID: "ed949cd2-5046-4ef7-870c-57671c2843f7") : secret "insights-runtime-extractor-tls" not found Sep 17 07:28:56.008227 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:56.008184 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67157086-7b61-49c2-8862-e594a9881821-service-ca-bundle\") pod \"router-default-764969ff56-mgc25\" (UID: \"67157086-7b61-49c2-8862-e594a9881821\") " pod="openshift-ingress/router-default-764969ff56-mgc25" Sep 17 07:28:56.008227 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:56.008227 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67157086-7b61-49c2-8862-e594a9881821-metrics-certs\") pod \"router-default-764969ff56-mgc25\" (UID: \"67157086-7b61-49c2-8862-e594a9881821\") " pod="openshift-ingress/router-default-764969ff56-mgc25" Sep 17 07:28:56.008721 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:28:56.008319 2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 17 07:28:56.008721 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:28:56.008342 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/67157086-7b61-49c2-8862-e594a9881821-service-ca-bundle podName:67157086-7b61-49c2-8862-e594a9881821 nodeName:}" failed. No retries permitted until 2026-09-17 07:29:12.008326859 +0000 UTC m=+95.889361196 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/67157086-7b61-49c2-8862-e594a9881821-service-ca-bundle") pod "router-default-764969ff56-mgc25" (UID: "67157086-7b61-49c2-8862-e594a9881821") : configmap references non-existent config key: service-ca.crt Sep 17 07:28:56.008721 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:28:56.008366 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67157086-7b61-49c2-8862-e594a9881821-metrics-certs podName:67157086-7b61-49c2-8862-e594a9881821 nodeName:}" failed. No retries permitted until 2026-09-17 07:29:12.008359427 +0000 UTC m=+95.889393763 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/67157086-7b61-49c2-8862-e594a9881821-metrics-certs") pod "router-default-764969ff56-mgc25" (UID: "67157086-7b61-49c2-8862-e594a9881821") : secret "router-metrics-certs-default" not found Sep 17 07:28:59.736323 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:59.736171 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ed949cd2-5046-4ef7-870c-57671c2843f7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zkvmd\" (UID: \"ed949cd2-5046-4ef7-870c-57671c2843f7\") " pod="openshift-insights/insights-runtime-extractor-zkvmd" Sep 17 07:28:59.736323 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:59.736279 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ed949cd2-5046-4ef7-870c-57671c2843f7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zkvmd\" (UID: \"ed949cd2-5046-4ef7-870c-57671c2843f7\") " pod="openshift-insights/insights-runtime-extractor-zkvmd" Sep 17 07:28:59.736323 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:28:59.736313 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 17 07:28:59.736952 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:28:59.736378 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed949cd2-5046-4ef7-870c-57671c2843f7-insights-runtime-extractor-tls podName:ed949cd2-5046-4ef7-870c-57671c2843f7 nodeName:}" failed. No retries permitted until 2026-09-17 07:29:15.736363491 +0000 UTC m=+99.617397853 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ed949cd2-5046-4ef7-870c-57671c2843f7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zkvmd" (UID: "ed949cd2-5046-4ef7-870c-57671c2843f7") : secret "insights-runtime-extractor-tls" not found Sep 17 07:28:59.736952 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:28:59.736938 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ed949cd2-5046-4ef7-870c-57671c2843f7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zkvmd\" (UID: \"ed949cd2-5046-4ef7-870c-57671c2843f7\") " pod="openshift-insights/insights-runtime-extractor-zkvmd" Sep 17 07:29:12.026438 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:12.026401 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67157086-7b61-49c2-8862-e594a9881821-service-ca-bundle\") pod \"router-default-764969ff56-mgc25\" (UID: \"67157086-7b61-49c2-8862-e594a9881821\") " pod="openshift-ingress/router-default-764969ff56-mgc25" Sep 17 07:29:12.026438 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:12.026436 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67157086-7b61-49c2-8862-e594a9881821-metrics-certs\") pod \"router-default-764969ff56-mgc25\" (UID: \"67157086-7b61-49c2-8862-e594a9881821\") " pod="openshift-ingress/router-default-764969ff56-mgc25" Sep 17 07:29:12.027199 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:12.027176 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67157086-7b61-49c2-8862-e594a9881821-service-ca-bundle\") pod \"router-default-764969ff56-mgc25\" (UID: \"67157086-7b61-49c2-8862-e594a9881821\") " pod="openshift-ingress/router-default-764969ff56-mgc25" Sep 17 07:29:12.028775 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:12.028753 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67157086-7b61-49c2-8862-e594a9881821-metrics-certs\") pod \"router-default-764969ff56-mgc25\" (UID: \"67157086-7b61-49c2-8862-e594a9881821\") " pod="openshift-ingress/router-default-764969ff56-mgc25" Sep 17 07:29:12.252035 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:12.252006 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-764969ff56-mgc25" Sep 17 07:29:12.521893 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:12.521846 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-764969ff56-mgc25"] Sep 17 07:29:12.524614 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:29:12.524590 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67157086_7b61_49c2_8862_e594a9881821.slice/crio-8a2b5ab11019d940b3d14ca66a3a496dac5b61eeaa16b3a7e668529e7c95a426 WatchSource:0}: Error finding container 8a2b5ab11019d940b3d14ca66a3a496dac5b61eeaa16b3a7e668529e7c95a426: Status 404 returned error can't find the container with id 8a2b5ab11019d940b3d14ca66a3a496dac5b61eeaa16b3a7e668529e7c95a426 Sep 17 07:29:12.951307 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:12.951276 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-764969ff56-mgc25" event={"ID":"67157086-7b61-49c2-8862-e594a9881821","Type":"ContainerStarted","Data":"cac4c8451a8ad92976d6918b61df44eb453a593ba4f1f455119a10a124a7d06b"} Sep 17 07:29:12.951452 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:12.951321 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-764969ff56-mgc25" event={"ID":"67157086-7b61-49c2-8862-e594a9881821","Type":"ContainerStarted","Data":"8a2b5ab11019d940b3d14ca66a3a496dac5b61eeaa16b3a7e668529e7c95a426"} Sep 17 07:29:13.252894 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:13.252805 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-764969ff56-mgc25" Sep 17 07:29:13.255468 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:13.255444 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-764969ff56-mgc25" Sep 17 07:29:13.292157 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:13.292114 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-764969ff56-mgc25" podStartSLOduration=33.292100383 podStartE2EDuration="33.292100383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 07:28:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 07:29:12.980315524 +0000 UTC m=+96.861349884" watchObservedRunningTime="2026-09-17 07:29:13.292100383 +0000 UTC m=+97.173134742" Sep 17 07:29:13.839541 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:13.839512 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/44eefc80-34c2-47a2-ab48-e7e685f7dfe3-registry-tls\") pod \"image-registry-648fcb787c-kxf78\" (UID: \"44eefc80-34c2-47a2-ab48-e7e685f7dfe3\") " pod="openshift-image-registry/image-registry-648fcb787c-kxf78" Sep 17 07:29:13.841893 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:13.841851 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/44eefc80-34c2-47a2-ab48-e7e685f7dfe3-registry-tls\") pod \"image-registry-648fcb787c-kxf78\" (UID: \"44eefc80-34c2-47a2-ab48-e7e685f7dfe3\") " pod="openshift-image-registry/image-registry-648fcb787c-kxf78" Sep 17 07:29:13.910030 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:13.910005 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-vtldk\"" Sep 17 07:29:13.916355 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:13.916339 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-648fcb787c-kxf78" Sep 17 07:29:13.940752 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:13.940726 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2fcfd678-3207-4e47-a6d4-3d1416568d1c-metrics-tls\") pod \"dns-default-nfjvh\" (UID: \"2fcfd678-3207-4e47-a6d4-3d1416568d1c\") " pod="openshift-dns/dns-default-nfjvh" Sep 17 07:29:13.942972 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:13.942944 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2fcfd678-3207-4e47-a6d4-3d1416568d1c-metrics-tls\") pod \"dns-default-nfjvh\" (UID: \"2fcfd678-3207-4e47-a6d4-3d1416568d1c\") " pod="openshift-dns/dns-default-nfjvh" Sep 17 07:29:13.954292 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:13.954273 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-764969ff56-mgc25" Sep 17 07:29:13.955492 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:13.955472 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-764969ff56-mgc25" Sep 17 07:29:14.189229 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:14.189200 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-648fcb787c-kxf78"] Sep 17 07:29:14.192953 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:29:14.192928 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44eefc80_34c2_47a2_ab48_e7e685f7dfe3.slice/crio-4f7ac77aa2de930155129abaa5c1d298cea21c932f882d04586750b3052538c7 WatchSource:0}: Error finding container 4f7ac77aa2de930155129abaa5c1d298cea21c932f882d04586750b3052538c7: Status 404 returned error can't find the container with id 4f7ac77aa2de930155129abaa5c1d298cea21c932f882d04586750b3052538c7 Sep 17 07:29:14.240688 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:14.240668 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-bnzll\"" Sep 17 07:29:14.245265 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:14.245247 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-nfjvh" Sep 17 07:29:14.531133 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:14.531067 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-nfjvh"] Sep 17 07:29:14.534431 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:29:14.534404 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fcfd678_3207_4e47_a6d4_3d1416568d1c.slice/crio-6eda2e5d7cd7c613453ae27193c7e6c6a79f3cf06f1fb8007bd6d13ad31c3410 WatchSource:0}: Error finding container 6eda2e5d7cd7c613453ae27193c7e6c6a79f3cf06f1fb8007bd6d13ad31c3410: Status 404 returned error can't find the container with id 6eda2e5d7cd7c613453ae27193c7e6c6a79f3cf06f1fb8007bd6d13ad31c3410 Sep 17 07:29:14.957930 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:14.957900 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nfjvh" event={"ID":"2fcfd678-3207-4e47-a6d4-3d1416568d1c","Type":"ContainerStarted","Data":"6eda2e5d7cd7c613453ae27193c7e6c6a79f3cf06f1fb8007bd6d13ad31c3410"} Sep 17 07:29:14.959351 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:14.959332 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-648fcb787c-kxf78" event={"ID":"44eefc80-34c2-47a2-ab48-e7e685f7dfe3","Type":"ContainerStarted","Data":"f784bfd05b8bc20da17f6d7f822a39780f27a30bbc6044f12c87ef9acb50c943"} Sep 17 07:29:14.959445 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:14.959367 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-648fcb787c-kxf78" event={"ID":"44eefc80-34c2-47a2-ab48-e7e685f7dfe3","Type":"ContainerStarted","Data":"4f7ac77aa2de930155129abaa5c1d298cea21c932f882d04586750b3052538c7"} Sep 17 07:29:14.959445 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:14.959435 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-648fcb787c-kxf78" Sep 17 07:29:14.997933 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:14.997878 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-648fcb787c-kxf78" podStartSLOduration=97.997844232 podStartE2EDuration="1m37.997844232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 07:27:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 07:29:14.996852726 +0000 UTC m=+98.877887097" watchObservedRunningTime="2026-09-17 07:29:14.997844232 +0000 UTC m=+98.878878592" Sep 17 07:29:15.753997 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:15.753955 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ed949cd2-5046-4ef7-870c-57671c2843f7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zkvmd\" (UID: \"ed949cd2-5046-4ef7-870c-57671c2843f7\") " pod="openshift-insights/insights-runtime-extractor-zkvmd" Sep 17 07:29:15.756486 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:15.756460 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ed949cd2-5046-4ef7-870c-57671c2843f7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zkvmd\" (UID: \"ed949cd2-5046-4ef7-870c-57671c2843f7\") " pod="openshift-insights/insights-runtime-extractor-zkvmd" Sep 17 07:29:15.825750 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:15.825715 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-zkvmd" Sep 17 07:29:16.289275 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:16.289208 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-zkvmd"] Sep 17 07:29:16.292427 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:29:16.292401 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded949cd2_5046_4ef7_870c_57671c2843f7.slice/crio-8b49deee1c783e3038cdd3aeb6b8b729791f0467d3fe74c3ec8642caf568ede6 WatchSource:0}: Error finding container 8b49deee1c783e3038cdd3aeb6b8b729791f0467d3fe74c3ec8642caf568ede6: Status 404 returned error can't find the container with id 8b49deee1c783e3038cdd3aeb6b8b729791f0467d3fe74c3ec8642caf568ede6 Sep 17 07:29:16.968825 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:16.968791 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nfjvh" event={"ID":"2fcfd678-3207-4e47-a6d4-3d1416568d1c","Type":"ContainerStarted","Data":"79397d1e2d3bfde5f6f7feb7b2399430a64f5b20315e0668af139000cdc89902"} Sep 17 07:29:16.969183 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:16.968892 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nfjvh" event={"ID":"2fcfd678-3207-4e47-a6d4-3d1416568d1c","Type":"ContainerStarted","Data":"cde4a58a0bf1df12fd54ff1340d9b0939fb42d0e2aacb71c1dae46ff3c374d58"} Sep 17 07:29:16.969183 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:16.968923 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-nfjvh" Sep 17 07:29:16.970419 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:16.970397 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zkvmd" event={"ID":"ed949cd2-5046-4ef7-870c-57671c2843f7","Type":"ContainerStarted","Data":"af9d27d5299b5e2e51ee885ab6f0dfee3f8d18e75c0be4e5a28f54f24741383a"} Sep 17 07:29:16.970476 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:16.970441 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zkvmd" event={"ID":"ed949cd2-5046-4ef7-870c-57671c2843f7","Type":"ContainerStarted","Data":"3fb9c6f5d6b1f858b1cba583fbd86322b51fe706657600d2cf2b909c118acb9e"} Sep 17 07:29:16.970476 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:16.970459 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zkvmd" event={"ID":"ed949cd2-5046-4ef7-870c-57671c2843f7","Type":"ContainerStarted","Data":"8b49deee1c783e3038cdd3aeb6b8b729791f0467d3fe74c3ec8642caf568ede6"} Sep 17 07:29:16.988673 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:16.988632 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-nfjvh" podStartSLOduration=66.480502596 podStartE2EDuration="1m7.988616201s" totalImagesPullingTime="1.508113605s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 07:28:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 07:29:16.987940641 +0000 UTC m=+100.868975001" watchObservedRunningTime="2026-09-17 07:29:16.988616201 +0000 UTC m=+100.869650559" Sep 17 07:29:18.895589 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:18.895564 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-bvxfq" Sep 17 07:29:18.981103 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:18.981079 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zkvmd" event={"ID":"ed949cd2-5046-4ef7-870c-57671c2843f7","Type":"ContainerStarted","Data":"4e52f512dc8cea0fc30d08331de9e925695abcf4a2e120e4f8113cd6eab76ba2"} Sep 17 07:29:20.584347 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:20.584296 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-zkvmd" podStartSLOduration=35.361444447 podStartE2EDuration="37.584277934s" totalImagesPullingTime="2.222833487s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 07:28:43 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 07:29:19.004052602 +0000 UTC m=+102.885086960" watchObservedRunningTime="2026-09-17 07:29:20.584277934 +0000 UTC m=+104.465312293" Sep 17 07:29:20.585095 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:20.585076 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-8hwpw"] Sep 17 07:29:20.606400 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:20.606378 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-8hwpw" Sep 17 07:29:20.613044 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:20.613022 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Sep 17 07:29:20.613175 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:20.613024 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-82gjc\"" Sep 17 07:29:20.613175 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:20.613069 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Sep 17 07:29:20.625776 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:20.622669 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-8hwpw"] Sep 17 07:29:20.666569 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:20.666533 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-bjh22"] Sep 17 07:29:20.688634 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:20.688609 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-bjh22" Sep 17 07:29:20.692101 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:20.692080 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0a5a4190-dc08-4f8e-ab3c-af69dea0d11f-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-8hwpw\" (UID: \"0a5a4190-dc08-4f8e-ab3c-af69dea0d11f\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-8hwpw" Sep 17 07:29:20.692227 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:20.692196 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0a5a4190-dc08-4f8e-ab3c-af69dea0d11f-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-8hwpw\" (UID: \"0a5a4190-dc08-4f8e-ab3c-af69dea0d11f\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-8hwpw" Sep 17 07:29:20.693965 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:20.693929 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-bjh22"] Sep 17 07:29:20.703281 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:20.703262 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-xbh7p\"" Sep 17 07:29:20.792914 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:20.792888 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqwbr\" (UniqueName: \"kubernetes.io/projected/4f97baec-aba5-4b9f-afd8-717a5ad6c84e-kube-api-access-lqwbr\") pod \"network-check-source-6f8b768455-bjh22\" (UID: \"4f97baec-aba5-4b9f-afd8-717a5ad6c84e\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-bjh22" Sep 17 07:29:20.793013 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:20.792938 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0a5a4190-dc08-4f8e-ab3c-af69dea0d11f-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-8hwpw\" (UID: \"0a5a4190-dc08-4f8e-ab3c-af69dea0d11f\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-8hwpw" Sep 17 07:29:20.793013 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:20.792968 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0a5a4190-dc08-4f8e-ab3c-af69dea0d11f-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-8hwpw\" (UID: \"0a5a4190-dc08-4f8e-ab3c-af69dea0d11f\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-8hwpw" Sep 17 07:29:20.793492 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:20.793469 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0a5a4190-dc08-4f8e-ab3c-af69dea0d11f-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-8hwpw\" (UID: \"0a5a4190-dc08-4f8e-ab3c-af69dea0d11f\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-8hwpw" Sep 17 07:29:20.795258 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:20.795240 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0a5a4190-dc08-4f8e-ab3c-af69dea0d11f-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-8hwpw\" (UID: \"0a5a4190-dc08-4f8e-ab3c-af69dea0d11f\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-8hwpw" Sep 17 07:29:20.849492 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:20.849433 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-9d2qq"] Sep 17 07:29:20.870137 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:20.870112 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-76f99584f4-z7dwl"] Sep 17 07:29:20.870272 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:20.870253 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-9d2qq" Sep 17 07:29:20.875442 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:20.875426 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Sep 17 07:29:20.884410 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:20.884391 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-z2dtp\"" Sep 17 07:29:20.892332 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:20.892294 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-z7dwl" Sep 17 07:29:20.893653 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:20.893607 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqwbr\" (UniqueName: \"kubernetes.io/projected/4f97baec-aba5-4b9f-afd8-717a5ad6c84e-kube-api-access-lqwbr\") pod \"network-check-source-6f8b768455-bjh22\" (UID: \"4f97baec-aba5-4b9f-afd8-717a5ad6c84e\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-bjh22" Sep 17 07:29:20.894067 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:20.894050 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-z7dwl"] Sep 17 07:29:20.895045 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:20.894951 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-9d2qq"] Sep 17 07:29:20.899044 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:20.899027 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 17 07:29:20.901239 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:20.901224 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 17 07:29:20.901437 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:20.901422 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-mjdzb\"" Sep 17 07:29:20.915082 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:20.915066 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-8hwpw" Sep 17 07:29:20.925006 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:20.924990 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqwbr\" (UniqueName: \"kubernetes.io/projected/4f97baec-aba5-4b9f-afd8-717a5ad6c84e-kube-api-access-lqwbr\") pod \"network-check-source-6f8b768455-bjh22\" (UID: \"4f97baec-aba5-4b9f-afd8-717a5ad6c84e\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-bjh22" Sep 17 07:29:20.998608 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:20.994423 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/a63c0215-fb21-4077-bc08-1f227674a5b7-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-9d2qq\" (UID: \"a63c0215-fb21-4077-bc08-1f227674a5b7\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-9d2qq" Sep 17 07:29:20.998608 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:20.994491 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qw2hd\" (UniqueName: \"kubernetes.io/projected/ef341fec-b3aa-45f8-9697-70bbf084fc7f-kube-api-access-qw2hd\") pod \"downloads-76f99584f4-z7dwl\" (UID: \"ef341fec-b3aa-45f8-9697-70bbf084fc7f\") " pod="openshift-console/downloads-76f99584f4-z7dwl" Sep 17 07:29:20.998608 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:20.998062 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-bjh22" Sep 17 07:29:21.095278 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:21.095250 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/a63c0215-fb21-4077-bc08-1f227674a5b7-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-9d2qq\" (UID: \"a63c0215-fb21-4077-bc08-1f227674a5b7\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-9d2qq" Sep 17 07:29:21.095393 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:21.095303 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qw2hd\" (UniqueName: \"kubernetes.io/projected/ef341fec-b3aa-45f8-9697-70bbf084fc7f-kube-api-access-qw2hd\") pod \"downloads-76f99584f4-z7dwl\" (UID: \"ef341fec-b3aa-45f8-9697-70bbf084fc7f\") " pod="openshift-console/downloads-76f99584f4-z7dwl" Sep 17 07:29:21.097407 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:21.097385 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/a63c0215-fb21-4077-bc08-1f227674a5b7-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-9d2qq\" (UID: \"a63c0215-fb21-4077-bc08-1f227674a5b7\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-9d2qq" Sep 17 07:29:21.106240 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:21.106197 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qw2hd\" (UniqueName: \"kubernetes.io/projected/ef341fec-b3aa-45f8-9697-70bbf084fc7f-kube-api-access-qw2hd\") pod \"downloads-76f99584f4-z7dwl\" (UID: \"ef341fec-b3aa-45f8-9697-70bbf084fc7f\") " pod="openshift-console/downloads-76f99584f4-z7dwl" Sep 17 07:29:21.178288 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:21.178266 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-9d2qq" Sep 17 07:29:21.201754 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:21.201726 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-z7dwl" Sep 17 07:29:21.202841 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:21.202821 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-8hwpw"] Sep 17 07:29:21.209713 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:29:21.209686 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a5a4190_dc08_4f8e_ab3c_af69dea0d11f.slice/crio-2f8507d78934cad85cc81284dd34ece10023c53d4ed69caa806c6a8f56ffdbe1 WatchSource:0}: Error finding container 2f8507d78934cad85cc81284dd34ece10023c53d4ed69caa806c6a8f56ffdbe1: Status 404 returned error can't find the container with id 2f8507d78934cad85cc81284dd34ece10023c53d4ed69caa806c6a8f56ffdbe1 Sep 17 07:29:21.317350 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:21.317325 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-bjh22"] Sep 17 07:29:21.325689 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:29:21.325667 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f97baec_aba5_4b9f_afd8_717a5ad6c84e.slice/crio-bcdeeb8a2b06be2206932650a2714d210c6b885d69de0ab45baeedf9cb20b476 WatchSource:0}: Error finding container bcdeeb8a2b06be2206932650a2714d210c6b885d69de0ab45baeedf9cb20b476: Status 404 returned error can't find the container with id bcdeeb8a2b06be2206932650a2714d210c6b885d69de0ab45baeedf9cb20b476 Sep 17 07:29:21.459611 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:21.459578 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-9d2qq"] Sep 17 07:29:21.462149 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:29:21.462124 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda63c0215_fb21_4077_bc08_1f227674a5b7.slice/crio-8e4e0d6ccff8c016e1cfca0c026ab6674d4f8cfbb230cd8b8d9a12e187865ad2 WatchSource:0}: Error finding container 8e4e0d6ccff8c016e1cfca0c026ab6674d4f8cfbb230cd8b8d9a12e187865ad2: Status 404 returned error can't find the container with id 8e4e0d6ccff8c016e1cfca0c026ab6674d4f8cfbb230cd8b8d9a12e187865ad2 Sep 17 07:29:21.480054 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:21.480032 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-z7dwl"] Sep 17 07:29:21.482817 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:29:21.482795 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef341fec_b3aa_45f8_9697_70bbf084fc7f.slice/crio-166e6a22607d361e37b96adb2479aeae56d6e81d3c7ff29b1fce27acb26642a5 WatchSource:0}: Error finding container 166e6a22607d361e37b96adb2479aeae56d6e81d3c7ff29b1fce27acb26642a5: Status 404 returned error can't find the container with id 166e6a22607d361e37b96adb2479aeae56d6e81d3c7ff29b1fce27acb26642a5 Sep 17 07:29:22.006049 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:22.006000 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-bjh22" event={"ID":"4f97baec-aba5-4b9f-afd8-717a5ad6c84e","Type":"ContainerStarted","Data":"0a442527357de0ba099d61a5d281fe4c0fdfce260ba386e1f6136ca150e23e51"} Sep 17 07:29:22.006489 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:22.006063 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-bjh22" event={"ID":"4f97baec-aba5-4b9f-afd8-717a5ad6c84e","Type":"ContainerStarted","Data":"bcdeeb8a2b06be2206932650a2714d210c6b885d69de0ab45baeedf9cb20b476"} Sep 17 07:29:22.009304 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:22.009268 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-8hwpw" event={"ID":"0a5a4190-dc08-4f8e-ab3c-af69dea0d11f","Type":"ContainerStarted","Data":"2f8507d78934cad85cc81284dd34ece10023c53d4ed69caa806c6a8f56ffdbe1"} Sep 17 07:29:22.010586 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:22.010559 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-z7dwl" event={"ID":"ef341fec-b3aa-45f8-9697-70bbf084fc7f","Type":"ContainerStarted","Data":"166e6a22607d361e37b96adb2479aeae56d6e81d3c7ff29b1fce27acb26642a5"} Sep 17 07:29:22.012736 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:22.012673 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-9d2qq" event={"ID":"a63c0215-fb21-4077-bc08-1f227674a5b7","Type":"ContainerStarted","Data":"8e4e0d6ccff8c016e1cfca0c026ab6674d4f8cfbb230cd8b8d9a12e187865ad2"} Sep 17 07:29:22.025968 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:22.025918 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f8b768455-bjh22" podStartSLOduration=2.025898178 podStartE2EDuration="2.025898178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 07:29:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 07:29:22.02385272 +0000 UTC m=+105.904887080" watchObservedRunningTime="2026-09-17 07:29:22.025898178 +0000 UTC m=+105.906932538" Sep 17 07:29:23.018796 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:23.018757 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-8hwpw" event={"ID":"0a5a4190-dc08-4f8e-ab3c-af69dea0d11f","Type":"ContainerStarted","Data":"303eadcca041532d9f6df37b6d35d75911b3c9726220aa86c679d9026a0d4aa9"} Sep 17 07:29:23.020508 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:23.020471 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-9d2qq" event={"ID":"a63c0215-fb21-4077-bc08-1f227674a5b7","Type":"ContainerStarted","Data":"4068262015c91968d55a0a1e3dc66341052db167e0d0b3b14b31c0d2b98eb085"} Sep 17 07:29:23.038342 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:23.038226 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-8hwpw" podStartSLOduration=1.661656084 podStartE2EDuration="3.03821049s" totalImagesPullingTime="1.376554406s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 07:29:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 07:29:23.037362777 +0000 UTC m=+106.918397137" watchObservedRunningTime="2026-09-17 07:29:23.03821049 +0000 UTC m=+106.919244851" Sep 17 07:29:23.058511 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:23.057200 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-9d2qq" podStartSLOduration=1.909940505 podStartE2EDuration="3.057181488s" totalImagesPullingTime="1.147240983s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 07:29:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 07:29:23.056497717 +0000 UTC m=+106.937532076" watchObservedRunningTime="2026-09-17 07:29:23.057181488 +0000 UTC m=+106.938215849" Sep 17 07:29:24.024073 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:24.024039 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-9d2qq" Sep 17 07:29:24.029172 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:24.029143 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-9d2qq" Sep 17 07:29:24.280939 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:24.280843 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-bcc9d4dd5-bn4zl"] Sep 17 07:29:24.283742 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:24.283722 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bcc9d4dd5-bn4zl" Sep 17 07:29:24.286369 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:24.286198 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 17 07:29:24.286369 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:24.286248 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 17 07:29:24.286369 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:24.286320 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 17 07:29:24.286369 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:24.286320 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 17 07:29:24.287087 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:24.287062 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 17 07:29:24.287196 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:24.287114 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-4hwfz\"" Sep 17 07:29:24.298573 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:24.298549 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-bcc9d4dd5-bn4zl"] Sep 17 07:29:24.369623 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:24.369596 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-m4894"] Sep 17 07:29:24.371808 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:24.371790 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-m4894" Sep 17 07:29:24.375349 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:24.375325 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 17 07:29:24.375848 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:24.375828 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Sep 17 07:29:24.375949 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:24.375896 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 17 07:29:24.376126 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:24.376096 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Sep 17 07:29:24.385909 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:24.385889 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 17 07:29:24.387000 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:24.386980 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-m4894"] Sep 17 07:29:24.392215 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:24.392194 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-758kp\"" Sep 17 07:29:24.420599 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:24.420572 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7e500620-3439-4f1e-834c-7134a5582b50-oauth-serving-cert\") pod \"console-bcc9d4dd5-bn4zl\" (UID: \"7e500620-3439-4f1e-834c-7134a5582b50\") " pod="openshift-console/console-bcc9d4dd5-bn4zl" Sep 17 07:29:24.420715 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:24.420631 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7e500620-3439-4f1e-834c-7134a5582b50-console-oauth-config\") pod \"console-bcc9d4dd5-bn4zl\" (UID: \"7e500620-3439-4f1e-834c-7134a5582b50\") " pod="openshift-console/console-bcc9d4dd5-bn4zl" Sep 17 07:29:24.420715 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:24.420691 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddqxw\" (UniqueName: \"kubernetes.io/projected/7e500620-3439-4f1e-834c-7134a5582b50-kube-api-access-ddqxw\") pod \"console-bcc9d4dd5-bn4zl\" (UID: \"7e500620-3439-4f1e-834c-7134a5582b50\") " pod="openshift-console/console-bcc9d4dd5-bn4zl" Sep 17 07:29:24.420820 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:24.420767 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7e500620-3439-4f1e-834c-7134a5582b50-console-config\") pod \"console-bcc9d4dd5-bn4zl\" (UID: \"7e500620-3439-4f1e-834c-7134a5582b50\") " pod="openshift-console/console-bcc9d4dd5-bn4zl" Sep 17 07:29:24.420820 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:24.420805 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7e500620-3439-4f1e-834c-7134a5582b50-service-ca\") pod \"console-bcc9d4dd5-bn4zl\" (UID: \"7e500620-3439-4f1e-834c-7134a5582b50\") " pod="openshift-console/console-bcc9d4dd5-bn4zl" Sep 17 07:29:24.420972 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:24.420838 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7e500620-3439-4f1e-834c-7134a5582b50-console-serving-cert\") pod \"console-bcc9d4dd5-bn4zl\" (UID: \"7e500620-3439-4f1e-834c-7134a5582b50\") " pod="openshift-console/console-bcc9d4dd5-bn4zl" Sep 17 07:29:24.521714 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:24.521682 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a7d2fc30-22e9-4052-82f1-83ec99c2e632-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-m4894\" (UID: \"a7d2fc30-22e9-4052-82f1-83ec99c2e632\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-m4894" Sep 17 07:29:24.521878 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:24.521727 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddqxw\" (UniqueName: \"kubernetes.io/projected/7e500620-3439-4f1e-834c-7134a5582b50-kube-api-access-ddqxw\") pod \"console-bcc9d4dd5-bn4zl\" (UID: \"7e500620-3439-4f1e-834c-7134a5582b50\") " pod="openshift-console/console-bcc9d4dd5-bn4zl" Sep 17 07:29:24.521878 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:24.521758 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bscvs\" (UniqueName: \"kubernetes.io/projected/a7d2fc30-22e9-4052-82f1-83ec99c2e632-kube-api-access-bscvs\") pod \"prometheus-operator-68bcd7f987-m4894\" (UID: \"a7d2fc30-22e9-4052-82f1-83ec99c2e632\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-m4894" Sep 17 07:29:24.521878 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:24.521839 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7e500620-3439-4f1e-834c-7134a5582b50-console-config\") pod \"console-bcc9d4dd5-bn4zl\" (UID: \"7e500620-3439-4f1e-834c-7134a5582b50\") " pod="openshift-console/console-bcc9d4dd5-bn4zl" Sep 17 07:29:24.522067 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:24.521895 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7e500620-3439-4f1e-834c-7134a5582b50-service-ca\") pod \"console-bcc9d4dd5-bn4zl\" (UID: \"7e500620-3439-4f1e-834c-7134a5582b50\") " pod="openshift-console/console-bcc9d4dd5-bn4zl" Sep 17 07:29:24.522067 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:24.521927 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a7d2fc30-22e9-4052-82f1-83ec99c2e632-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-m4894\" (UID: \"a7d2fc30-22e9-4052-82f1-83ec99c2e632\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-m4894" Sep 17 07:29:24.522067 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:24.521965 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7e500620-3439-4f1e-834c-7134a5582b50-console-serving-cert\") pod \"console-bcc9d4dd5-bn4zl\" (UID: \"7e500620-3439-4f1e-834c-7134a5582b50\") " pod="openshift-console/console-bcc9d4dd5-bn4zl" Sep 17 07:29:24.522067 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:24.522001 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7e500620-3439-4f1e-834c-7134a5582b50-oauth-serving-cert\") pod \"console-bcc9d4dd5-bn4zl\" (UID: \"7e500620-3439-4f1e-834c-7134a5582b50\") " pod="openshift-console/console-bcc9d4dd5-bn4zl" Sep 17 07:29:24.522067 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:24.522050 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7e500620-3439-4f1e-834c-7134a5582b50-console-oauth-config\") pod \"console-bcc9d4dd5-bn4zl\" (UID: \"7e500620-3439-4f1e-834c-7134a5582b50\") " pod="openshift-console/console-bcc9d4dd5-bn4zl" Sep 17 07:29:24.522315 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:24.522083 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/a7d2fc30-22e9-4052-82f1-83ec99c2e632-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-m4894\" (UID: \"a7d2fc30-22e9-4052-82f1-83ec99c2e632\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-m4894" Sep 17 07:29:24.522622 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:24.522597 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7e500620-3439-4f1e-834c-7134a5582b50-console-config\") pod \"console-bcc9d4dd5-bn4zl\" (UID: \"7e500620-3439-4f1e-834c-7134a5582b50\") " pod="openshift-console/console-bcc9d4dd5-bn4zl" Sep 17 07:29:24.522814 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:24.522795 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7e500620-3439-4f1e-834c-7134a5582b50-service-ca\") pod \"console-bcc9d4dd5-bn4zl\" (UID: \"7e500620-3439-4f1e-834c-7134a5582b50\") " pod="openshift-console/console-bcc9d4dd5-bn4zl" Sep 17 07:29:24.522884 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:24.522827 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7e500620-3439-4f1e-834c-7134a5582b50-oauth-serving-cert\") pod \"console-bcc9d4dd5-bn4zl\" (UID: \"7e500620-3439-4f1e-834c-7134a5582b50\") " pod="openshift-console/console-bcc9d4dd5-bn4zl" Sep 17 07:29:24.524642 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:24.524620 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7e500620-3439-4f1e-834c-7134a5582b50-console-serving-cert\") pod \"console-bcc9d4dd5-bn4zl\" (UID: \"7e500620-3439-4f1e-834c-7134a5582b50\") " pod="openshift-console/console-bcc9d4dd5-bn4zl" Sep 17 07:29:24.524803 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:24.524781 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7e500620-3439-4f1e-834c-7134a5582b50-console-oauth-config\") pod \"console-bcc9d4dd5-bn4zl\" (UID: \"7e500620-3439-4f1e-834c-7134a5582b50\") " pod="openshift-console/console-bcc9d4dd5-bn4zl" Sep 17 07:29:24.531930 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:24.531886 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddqxw\" (UniqueName: \"kubernetes.io/projected/7e500620-3439-4f1e-834c-7134a5582b50-kube-api-access-ddqxw\") pod \"console-bcc9d4dd5-bn4zl\" (UID: \"7e500620-3439-4f1e-834c-7134a5582b50\") " pod="openshift-console/console-bcc9d4dd5-bn4zl" Sep 17 07:29:24.594017 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:24.593979 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bcc9d4dd5-bn4zl" Sep 17 07:29:24.622720 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:24.622685 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a7d2fc30-22e9-4052-82f1-83ec99c2e632-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-m4894\" (UID: \"a7d2fc30-22e9-4052-82f1-83ec99c2e632\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-m4894" Sep 17 07:29:24.622853 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:24.622745 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bscvs\" (UniqueName: \"kubernetes.io/projected/a7d2fc30-22e9-4052-82f1-83ec99c2e632-kube-api-access-bscvs\") pod \"prometheus-operator-68bcd7f987-m4894\" (UID: \"a7d2fc30-22e9-4052-82f1-83ec99c2e632\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-m4894" Sep 17 07:29:24.622853 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:24.622801 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a7d2fc30-22e9-4052-82f1-83ec99c2e632-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-m4894\" (UID: \"a7d2fc30-22e9-4052-82f1-83ec99c2e632\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-m4894" Sep 17 07:29:24.623019 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:24.622992 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/a7d2fc30-22e9-4052-82f1-83ec99c2e632-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-m4894\" (UID: \"a7d2fc30-22e9-4052-82f1-83ec99c2e632\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-m4894" Sep 17 07:29:24.623371 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:29:24.623075 2578 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Sep 17 07:29:24.623371 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:29:24.623153 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7d2fc30-22e9-4052-82f1-83ec99c2e632-prometheus-operator-tls podName:a7d2fc30-22e9-4052-82f1-83ec99c2e632 nodeName:}" failed. No retries permitted until 2026-09-17 07:29:25.123130104 +0000 UTC m=+109.004164448 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/a7d2fc30-22e9-4052-82f1-83ec99c2e632-prometheus-operator-tls") pod "prometheus-operator-68bcd7f987-m4894" (UID: "a7d2fc30-22e9-4052-82f1-83ec99c2e632") : secret "prometheus-operator-tls" not found Sep 17 07:29:24.623564 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:24.623543 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a7d2fc30-22e9-4052-82f1-83ec99c2e632-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-m4894\" (UID: \"a7d2fc30-22e9-4052-82f1-83ec99c2e632\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-m4894" Sep 17 07:29:24.625525 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:24.625474 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a7d2fc30-22e9-4052-82f1-83ec99c2e632-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-m4894\" (UID: \"a7d2fc30-22e9-4052-82f1-83ec99c2e632\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-m4894" Sep 17 07:29:24.633628 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:24.633604 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bscvs\" (UniqueName: \"kubernetes.io/projected/a7d2fc30-22e9-4052-82f1-83ec99c2e632-kube-api-access-bscvs\") pod \"prometheus-operator-68bcd7f987-m4894\" (UID: \"a7d2fc30-22e9-4052-82f1-83ec99c2e632\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-m4894" Sep 17 07:29:24.880535 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:24.880501 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-bcc9d4dd5-bn4zl"] Sep 17 07:29:24.883172 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:29:24.883133 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e500620_3439_4f1e_834c_7134a5582b50.slice/crio-4e0d39009d2788f29e2c33de24637ce3a18937a50a6daddadbbe2c27a17157d6 WatchSource:0}: Error finding container 4e0d39009d2788f29e2c33de24637ce3a18937a50a6daddadbbe2c27a17157d6: Status 404 returned error can't find the container with id 4e0d39009d2788f29e2c33de24637ce3a18937a50a6daddadbbe2c27a17157d6 Sep 17 07:29:25.029115 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:25.029069 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bcc9d4dd5-bn4zl" event={"ID":"7e500620-3439-4f1e-834c-7134a5582b50","Type":"ContainerStarted","Data":"4e0d39009d2788f29e2c33de24637ce3a18937a50a6daddadbbe2c27a17157d6"} Sep 17 07:29:25.127978 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:25.127945 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/a7d2fc30-22e9-4052-82f1-83ec99c2e632-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-m4894\" (UID: \"a7d2fc30-22e9-4052-82f1-83ec99c2e632\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-m4894" Sep 17 07:29:25.130720 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:25.130662 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/a7d2fc30-22e9-4052-82f1-83ec99c2e632-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-m4894\" (UID: \"a7d2fc30-22e9-4052-82f1-83ec99c2e632\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-m4894" Sep 17 07:29:25.282348 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:25.282316 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-m4894" Sep 17 07:29:25.565881 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:25.565792 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-m4894"] Sep 17 07:29:25.569654 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:29:25.569623 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7d2fc30_22e9_4052_82f1_83ec99c2e632.slice/crio-979b0b0c51114d2ad7ae294dee736d93b1d3545e4f8256dea3d27949c3d9a97b WatchSource:0}: Error finding container 979b0b0c51114d2ad7ae294dee736d93b1d3545e4f8256dea3d27949c3d9a97b: Status 404 returned error can't find the container with id 979b0b0c51114d2ad7ae294dee736d93b1d3545e4f8256dea3d27949c3d9a97b Sep 17 07:29:26.036654 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:26.036583 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-m4894" event={"ID":"a7d2fc30-22e9-4052-82f1-83ec99c2e632","Type":"ContainerStarted","Data":"979b0b0c51114d2ad7ae294dee736d93b1d3545e4f8256dea3d27949c3d9a97b"} Sep 17 07:29:26.977024 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:26.976993 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-nfjvh" Sep 17 07:29:29.052168 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:29.052131 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bcc9d4dd5-bn4zl" event={"ID":"7e500620-3439-4f1e-834c-7134a5582b50","Type":"ContainerStarted","Data":"2da01d9877fd029ce1e90d543d45061080f96ea12697586a2e0b4eb79f9c1abd"} Sep 17 07:29:29.054254 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:29.054225 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-m4894" event={"ID":"a7d2fc30-22e9-4052-82f1-83ec99c2e632","Type":"ContainerStarted","Data":"355e4ae7046a40ae31a1608d64d4e0a091bcfa93773ba440ee969cd8b3579352"} Sep 17 07:29:29.054391 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:29.054275 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-m4894" event={"ID":"a7d2fc30-22e9-4052-82f1-83ec99c2e632","Type":"ContainerStarted","Data":"8db737ae7aa83a3e2b96dd68e81de2af3d2781cd68dc43c904da12a2118e15ed"} Sep 17 07:29:29.077244 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:29.077195 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-bcc9d4dd5-bn4zl" podStartSLOduration=1.8678501170000001 podStartE2EDuration="5.077177053s" totalImagesPullingTime="3.209326936s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 07:29:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 07:29:29.076554777 +0000 UTC m=+112.957589138" watchObservedRunningTime="2026-09-17 07:29:29.077177053 +0000 UTC m=+112.958211412" Sep 17 07:29:29.107608 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:29.107524 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-68bcd7f987-m4894" podStartSLOduration=2.5921565319999997 podStartE2EDuration="5.107506632s" totalImagesPullingTime="2.5153501s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 07:29:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 07:29:29.107177004 +0000 UTC m=+112.988211363" watchObservedRunningTime="2026-09-17 07:29:29.107506632 +0000 UTC m=+112.988540986" Sep 17 07:29:29.216673 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:29.216646 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-67fd756cb9-f96vt"] Sep 17 07:29:29.219633 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:29.219612 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67fd756cb9-f96vt" Sep 17 07:29:29.228645 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:29.228623 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Sep 17 07:29:29.231689 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:29.231669 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-67fd756cb9-f96vt"] Sep 17 07:29:29.363433 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:29.363355 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/93b13dbc-04df-4472-890b-9bd51419e06c-trusted-ca-bundle\") pod \"console-67fd756cb9-f96vt\" (UID: \"93b13dbc-04df-4472-890b-9bd51419e06c\") " pod="openshift-console/console-67fd756cb9-f96vt" Sep 17 07:29:29.363574 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:29.363446 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpqr4\" (UniqueName: \"kubernetes.io/projected/93b13dbc-04df-4472-890b-9bd51419e06c-kube-api-access-zpqr4\") pod \"console-67fd756cb9-f96vt\" (UID: \"93b13dbc-04df-4472-890b-9bd51419e06c\") " pod="openshift-console/console-67fd756cb9-f96vt" Sep 17 07:29:29.363574 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:29.363490 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/93b13dbc-04df-4472-890b-9bd51419e06c-console-oauth-config\") pod \"console-67fd756cb9-f96vt\" (UID: \"93b13dbc-04df-4472-890b-9bd51419e06c\") " pod="openshift-console/console-67fd756cb9-f96vt" Sep 17 07:29:29.363574 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:29.363524 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/93b13dbc-04df-4472-890b-9bd51419e06c-oauth-serving-cert\") pod \"console-67fd756cb9-f96vt\" (UID: \"93b13dbc-04df-4472-890b-9bd51419e06c\") " pod="openshift-console/console-67fd756cb9-f96vt" Sep 17 07:29:29.363574 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:29.363556 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/93b13dbc-04df-4472-890b-9bd51419e06c-console-serving-cert\") pod \"console-67fd756cb9-f96vt\" (UID: \"93b13dbc-04df-4472-890b-9bd51419e06c\") " pod="openshift-console/console-67fd756cb9-f96vt" Sep 17 07:29:29.363775 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:29.363582 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/93b13dbc-04df-4472-890b-9bd51419e06c-console-config\") pod \"console-67fd756cb9-f96vt\" (UID: \"93b13dbc-04df-4472-890b-9bd51419e06c\") " pod="openshift-console/console-67fd756cb9-f96vt" Sep 17 07:29:29.363775 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:29.363635 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/93b13dbc-04df-4472-890b-9bd51419e06c-service-ca\") pod \"console-67fd756cb9-f96vt\" (UID: \"93b13dbc-04df-4472-890b-9bd51419e06c\") " pod="openshift-console/console-67fd756cb9-f96vt" Sep 17 07:29:29.464158 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:29.464120 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpqr4\" (UniqueName: \"kubernetes.io/projected/93b13dbc-04df-4472-890b-9bd51419e06c-kube-api-access-zpqr4\") pod \"console-67fd756cb9-f96vt\" (UID: \"93b13dbc-04df-4472-890b-9bd51419e06c\") " pod="openshift-console/console-67fd756cb9-f96vt" Sep 17 07:29:29.464315 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:29.464176 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/93b13dbc-04df-4472-890b-9bd51419e06c-console-oauth-config\") pod \"console-67fd756cb9-f96vt\" (UID: \"93b13dbc-04df-4472-890b-9bd51419e06c\") " pod="openshift-console/console-67fd756cb9-f96vt" Sep 17 07:29:29.464315 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:29.464206 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/93b13dbc-04df-4472-890b-9bd51419e06c-oauth-serving-cert\") pod \"console-67fd756cb9-f96vt\" (UID: \"93b13dbc-04df-4472-890b-9bd51419e06c\") " pod="openshift-console/console-67fd756cb9-f96vt" Sep 17 07:29:29.464315 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:29.464236 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/93b13dbc-04df-4472-890b-9bd51419e06c-console-serving-cert\") pod \"console-67fd756cb9-f96vt\" (UID: \"93b13dbc-04df-4472-890b-9bd51419e06c\") " pod="openshift-console/console-67fd756cb9-f96vt" Sep 17 07:29:29.464315 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:29.464258 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/93b13dbc-04df-4472-890b-9bd51419e06c-console-config\") pod \"console-67fd756cb9-f96vt\" (UID: \"93b13dbc-04df-4472-890b-9bd51419e06c\") " pod="openshift-console/console-67fd756cb9-f96vt" Sep 17 07:29:29.464315 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:29.464282 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/93b13dbc-04df-4472-890b-9bd51419e06c-service-ca\") pod \"console-67fd756cb9-f96vt\" (UID: \"93b13dbc-04df-4472-890b-9bd51419e06c\") " pod="openshift-console/console-67fd756cb9-f96vt" Sep 17 07:29:29.464598 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:29.464325 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/93b13dbc-04df-4472-890b-9bd51419e06c-trusted-ca-bundle\") pod \"console-67fd756cb9-f96vt\" (UID: \"93b13dbc-04df-4472-890b-9bd51419e06c\") " pod="openshift-console/console-67fd756cb9-f96vt" Sep 17 07:29:29.465098 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:29.465075 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/93b13dbc-04df-4472-890b-9bd51419e06c-console-config\") pod \"console-67fd756cb9-f96vt\" (UID: \"93b13dbc-04df-4472-890b-9bd51419e06c\") " pod="openshift-console/console-67fd756cb9-f96vt" Sep 17 07:29:29.465431 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:29.465399 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/93b13dbc-04df-4472-890b-9bd51419e06c-oauth-serving-cert\") pod \"console-67fd756cb9-f96vt\" (UID: \"93b13dbc-04df-4472-890b-9bd51419e06c\") " pod="openshift-console/console-67fd756cb9-f96vt" Sep 17 07:29:29.465960 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:29.465934 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/93b13dbc-04df-4472-890b-9bd51419e06c-service-ca\") pod \"console-67fd756cb9-f96vt\" (UID: \"93b13dbc-04df-4472-890b-9bd51419e06c\") " pod="openshift-console/console-67fd756cb9-f96vt" Sep 17 07:29:29.466097 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:29.466078 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/93b13dbc-04df-4472-890b-9bd51419e06c-trusted-ca-bundle\") pod \"console-67fd756cb9-f96vt\" (UID: \"93b13dbc-04df-4472-890b-9bd51419e06c\") " pod="openshift-console/console-67fd756cb9-f96vt" Sep 17 07:29:29.467183 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:29.467161 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/93b13dbc-04df-4472-890b-9bd51419e06c-console-oauth-config\") pod \"console-67fd756cb9-f96vt\" (UID: \"93b13dbc-04df-4472-890b-9bd51419e06c\") " pod="openshift-console/console-67fd756cb9-f96vt" Sep 17 07:29:29.467348 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:29.467330 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/93b13dbc-04df-4472-890b-9bd51419e06c-console-serving-cert\") pod \"console-67fd756cb9-f96vt\" (UID: \"93b13dbc-04df-4472-890b-9bd51419e06c\") " pod="openshift-console/console-67fd756cb9-f96vt" Sep 17 07:29:29.474622 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:29.474602 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpqr4\" (UniqueName: \"kubernetes.io/projected/93b13dbc-04df-4472-890b-9bd51419e06c-kube-api-access-zpqr4\") pod \"console-67fd756cb9-f96vt\" (UID: \"93b13dbc-04df-4472-890b-9bd51419e06c\") " pod="openshift-console/console-67fd756cb9-f96vt" Sep 17 07:29:29.531089 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:29.531065 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67fd756cb9-f96vt" Sep 17 07:29:29.815659 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:29.815597 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-67fd756cb9-f96vt"] Sep 17 07:29:29.819547 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:29:29.819515 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93b13dbc_04df_4472_890b_9bd51419e06c.slice/crio-54bf30b93c2df1ff6ef72f2178896df6c943ff60cca17a786a1bd691c7808070 WatchSource:0}: Error finding container 54bf30b93c2df1ff6ef72f2178896df6c943ff60cca17a786a1bd691c7808070: Status 404 returned error can't find the container with id 54bf30b93c2df1ff6ef72f2178896df6c943ff60cca17a786a1bd691c7808070 Sep 17 07:29:30.059726 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:30.059685 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67fd756cb9-f96vt" event={"ID":"93b13dbc-04df-4472-890b-9bd51419e06c","Type":"ContainerStarted","Data":"df228ea26e695a55fa7ce1127d71efb0abdd3926436c93edecdf3fce217b60ee"} Sep 17 07:29:30.060212 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:30.059751 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67fd756cb9-f96vt" event={"ID":"93b13dbc-04df-4472-890b-9bd51419e06c","Type":"ContainerStarted","Data":"54bf30b93c2df1ff6ef72f2178896df6c943ff60cca17a786a1bd691c7808070"} Sep 17 07:29:30.090601 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:30.090509 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-67fd756cb9-f96vt" podStartSLOduration=1.090493387 podStartE2EDuration="1.090493387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 07:29:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 07:29:30.08838796 +0000 UTC m=+113.969422320" watchObservedRunningTime="2026-09-17 07:29:30.090493387 +0000 UTC m=+113.971527746" Sep 17 07:29:30.985531 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:30.985493 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-qd64r"] Sep 17 07:29:30.988951 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:30.988929 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-qd64r" Sep 17 07:29:31.013191 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:31.013170 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Sep 17 07:29:31.016245 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:31.016225 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-lft9c\"" Sep 17 07:29:31.017258 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:31.017235 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Sep 17 07:29:31.028551 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:31.028523 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-qd64r"] Sep 17 07:29:31.032774 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:31.032750 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-fczng"] Sep 17 07:29:31.035763 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:31.035742 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-fczng" Sep 17 07:29:31.041691 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:31.041671 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 17 07:29:31.046202 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:31.046181 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-ctkx8\"" Sep 17 07:29:31.046439 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:31.046425 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 17 07:29:31.047677 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:31.047657 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 17 07:29:31.051981 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:31.051961 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-8pn4t"] Sep 17 07:29:31.054838 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:31.054822 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-8pn4t" Sep 17 07:29:31.058510 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:31.058479 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-gdgwn\"" Sep 17 07:29:31.058605 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:31.058483 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Sep 17 07:29:31.058807 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:31.058790 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Sep 17 07:29:31.060789 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:31.060635 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Sep 17 07:29:31.072318 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:31.072295 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-8pn4t"] Sep 17 07:29:31.079373 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:31.079349 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8dadfa8e-9e62-4a67-8aeb-8eb92e2410ba-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-qd64r\" (UID: \"8dadfa8e-9e62-4a67-8aeb-8eb92e2410ba\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-qd64r" Sep 17 07:29:31.079475 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:31.079407 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8dadfa8e-9e62-4a67-8aeb-8eb92e2410ba-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-qd64r\" (UID: \"8dadfa8e-9e62-4a67-8aeb-8eb92e2410ba\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-qd64r" Sep 17 07:29:31.079475 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:31.079432 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8dadfa8e-9e62-4a67-8aeb-8eb92e2410ba-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-qd64r\" (UID: \"8dadfa8e-9e62-4a67-8aeb-8eb92e2410ba\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-qd64r" Sep 17 07:29:31.079568 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:31.079511 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jg5hq\" (UniqueName: \"kubernetes.io/projected/8dadfa8e-9e62-4a67-8aeb-8eb92e2410ba-kube-api-access-jg5hq\") pod \"openshift-state-metrics-67dcf8c979-qd64r\" (UID: \"8dadfa8e-9e62-4a67-8aeb-8eb92e2410ba\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-qd64r" Sep 17 07:29:31.180587 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:31.180151 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8dadfa8e-9e62-4a67-8aeb-8eb92e2410ba-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-qd64r\" (UID: \"8dadfa8e-9e62-4a67-8aeb-8eb92e2410ba\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-qd64r" Sep 17 07:29:31.180587 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:31.180193 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8dadfa8e-9e62-4a67-8aeb-8eb92e2410ba-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-qd64r\" (UID: \"8dadfa8e-9e62-4a67-8aeb-8eb92e2410ba\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-qd64r" Sep 17 07:29:31.180587 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:31.180224 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cc77ee52-e75e-4485-8bfb-40dbfcaf83e9-sys\") pod \"node-exporter-fczng\" (UID: \"cc77ee52-e75e-4485-8bfb-40dbfcaf83e9\") " pod="openshift-monitoring/node-exporter-fczng" Sep 17 07:29:31.180587 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:31.180248 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/b6f2509c-cdcc-4938-a577-f9d7d409c95d-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-8pn4t\" (UID: \"b6f2509c-cdcc-4938-a577-f9d7d409c95d\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-8pn4t" Sep 17 07:29:31.180587 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:31.180302 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cc77ee52-e75e-4485-8bfb-40dbfcaf83e9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fczng\" (UID: \"cc77ee52-e75e-4485-8bfb-40dbfcaf83e9\") " pod="openshift-monitoring/node-exporter-fczng" Sep 17 07:29:31.180587 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:31.180328 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbrqn\" (UniqueName: \"kubernetes.io/projected/cc77ee52-e75e-4485-8bfb-40dbfcaf83e9-kube-api-access-lbrqn\") pod \"node-exporter-fczng\" (UID: \"cc77ee52-e75e-4485-8bfb-40dbfcaf83e9\") " pod="openshift-monitoring/node-exporter-fczng" Sep 17 07:29:31.180587 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:31.180351 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b6f2509c-cdcc-4938-a577-f9d7d409c95d-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-8pn4t\" (UID: \"b6f2509c-cdcc-4938-a577-f9d7d409c95d\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-8pn4t" Sep 17 07:29:31.180587 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:31.180396 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/b6f2509c-cdcc-4938-a577-f9d7d409c95d-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-8pn4t\" (UID: \"b6f2509c-cdcc-4938-a577-f9d7d409c95d\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-8pn4t" Sep 17 07:29:31.180587 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:31.180425 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jg5hq\" (UniqueName: \"kubernetes.io/projected/8dadfa8e-9e62-4a67-8aeb-8eb92e2410ba-kube-api-access-jg5hq\") pod \"openshift-state-metrics-67dcf8c979-qd64r\" (UID: \"8dadfa8e-9e62-4a67-8aeb-8eb92e2410ba\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-qd64r" Sep 17 07:29:31.180587 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:31.180450 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lr9dp\" (UniqueName: \"kubernetes.io/projected/b6f2509c-cdcc-4938-a577-f9d7d409c95d-kube-api-access-lr9dp\") pod \"kube-state-metrics-59fbdd77d-8pn4t\" (UID: \"b6f2509c-cdcc-4938-a577-f9d7d409c95d\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-8pn4t" Sep 17 07:29:31.180587 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:31.180511 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/cc77ee52-e75e-4485-8bfb-40dbfcaf83e9-root\") pod \"node-exporter-fczng\" (UID: \"cc77ee52-e75e-4485-8bfb-40dbfcaf83e9\") " pod="openshift-monitoring/node-exporter-fczng" Sep 17 07:29:31.180587 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:31.180538 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/cc77ee52-e75e-4485-8bfb-40dbfcaf83e9-node-exporter-accelerators-collector-config\") pod \"node-exporter-fczng\" (UID: \"cc77ee52-e75e-4485-8bfb-40dbfcaf83e9\") " pod="openshift-monitoring/node-exporter-fczng" Sep 17 07:29:31.181357 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:31.180799 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b6f2509c-cdcc-4938-a577-f9d7d409c95d-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-8pn4t\" (UID: \"b6f2509c-cdcc-4938-a577-f9d7d409c95d\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-8pn4t" Sep 17 07:29:31.181357 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:31.180915 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cc77ee52-e75e-4485-8bfb-40dbfcaf83e9-metrics-client-ca\") pod \"node-exporter-fczng\" (UID: \"cc77ee52-e75e-4485-8bfb-40dbfcaf83e9\") " pod="openshift-monitoring/node-exporter-fczng" Sep 17 07:29:31.181357 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:31.181332 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b6f2509c-cdcc-4938-a577-f9d7d409c95d-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-8pn4t\" (UID: \"b6f2509c-cdcc-4938-a577-f9d7d409c95d\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-8pn4t" Sep 17 07:29:31.181521 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:31.181422 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/cc77ee52-e75e-4485-8bfb-40dbfcaf83e9-node-exporter-textfile\") pod \"node-exporter-fczng\" (UID: \"cc77ee52-e75e-4485-8bfb-40dbfcaf83e9\") " pod="openshift-monitoring/node-exporter-fczng" Sep 17 07:29:31.181521 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:31.181483 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8dadfa8e-9e62-4a67-8aeb-8eb92e2410ba-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-qd64r\" (UID: \"8dadfa8e-9e62-4a67-8aeb-8eb92e2410ba\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-qd64r" Sep 17 07:29:31.181521 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:31.181515 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/cc77ee52-e75e-4485-8bfb-40dbfcaf83e9-node-exporter-tls\") pod \"node-exporter-fczng\" (UID: \"cc77ee52-e75e-4485-8bfb-40dbfcaf83e9\") " pod="openshift-monitoring/node-exporter-fczng" Sep 17 07:29:31.181657 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:31.181596 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/cc77ee52-e75e-4485-8bfb-40dbfcaf83e9-node-exporter-wtmp\") pod \"node-exporter-fczng\" (UID: \"cc77ee52-e75e-4485-8bfb-40dbfcaf83e9\") " pod="openshift-monitoring/node-exporter-fczng" Sep 17 07:29:31.182172 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:31.182154 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8dadfa8e-9e62-4a67-8aeb-8eb92e2410ba-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-qd64r\" (UID: \"8dadfa8e-9e62-4a67-8aeb-8eb92e2410ba\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-qd64r" Sep 17 07:29:31.183903 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:31.183877 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8dadfa8e-9e62-4a67-8aeb-8eb92e2410ba-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-qd64r\" (UID: \"8dadfa8e-9e62-4a67-8aeb-8eb92e2410ba\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-qd64r" Sep 17 07:29:31.184001 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:31.183939 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8dadfa8e-9e62-4a67-8aeb-8eb92e2410ba-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-qd64r\" (UID: \"8dadfa8e-9e62-4a67-8aeb-8eb92e2410ba\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-qd64r" Sep 17 07:29:31.208234 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:31.208209 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jg5hq\" (UniqueName: \"kubernetes.io/projected/8dadfa8e-9e62-4a67-8aeb-8eb92e2410ba-kube-api-access-jg5hq\") pod \"openshift-state-metrics-67dcf8c979-qd64r\" (UID: \"8dadfa8e-9e62-4a67-8aeb-8eb92e2410ba\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-qd64r" Sep 17 07:29:31.282883 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:31.282792 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/cc77ee52-e75e-4485-8bfb-40dbfcaf83e9-node-exporter-textfile\") pod \"node-exporter-fczng\" (UID: \"cc77ee52-e75e-4485-8bfb-40dbfcaf83e9\") " pod="openshift-monitoring/node-exporter-fczng" Sep 17 07:29:31.282883 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:31.282827 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/cc77ee52-e75e-4485-8bfb-40dbfcaf83e9-node-exporter-tls\") pod \"node-exporter-fczng\" (UID: \"cc77ee52-e75e-4485-8bfb-40dbfcaf83e9\") " pod="openshift-monitoring/node-exporter-fczng" Sep 17 07:29:31.283076 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:31.282888 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/cc77ee52-e75e-4485-8bfb-40dbfcaf83e9-node-exporter-wtmp\") pod \"node-exporter-fczng\" (UID: \"cc77ee52-e75e-4485-8bfb-40dbfcaf83e9\") " pod="openshift-monitoring/node-exporter-fczng" Sep 17 07:29:31.283076 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:31.282919 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cc77ee52-e75e-4485-8bfb-40dbfcaf83e9-sys\") pod \"node-exporter-fczng\" (UID: \"cc77ee52-e75e-4485-8bfb-40dbfcaf83e9\") " pod="openshift-monitoring/node-exporter-fczng" Sep 17 07:29:31.283076 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:31.282976 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cc77ee52-e75e-4485-8bfb-40dbfcaf83e9-sys\") pod \"node-exporter-fczng\" (UID: \"cc77ee52-e75e-4485-8bfb-40dbfcaf83e9\") " pod="openshift-monitoring/node-exporter-fczng" Sep 17 07:29:31.283076 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:31.283037 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/b6f2509c-cdcc-4938-a577-f9d7d409c95d-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-8pn4t\" (UID: \"b6f2509c-cdcc-4938-a577-f9d7d409c95d\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-8pn4t" Sep 17 07:29:31.283076 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:31.283060 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/cc77ee52-e75e-4485-8bfb-40dbfcaf83e9-node-exporter-wtmp\") pod \"node-exporter-fczng\" (UID: \"cc77ee52-e75e-4485-8bfb-40dbfcaf83e9\") " pod="openshift-monitoring/node-exporter-fczng" Sep 17 07:29:31.283325 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:31.283122 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cc77ee52-e75e-4485-8bfb-40dbfcaf83e9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fczng\" (UID: \"cc77ee52-e75e-4485-8bfb-40dbfcaf83e9\") " pod="openshift-monitoring/node-exporter-fczng" Sep 17 07:29:31.283325 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:31.283156 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbrqn\" (UniqueName: \"kubernetes.io/projected/cc77ee52-e75e-4485-8bfb-40dbfcaf83e9-kube-api-access-lbrqn\") pod \"node-exporter-fczng\" (UID: \"cc77ee52-e75e-4485-8bfb-40dbfcaf83e9\") " pod="openshift-monitoring/node-exporter-fczng" Sep 17 07:29:31.283325 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:31.283187 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b6f2509c-cdcc-4938-a577-f9d7d409c95d-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-8pn4t\" (UID: \"b6f2509c-cdcc-4938-a577-f9d7d409c95d\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-8pn4t" Sep 17 07:29:31.283325 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:31.283232 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/b6f2509c-cdcc-4938-a577-f9d7d409c95d-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-8pn4t\" (UID: \"b6f2509c-cdcc-4938-a577-f9d7d409c95d\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-8pn4t" Sep 17 07:29:31.283325 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:31.283268 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lr9dp\" (UniqueName: \"kubernetes.io/projected/b6f2509c-cdcc-4938-a577-f9d7d409c95d-kube-api-access-lr9dp\") pod \"kube-state-metrics-59fbdd77d-8pn4t\" (UID: \"b6f2509c-cdcc-4938-a577-f9d7d409c95d\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-8pn4t" Sep 17 07:29:31.283325 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:31.283297 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/cc77ee52-e75e-4485-8bfb-40dbfcaf83e9-root\") pod \"node-exporter-fczng\" (UID: \"cc77ee52-e75e-4485-8bfb-40dbfcaf83e9\") " pod="openshift-monitoring/node-exporter-fczng" Sep 17 07:29:31.283650 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:31.283324 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/cc77ee52-e75e-4485-8bfb-40dbfcaf83e9-node-exporter-accelerators-collector-config\") pod \"node-exporter-fczng\" (UID: \"cc77ee52-e75e-4485-8bfb-40dbfcaf83e9\") " pod="openshift-monitoring/node-exporter-fczng" Sep 17 07:29:31.283650 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:31.283188 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/cc77ee52-e75e-4485-8bfb-40dbfcaf83e9-node-exporter-textfile\") pod \"node-exporter-fczng\" (UID: \"cc77ee52-e75e-4485-8bfb-40dbfcaf83e9\") " pod="openshift-monitoring/node-exporter-fczng" Sep 17 07:29:31.283650 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:31.283385 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/b6f2509c-cdcc-4938-a577-f9d7d409c95d-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-8pn4t\" (UID: \"b6f2509c-cdcc-4938-a577-f9d7d409c95d\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-8pn4t" Sep 17 07:29:31.283650 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:31.283393 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b6f2509c-cdcc-4938-a577-f9d7d409c95d-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-8pn4t\" (UID: \"b6f2509c-cdcc-4938-a577-f9d7d409c95d\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-8pn4t" Sep 17 07:29:31.283650 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:31.283441 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cc77ee52-e75e-4485-8bfb-40dbfcaf83e9-metrics-client-ca\") pod \"node-exporter-fczng\" (UID: \"cc77ee52-e75e-4485-8bfb-40dbfcaf83e9\") " pod="openshift-monitoring/node-exporter-fczng" Sep 17 07:29:31.283650 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:31.283485 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b6f2509c-cdcc-4938-a577-f9d7d409c95d-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-8pn4t\" (UID: \"b6f2509c-cdcc-4938-a577-f9d7d409c95d\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-8pn4t" Sep 17 07:29:31.283650 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:31.283509 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/cc77ee52-e75e-4485-8bfb-40dbfcaf83e9-root\") pod \"node-exporter-fczng\" (UID: \"cc77ee52-e75e-4485-8bfb-40dbfcaf83e9\") " pod="openshift-monitoring/node-exporter-fczng" Sep 17 07:29:31.284015 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:31.283923 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/cc77ee52-e75e-4485-8bfb-40dbfcaf83e9-node-exporter-accelerators-collector-config\") pod \"node-exporter-fczng\" (UID: \"cc77ee52-e75e-4485-8bfb-40dbfcaf83e9\") " pod="openshift-monitoring/node-exporter-fczng" Sep 17 07:29:31.284068 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:31.284054 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cc77ee52-e75e-4485-8bfb-40dbfcaf83e9-metrics-client-ca\") pod \"node-exporter-fczng\" (UID: \"cc77ee52-e75e-4485-8bfb-40dbfcaf83e9\") " pod="openshift-monitoring/node-exporter-fczng" Sep 17 07:29:31.284422 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:31.284393 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/b6f2509c-cdcc-4938-a577-f9d7d409c95d-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-8pn4t\" (UID: \"b6f2509c-cdcc-4938-a577-f9d7d409c95d\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-8pn4t" Sep 17 07:29:31.284467 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:31.284439 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b6f2509c-cdcc-4938-a577-f9d7d409c95d-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-8pn4t\" (UID: \"b6f2509c-cdcc-4938-a577-f9d7d409c95d\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-8pn4t" Sep 17 07:29:31.285562 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:31.285538 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cc77ee52-e75e-4485-8bfb-40dbfcaf83e9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fczng\" (UID: \"cc77ee52-e75e-4485-8bfb-40dbfcaf83e9\") " pod="openshift-monitoring/node-exporter-fczng" Sep 17 07:29:31.285828 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:31.285795 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/cc77ee52-e75e-4485-8bfb-40dbfcaf83e9-node-exporter-tls\") pod \"node-exporter-fczng\" (UID: \"cc77ee52-e75e-4485-8bfb-40dbfcaf83e9\") " pod="openshift-monitoring/node-exporter-fczng" Sep 17 07:29:31.286168 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:31.286144 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b6f2509c-cdcc-4938-a577-f9d7d409c95d-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-8pn4t\" (UID: \"b6f2509c-cdcc-4938-a577-f9d7d409c95d\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-8pn4t" Sep 17 07:29:31.286579 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:31.286563 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b6f2509c-cdcc-4938-a577-f9d7d409c95d-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-8pn4t\" (UID: \"b6f2509c-cdcc-4938-a577-f9d7d409c95d\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-8pn4t" Sep 17 07:29:31.298824 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:31.298799 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-qd64r" Sep 17 07:29:31.299167 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:31.299149 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbrqn\" (UniqueName: \"kubernetes.io/projected/cc77ee52-e75e-4485-8bfb-40dbfcaf83e9-kube-api-access-lbrqn\") pod \"node-exporter-fczng\" (UID: \"cc77ee52-e75e-4485-8bfb-40dbfcaf83e9\") " pod="openshift-monitoring/node-exporter-fczng" Sep 17 07:29:31.301244 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:31.301224 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lr9dp\" (UniqueName: \"kubernetes.io/projected/b6f2509c-cdcc-4938-a577-f9d7d409c95d-kube-api-access-lr9dp\") pod \"kube-state-metrics-59fbdd77d-8pn4t\" (UID: \"b6f2509c-cdcc-4938-a577-f9d7d409c95d\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-8pn4t" Sep 17 07:29:31.345792 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:31.345763 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-fczng" Sep 17 07:29:31.365586 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:31.365563 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-8pn4t" Sep 17 07:29:32.070591 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:32.070559 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 17 07:29:32.077444 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:32.077416 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 17 07:29:32.081833 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:32.081802 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Sep 17 07:29:32.081953 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:32.081847 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Sep 17 07:29:32.081953 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:32.081889 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Sep 17 07:29:32.082183 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:32.082165 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Sep 17 07:29:32.082264 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:32.082170 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Sep 17 07:29:32.082264 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:32.082170 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Sep 17 07:29:32.082458 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:32.082436 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Sep 17 07:29:32.082569 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:32.082490 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Sep 17 07:29:32.082569 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:32.082510 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-mww9g\"" Sep 17 07:29:32.082789 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:32.082768 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Sep 17 07:29:32.091586 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:32.091566 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 17 07:29:32.191753 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:32.191714 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1ccdd24e-b616-4c0f-b552-17948e1f2f87-config-out\") pod \"alertmanager-main-0\" (UID: \"1ccdd24e-b616-4c0f-b552-17948e1f2f87\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 07:29:32.191949 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:32.191761 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1ccdd24e-b616-4c0f-b552-17948e1f2f87-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1ccdd24e-b616-4c0f-b552-17948e1f2f87\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 07:29:32.191949 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:32.191816 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ccdd24e-b616-4c0f-b552-17948e1f2f87-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1ccdd24e-b616-4c0f-b552-17948e1f2f87\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 07:29:32.191949 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:32.191843 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1ccdd24e-b616-4c0f-b552-17948e1f2f87-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1ccdd24e-b616-4c0f-b552-17948e1f2f87\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 07:29:32.191949 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:32.191908 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1ccdd24e-b616-4c0f-b552-17948e1f2f87-web-config\") pod \"alertmanager-main-0\" (UID: \"1ccdd24e-b616-4c0f-b552-17948e1f2f87\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 07:29:32.191949 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:32.191944 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1ccdd24e-b616-4c0f-b552-17948e1f2f87-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1ccdd24e-b616-4c0f-b552-17948e1f2f87\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 07:29:32.192151 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:32.191979 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnjpw\" (UniqueName: \"kubernetes.io/projected/1ccdd24e-b616-4c0f-b552-17948e1f2f87-kube-api-access-pnjpw\") pod \"alertmanager-main-0\" (UID: \"1ccdd24e-b616-4c0f-b552-17948e1f2f87\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 07:29:32.192151 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:32.192002 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1ccdd24e-b616-4c0f-b552-17948e1f2f87-config-volume\") pod \"alertmanager-main-0\" (UID: \"1ccdd24e-b616-4c0f-b552-17948e1f2f87\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 07:29:32.192151 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:32.192036 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1ccdd24e-b616-4c0f-b552-17948e1f2f87-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1ccdd24e-b616-4c0f-b552-17948e1f2f87\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 07:29:32.192151 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:32.192087 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1ccdd24e-b616-4c0f-b552-17948e1f2f87-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1ccdd24e-b616-4c0f-b552-17948e1f2f87\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 07:29:32.192151 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:32.192115 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1ccdd24e-b616-4c0f-b552-17948e1f2f87-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1ccdd24e-b616-4c0f-b552-17948e1f2f87\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 07:29:32.192151 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:32.192137 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1ccdd24e-b616-4c0f-b552-17948e1f2f87-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1ccdd24e-b616-4c0f-b552-17948e1f2f87\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 07:29:32.192446 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:32.192160 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1ccdd24e-b616-4c0f-b552-17948e1f2f87-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1ccdd24e-b616-4c0f-b552-17948e1f2f87\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 07:29:32.292972 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:32.292940 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnjpw\" (UniqueName: \"kubernetes.io/projected/1ccdd24e-b616-4c0f-b552-17948e1f2f87-kube-api-access-pnjpw\") pod \"alertmanager-main-0\" (UID: \"1ccdd24e-b616-4c0f-b552-17948e1f2f87\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 07:29:32.293156 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:32.292982 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1ccdd24e-b616-4c0f-b552-17948e1f2f87-config-volume\") pod \"alertmanager-main-0\" (UID: \"1ccdd24e-b616-4c0f-b552-17948e1f2f87\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 07:29:32.293156 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:32.293010 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1ccdd24e-b616-4c0f-b552-17948e1f2f87-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1ccdd24e-b616-4c0f-b552-17948e1f2f87\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 07:29:32.293156 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:32.293053 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1ccdd24e-b616-4c0f-b552-17948e1f2f87-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1ccdd24e-b616-4c0f-b552-17948e1f2f87\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 07:29:32.293156 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:32.293080 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1ccdd24e-b616-4c0f-b552-17948e1f2f87-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1ccdd24e-b616-4c0f-b552-17948e1f2f87\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 07:29:32.293156 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:32.293104 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1ccdd24e-b616-4c0f-b552-17948e1f2f87-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1ccdd24e-b616-4c0f-b552-17948e1f2f87\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 07:29:32.293156 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:32.293128 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1ccdd24e-b616-4c0f-b552-17948e1f2f87-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1ccdd24e-b616-4c0f-b552-17948e1f2f87\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 07:29:32.293458 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:32.293163 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1ccdd24e-b616-4c0f-b552-17948e1f2f87-config-out\") pod \"alertmanager-main-0\" (UID: \"1ccdd24e-b616-4c0f-b552-17948e1f2f87\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 07:29:32.293458 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:32.293185 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1ccdd24e-b616-4c0f-b552-17948e1f2f87-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1ccdd24e-b616-4c0f-b552-17948e1f2f87\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 07:29:32.293458 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:32.293232 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ccdd24e-b616-4c0f-b552-17948e1f2f87-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1ccdd24e-b616-4c0f-b552-17948e1f2f87\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 07:29:32.293458 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:32.293256 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1ccdd24e-b616-4c0f-b552-17948e1f2f87-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1ccdd24e-b616-4c0f-b552-17948e1f2f87\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 07:29:32.293458 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:32.293294 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1ccdd24e-b616-4c0f-b552-17948e1f2f87-web-config\") pod \"alertmanager-main-0\" (UID: \"1ccdd24e-b616-4c0f-b552-17948e1f2f87\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 07:29:32.293458 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:32.293335 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1ccdd24e-b616-4c0f-b552-17948e1f2f87-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1ccdd24e-b616-4c0f-b552-17948e1f2f87\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 07:29:32.293458 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:32.293432 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1ccdd24e-b616-4c0f-b552-17948e1f2f87-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1ccdd24e-b616-4c0f-b552-17948e1f2f87\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 07:29:32.294563 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:32.294459 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1ccdd24e-b616-4c0f-b552-17948e1f2f87-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1ccdd24e-b616-4c0f-b552-17948e1f2f87\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 07:29:32.296569 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:32.296452 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1ccdd24e-b616-4c0f-b552-17948e1f2f87-web-config\") pod \"alertmanager-main-0\" (UID: \"1ccdd24e-b616-4c0f-b552-17948e1f2f87\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 07:29:32.297136 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:32.296759 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1ccdd24e-b616-4c0f-b552-17948e1f2f87-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1ccdd24e-b616-4c0f-b552-17948e1f2f87\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 07:29:32.297136 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:32.296992 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1ccdd24e-b616-4c0f-b552-17948e1f2f87-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1ccdd24e-b616-4c0f-b552-17948e1f2f87\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 07:29:32.297136 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:32.297098 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1ccdd24e-b616-4c0f-b552-17948e1f2f87-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1ccdd24e-b616-4c0f-b552-17948e1f2f87\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 07:29:32.297329 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:32.297276 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1ccdd24e-b616-4c0f-b552-17948e1f2f87-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1ccdd24e-b616-4c0f-b552-17948e1f2f87\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 07:29:32.297389 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:32.297334 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ccdd24e-b616-4c0f-b552-17948e1f2f87-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1ccdd24e-b616-4c0f-b552-17948e1f2f87\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 07:29:32.297439 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:32.297385 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1ccdd24e-b616-4c0f-b552-17948e1f2f87-config-volume\") pod \"alertmanager-main-0\" (UID: \"1ccdd24e-b616-4c0f-b552-17948e1f2f87\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 07:29:32.297830 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:32.297765 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1ccdd24e-b616-4c0f-b552-17948e1f2f87-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1ccdd24e-b616-4c0f-b552-17948e1f2f87\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 07:29:32.298409 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:32.298387 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1ccdd24e-b616-4c0f-b552-17948e1f2f87-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1ccdd24e-b616-4c0f-b552-17948e1f2f87\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 07:29:32.299003 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:32.298987 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1ccdd24e-b616-4c0f-b552-17948e1f2f87-config-out\") pod \"alertmanager-main-0\" (UID: \"1ccdd24e-b616-4c0f-b552-17948e1f2f87\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 07:29:32.304463 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:32.304443 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnjpw\" (UniqueName: \"kubernetes.io/projected/1ccdd24e-b616-4c0f-b552-17948e1f2f87-kube-api-access-pnjpw\") pod \"alertmanager-main-0\" (UID: \"1ccdd24e-b616-4c0f-b552-17948e1f2f87\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 07:29:32.390754 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:32.390724 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 17 07:29:34.594391 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:34.594358 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-bcc9d4dd5-bn4zl" Sep 17 07:29:34.594967 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:34.594417 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-bcc9d4dd5-bn4zl" Sep 17 07:29:34.599671 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:34.599648 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-bcc9d4dd5-bn4zl" Sep 17 07:29:35.094098 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:35.094067 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-bcc9d4dd5-bn4zl" Sep 17 07:29:35.625498 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:35.625460 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-665f5f65f8-sdxtt"] Sep 17 07:29:35.629108 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:35.629086 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-665f5f65f8-sdxtt" Sep 17 07:29:35.632296 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:35.632273 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-kc6p8\"" Sep 17 07:29:35.632531 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:35.632512 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Sep 17 07:29:35.633393 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:35.633369 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Sep 17 07:29:35.633393 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:35.633381 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-amif99muom3i0\"" Sep 17 07:29:35.633595 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:35.633393 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Sep 17 07:29:35.633647 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:35.633629 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Sep 17 07:29:35.640074 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:35.640022 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-665f5f65f8-sdxtt"] Sep 17 07:29:35.724725 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:35.724689 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/38179dd9-b912-4fb0-9dd0-f4c57f4c5ac1-metrics-server-audit-profiles\") pod \"metrics-server-665f5f65f8-sdxtt\" (UID: \"38179dd9-b912-4fb0-9dd0-f4c57f4c5ac1\") " pod="openshift-monitoring/metrics-server-665f5f65f8-sdxtt" Sep 17 07:29:35.724913 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:35.724762 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/38179dd9-b912-4fb0-9dd0-f4c57f4c5ac1-audit-log\") pod \"metrics-server-665f5f65f8-sdxtt\" (UID: \"38179dd9-b912-4fb0-9dd0-f4c57f4c5ac1\") " pod="openshift-monitoring/metrics-server-665f5f65f8-sdxtt" Sep 17 07:29:35.724913 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:35.724791 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/38179dd9-b912-4fb0-9dd0-f4c57f4c5ac1-secret-metrics-server-tls\") pod \"metrics-server-665f5f65f8-sdxtt\" (UID: \"38179dd9-b912-4fb0-9dd0-f4c57f4c5ac1\") " pod="openshift-monitoring/metrics-server-665f5f65f8-sdxtt" Sep 17 07:29:35.725028 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:35.724930 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggdhw\" (UniqueName: \"kubernetes.io/projected/38179dd9-b912-4fb0-9dd0-f4c57f4c5ac1-kube-api-access-ggdhw\") pod \"metrics-server-665f5f65f8-sdxtt\" (UID: \"38179dd9-b912-4fb0-9dd0-f4c57f4c5ac1\") " pod="openshift-monitoring/metrics-server-665f5f65f8-sdxtt" Sep 17 07:29:35.725068 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:35.725037 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/38179dd9-b912-4fb0-9dd0-f4c57f4c5ac1-secret-metrics-server-client-certs\") pod \"metrics-server-665f5f65f8-sdxtt\" (UID: \"38179dd9-b912-4fb0-9dd0-f4c57f4c5ac1\") " pod="openshift-monitoring/metrics-server-665f5f65f8-sdxtt" Sep 17 07:29:35.725068 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:35.725062 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/38179dd9-b912-4fb0-9dd0-f4c57f4c5ac1-client-ca-bundle\") pod \"metrics-server-665f5f65f8-sdxtt\" (UID: \"38179dd9-b912-4fb0-9dd0-f4c57f4c5ac1\") " pod="openshift-monitoring/metrics-server-665f5f65f8-sdxtt" Sep 17 07:29:35.725148 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:35.725086 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/38179dd9-b912-4fb0-9dd0-f4c57f4c5ac1-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-665f5f65f8-sdxtt\" (UID: \"38179dd9-b912-4fb0-9dd0-f4c57f4c5ac1\") " pod="openshift-monitoring/metrics-server-665f5f65f8-sdxtt" Sep 17 07:29:35.826507 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:35.826472 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/38179dd9-b912-4fb0-9dd0-f4c57f4c5ac1-metrics-server-audit-profiles\") pod \"metrics-server-665f5f65f8-sdxtt\" (UID: \"38179dd9-b912-4fb0-9dd0-f4c57f4c5ac1\") " pod="openshift-monitoring/metrics-server-665f5f65f8-sdxtt" Sep 17 07:29:35.826681 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:35.826523 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/38179dd9-b912-4fb0-9dd0-f4c57f4c5ac1-audit-log\") pod \"metrics-server-665f5f65f8-sdxtt\" (UID: \"38179dd9-b912-4fb0-9dd0-f4c57f4c5ac1\") " pod="openshift-monitoring/metrics-server-665f5f65f8-sdxtt" Sep 17 07:29:35.826681 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:35.826646 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/38179dd9-b912-4fb0-9dd0-f4c57f4c5ac1-secret-metrics-server-tls\") pod \"metrics-server-665f5f65f8-sdxtt\" (UID: \"38179dd9-b912-4fb0-9dd0-f4c57f4c5ac1\") " pod="openshift-monitoring/metrics-server-665f5f65f8-sdxtt" Sep 17 07:29:35.826876 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:35.826725 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggdhw\" (UniqueName: \"kubernetes.io/projected/38179dd9-b912-4fb0-9dd0-f4c57f4c5ac1-kube-api-access-ggdhw\") pod \"metrics-server-665f5f65f8-sdxtt\" (UID: \"38179dd9-b912-4fb0-9dd0-f4c57f4c5ac1\") " pod="openshift-monitoring/metrics-server-665f5f65f8-sdxtt" Sep 17 07:29:35.826876 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:35.826780 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/38179dd9-b912-4fb0-9dd0-f4c57f4c5ac1-secret-metrics-server-client-certs\") pod \"metrics-server-665f5f65f8-sdxtt\" (UID: \"38179dd9-b912-4fb0-9dd0-f4c57f4c5ac1\") " pod="openshift-monitoring/metrics-server-665f5f65f8-sdxtt" Sep 17 07:29:35.826876 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:35.826807 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/38179dd9-b912-4fb0-9dd0-f4c57f4c5ac1-client-ca-bundle\") pod \"metrics-server-665f5f65f8-sdxtt\" (UID: \"38179dd9-b912-4fb0-9dd0-f4c57f4c5ac1\") " pod="openshift-monitoring/metrics-server-665f5f65f8-sdxtt" Sep 17 07:29:35.826876 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:35.826841 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/38179dd9-b912-4fb0-9dd0-f4c57f4c5ac1-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-665f5f65f8-sdxtt\" (UID: \"38179dd9-b912-4fb0-9dd0-f4c57f4c5ac1\") " pod="openshift-monitoring/metrics-server-665f5f65f8-sdxtt" Sep 17 07:29:35.827081 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:35.826908 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/38179dd9-b912-4fb0-9dd0-f4c57f4c5ac1-audit-log\") pod \"metrics-server-665f5f65f8-sdxtt\" (UID: \"38179dd9-b912-4fb0-9dd0-f4c57f4c5ac1\") " pod="openshift-monitoring/metrics-server-665f5f65f8-sdxtt" Sep 17 07:29:35.827550 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:35.827523 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/38179dd9-b912-4fb0-9dd0-f4c57f4c5ac1-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-665f5f65f8-sdxtt\" (UID: \"38179dd9-b912-4fb0-9dd0-f4c57f4c5ac1\") " pod="openshift-monitoring/metrics-server-665f5f65f8-sdxtt" Sep 17 07:29:35.827685 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:35.827595 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/38179dd9-b912-4fb0-9dd0-f4c57f4c5ac1-metrics-server-audit-profiles\") pod \"metrics-server-665f5f65f8-sdxtt\" (UID: \"38179dd9-b912-4fb0-9dd0-f4c57f4c5ac1\") " pod="openshift-monitoring/metrics-server-665f5f65f8-sdxtt" Sep 17 07:29:35.829449 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:35.829427 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/38179dd9-b912-4fb0-9dd0-f4c57f4c5ac1-client-ca-bundle\") pod \"metrics-server-665f5f65f8-sdxtt\" (UID: \"38179dd9-b912-4fb0-9dd0-f4c57f4c5ac1\") " pod="openshift-monitoring/metrics-server-665f5f65f8-sdxtt" Sep 17 07:29:35.829604 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:35.829583 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/38179dd9-b912-4fb0-9dd0-f4c57f4c5ac1-secret-metrics-server-client-certs\") pod \"metrics-server-665f5f65f8-sdxtt\" (UID: \"38179dd9-b912-4fb0-9dd0-f4c57f4c5ac1\") " pod="openshift-monitoring/metrics-server-665f5f65f8-sdxtt" Sep 17 07:29:35.829878 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:35.829845 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/38179dd9-b912-4fb0-9dd0-f4c57f4c5ac1-secret-metrics-server-tls\") pod \"metrics-server-665f5f65f8-sdxtt\" (UID: \"38179dd9-b912-4fb0-9dd0-f4c57f4c5ac1\") " pod="openshift-monitoring/metrics-server-665f5f65f8-sdxtt" Sep 17 07:29:35.840038 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:35.840017 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggdhw\" (UniqueName: \"kubernetes.io/projected/38179dd9-b912-4fb0-9dd0-f4c57f4c5ac1-kube-api-access-ggdhw\") pod \"metrics-server-665f5f65f8-sdxtt\" (UID: \"38179dd9-b912-4fb0-9dd0-f4c57f4c5ac1\") " pod="openshift-monitoring/metrics-server-665f5f65f8-sdxtt" Sep 17 07:29:35.943760 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:35.943726 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-665f5f65f8-sdxtt" Sep 17 07:29:35.967224 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:35.967200 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-648fcb787c-kxf78" Sep 17 07:29:38.028597 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:29:38.028558 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc77ee52_e75e_4485_8bfb_40dbfcaf83e9.slice/crio-494531c19c165f4a981bfa7106879469b4505604bc3b2aaa63f7d449fd5859a8 WatchSource:0}: Error finding container 494531c19c165f4a981bfa7106879469b4505604bc3b2aaa63f7d449fd5859a8: Status 404 returned error can't find the container with id 494531c19c165f4a981bfa7106879469b4505604bc3b2aaa63f7d449fd5859a8 Sep 17 07:29:38.104228 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:38.104191 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fczng" event={"ID":"cc77ee52-e75e-4485-8bfb-40dbfcaf83e9","Type":"ContainerStarted","Data":"494531c19c165f4a981bfa7106879469b4505604bc3b2aaa63f7d449fd5859a8"} Sep 17 07:29:38.316112 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:38.316082 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-qd64r"] Sep 17 07:29:38.317719 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:29:38.317688 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dadfa8e_9e62_4a67_8aeb_8eb92e2410ba.slice/crio-77cd07e8bf98b9e9e658e8ce2bcb2267b9e1501c88ebbcad13b4480a7268460b WatchSource:0}: Error finding container 77cd07e8bf98b9e9e658e8ce2bcb2267b9e1501c88ebbcad13b4480a7268460b: Status 404 returned error can't find the container with id 77cd07e8bf98b9e9e658e8ce2bcb2267b9e1501c88ebbcad13b4480a7268460b Sep 17 07:29:38.328706 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:38.328679 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 17 07:29:38.333913 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:38.333070 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-8pn4t"] Sep 17 07:29:38.333913 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:29:38.333400 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6f2509c_cdcc_4938_a577_f9d7d409c95d.slice/crio-6c65edbc4216d2f9aac5773d3b28c7b955e606bc27627edd7e80350f26313315 WatchSource:0}: Error finding container 6c65edbc4216d2f9aac5773d3b28c7b955e606bc27627edd7e80350f26313315: Status 404 returned error can't find the container with id 6c65edbc4216d2f9aac5773d3b28c7b955e606bc27627edd7e80350f26313315 Sep 17 07:29:38.334690 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:38.334668 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-665f5f65f8-sdxtt"] Sep 17 07:29:38.338619 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:29:38.338575 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38179dd9_b912_4fb0_9dd0_f4c57f4c5ac1.slice/crio-f2b6cecfd77da00b567948c81f86928f368df13c52ec48a480b4322ec8894842 WatchSource:0}: Error finding container f2b6cecfd77da00b567948c81f86928f368df13c52ec48a480b4322ec8894842: Status 404 returned error can't find the container with id f2b6cecfd77da00b567948c81f86928f368df13c52ec48a480b4322ec8894842 Sep 17 07:29:39.111888 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:39.111526 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-z7dwl" event={"ID":"ef341fec-b3aa-45f8-9697-70bbf084fc7f","Type":"ContainerStarted","Data":"571c1fd78f444493c4758dd7a780f51f6a5681e540a4a921117caff0a347ba06"} Sep 17 07:29:39.115256 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:39.115157 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-76f99584f4-z7dwl" Sep 17 07:29:39.116926 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:39.116818 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-665f5f65f8-sdxtt" event={"ID":"38179dd9-b912-4fb0-9dd0-f4c57f4c5ac1","Type":"ContainerStarted","Data":"f2b6cecfd77da00b567948c81f86928f368df13c52ec48a480b4322ec8894842"} Sep 17 07:29:39.121240 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:39.121109 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-qd64r" event={"ID":"8dadfa8e-9e62-4a67-8aeb-8eb92e2410ba","Type":"ContainerStarted","Data":"9a5eb9a4fe9dbbf299ab7a20e4d0d2448f47a87fcc8eae1fa8956c820e90a48b"} Sep 17 07:29:39.121240 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:39.121157 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-qd64r" event={"ID":"8dadfa8e-9e62-4a67-8aeb-8eb92e2410ba","Type":"ContainerStarted","Data":"e42b3e18722c74dbcc3b22970a407596c17a9920814a588b912b0f63cb407c5d"} Sep 17 07:29:39.121240 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:39.121187 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-qd64r" event={"ID":"8dadfa8e-9e62-4a67-8aeb-8eb92e2410ba","Type":"ContainerStarted","Data":"77cd07e8bf98b9e9e658e8ce2bcb2267b9e1501c88ebbcad13b4480a7268460b"} Sep 17 07:29:39.122673 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:39.122647 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1ccdd24e-b616-4c0f-b552-17948e1f2f87","Type":"ContainerStarted","Data":"bca27e4dbf57239d7c5e2ac73e4f324d45d51d7c6341a41b13a9d6256e15ac99"} Sep 17 07:29:39.123094 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:39.123068 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-76f99584f4-z7dwl" Sep 17 07:29:39.126155 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:39.126127 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-8pn4t" event={"ID":"b6f2509c-cdcc-4938-a577-f9d7d409c95d","Type":"ContainerStarted","Data":"6c65edbc4216d2f9aac5773d3b28c7b955e606bc27627edd7e80350f26313315"} Sep 17 07:29:39.171520 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:39.171449 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-76f99584f4-z7dwl" podStartSLOduration=2.512314839 podStartE2EDuration="19.171428572s" totalImagesPullingTime="16.659113733s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 07:29:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 07:29:39.141833549 +0000 UTC m=+123.022867887" watchObservedRunningTime="2026-09-17 07:29:39.171428572 +0000 UTC m=+123.052462933" Sep 17 07:29:39.409506 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:39.409462 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-67fd756cb9-f96vt"] Sep 17 07:29:39.531640 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:39.531549 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-67fd756cb9-f96vt" Sep 17 07:29:43.145721 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:43.145682 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-665f5f65f8-sdxtt" event={"ID":"38179dd9-b912-4fb0-9dd0-f4c57f4c5ac1","Type":"ContainerStarted","Data":"2f04855eca2bd9f817f0508fb4217b2bdb55d5fe81d74046ab80ca452d76221d"} Sep 17 07:29:43.148428 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:43.148370 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-qd64r" event={"ID":"8dadfa8e-9e62-4a67-8aeb-8eb92e2410ba","Type":"ContainerStarted","Data":"2c5e91f1413dfa6e14cdab2437cd44f8fa39bf681ed27d61617b6ef09a4080b9"} Sep 17 07:29:43.150271 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:43.150241 2578 generic.go:356] "Generic (PLEG): container finished" podID="1ccdd24e-b616-4c0f-b552-17948e1f2f87" containerID="3d7910f4feb1a92e822e68afbf3c03f530d2efd53814295584b8d8442564f47c" exitCode=0 Sep 17 07:29:43.150387 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:43.150326 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1ccdd24e-b616-4c0f-b552-17948e1f2f87","Type":"ContainerDied","Data":"3d7910f4feb1a92e822e68afbf3c03f530d2efd53814295584b8d8442564f47c"} Sep 17 07:29:43.152903 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:43.152868 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-8pn4t" event={"ID":"b6f2509c-cdcc-4938-a577-f9d7d409c95d","Type":"ContainerStarted","Data":"b5d0629bdb15f5596a252248e59575bcebccefb2648cafdcca0ad07dd9e9083e"} Sep 17 07:29:43.153012 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:43.152939 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-8pn4t" event={"ID":"b6f2509c-cdcc-4938-a577-f9d7d409c95d","Type":"ContainerStarted","Data":"6117dcb8f01c75a4fa6c831b5e3cbca0e9f5baa84b7e196404d3cf835502991e"} Sep 17 07:29:43.153012 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:43.152970 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-8pn4t" event={"ID":"b6f2509c-cdcc-4938-a577-f9d7d409c95d","Type":"ContainerStarted","Data":"f05cb1583ead2d2c64a3f317de753a1a00ea6ecaa4c59cb545652761365a165a"} Sep 17 07:29:43.154832 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:43.154808 2578 generic.go:356] "Generic (PLEG): container finished" podID="cc77ee52-e75e-4485-8bfb-40dbfcaf83e9" containerID="1e8d57b8176f78027084dc210b1e5663630914f27bde2392ecea7ad4ba6ec530" exitCode=0 Sep 17 07:29:43.154964 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:43.154883 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fczng" event={"ID":"cc77ee52-e75e-4485-8bfb-40dbfcaf83e9","Type":"ContainerDied","Data":"1e8d57b8176f78027084dc210b1e5663630914f27bde2392ecea7ad4ba6ec530"} Sep 17 07:29:43.168230 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:43.168184 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-665f5f65f8-sdxtt" podStartSLOduration=4.414467366 podStartE2EDuration="8.168167896s" totalImagesPullingTime="3.75370053s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 07:29:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 07:29:43.166511178 +0000 UTC m=+127.047545537" watchObservedRunningTime="2026-09-17 07:29:43.168167896 +0000 UTC m=+127.049202329" Sep 17 07:29:43.231553 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:43.231495 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-qd64r" podStartSLOduration=9.599894922 podStartE2EDuration="13.231475456s" totalImagesPullingTime="3.631580534s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 07:29:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 07:29:43.223349253 +0000 UTC m=+127.104383818" watchObservedRunningTime="2026-09-17 07:29:43.231475456 +0000 UTC m=+127.112509819" Sep 17 07:29:43.256812 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:43.256765 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-8pn4t" podStartSLOduration=8.501082956 podStartE2EDuration="12.256749664s" totalImagesPullingTime="3.755666708s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 07:29:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 07:29:43.252742563 +0000 UTC m=+127.133776922" watchObservedRunningTime="2026-09-17 07:29:43.256749664 +0000 UTC m=+127.137784023" Sep 17 07:29:44.164978 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:44.164775 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fczng" event={"ID":"cc77ee52-e75e-4485-8bfb-40dbfcaf83e9","Type":"ContainerStarted","Data":"df69f553f5e93327767984b7a93caf368eedac51252bb734fce4452c1dadb92d"} Sep 17 07:29:44.164978 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:44.164897 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fczng" event={"ID":"cc77ee52-e75e-4485-8bfb-40dbfcaf83e9","Type":"ContainerStarted","Data":"b90abaecc279b4b6f6c398a8ae2f2a0c23f22cea84f968be59ff0eac7620d87f"} Sep 17 07:29:44.190220 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:44.189630 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-fczng" podStartSLOduration=10.084104074 podStartE2EDuration="14.189609378s" totalImagesPullingTime="3.971047129s" totalInitContainerRuntime="134.458175ms" isStatefulPod=true podCreationTimestamp="2026-09-17 07:29:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 07:29:44.18899582 +0000 UTC m=+128.070030180" watchObservedRunningTime="2026-09-17 07:29:44.189609378 +0000 UTC m=+128.070643737" Sep 17 07:29:44.379995 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:44.379955 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-648fcb787c-kxf78"] Sep 17 07:29:46.176372 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:46.176331 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1ccdd24e-b616-4c0f-b552-17948e1f2f87","Type":"ContainerStarted","Data":"56d851184361d5da953d9dcf721653cb3982db6298c4a10ab7b216e47d1617e2"} Sep 17 07:29:46.176816 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:46.176450 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1ccdd24e-b616-4c0f-b552-17948e1f2f87","Type":"ContainerStarted","Data":"acd62b14dd106b0edd2b87eaed16042831d98ad81527b8601d099641ca13e4ef"} Sep 17 07:29:46.176816 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:46.176491 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1ccdd24e-b616-4c0f-b552-17948e1f2f87","Type":"ContainerStarted","Data":"a31b96d80065b73ed5005a7a277cfca8c7ff4908a4d30cc2585744fbbd9d537b"} Sep 17 07:29:46.176816 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:46.176518 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1ccdd24e-b616-4c0f-b552-17948e1f2f87","Type":"ContainerStarted","Data":"10ae6ed023efe8a28596bcaa71542a640e110c377b4999cd17b94d7c42771374"} Sep 17 07:29:46.176816 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:46.176549 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1ccdd24e-b616-4c0f-b552-17948e1f2f87","Type":"ContainerStarted","Data":"c609d57988d766825a6844048d3ff25f1e1c5b90a68656473f45b91583cbce39"} Sep 17 07:29:46.339587 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:46.339551 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/737e0243-b8eb-47e4-bf0f-c6b31e6ba787-metrics-certs\") pod \"network-metrics-daemon-lnpjx\" (UID: \"737e0243-b8eb-47e4-bf0f-c6b31e6ba787\") " pod="openshift-multus/network-metrics-daemon-lnpjx" Sep 17 07:29:46.342129 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:46.342105 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/737e0243-b8eb-47e4-bf0f-c6b31e6ba787-metrics-certs\") pod \"network-metrics-daemon-lnpjx\" (UID: \"737e0243-b8eb-47e4-bf0f-c6b31e6ba787\") " pod="openshift-multus/network-metrics-daemon-lnpjx" Sep 17 07:29:46.438609 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:46.438542 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-rjccm\"" Sep 17 07:29:46.446247 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:46.446221 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lnpjx" Sep 17 07:29:46.760125 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:46.760047 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-lnpjx"] Sep 17 07:29:46.902216 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:29:46.902185 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod737e0243_b8eb_47e4_bf0f_c6b31e6ba787.slice/crio-f6a15e5593101f7a34ddd020c23e8021a099cf612849b87a6fcff91afe165eb5 WatchSource:0}: Error finding container f6a15e5593101f7a34ddd020c23e8021a099cf612849b87a6fcff91afe165eb5: Status 404 returned error can't find the container with id f6a15e5593101f7a34ddd020c23e8021a099cf612849b87a6fcff91afe165eb5 Sep 17 07:29:47.181413 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:47.181380 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lnpjx" event={"ID":"737e0243-b8eb-47e4-bf0f-c6b31e6ba787","Type":"ContainerStarted","Data":"f6a15e5593101f7a34ddd020c23e8021a099cf612849b87a6fcff91afe165eb5"} Sep 17 07:29:48.188933 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:48.188900 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1ccdd24e-b616-4c0f-b552-17948e1f2f87","Type":"ContainerStarted","Data":"e586e8ecfd1167d50d2df4c81847b083201da0ab9426a84e48c03891e8a5d8d6"} Sep 17 07:29:48.222611 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:48.222455 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=8.980022362 podStartE2EDuration="16.222405186s" totalImagesPullingTime="7.220255074s" totalInitContainerRuntime="22.12775ms" isStatefulPod=false podCreationTimestamp="2026-09-17 07:29:32 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 07:29:48.219556023 +0000 UTC m=+132.100590394" watchObservedRunningTime="2026-09-17 07:29:48.222405186 +0000 UTC m=+132.103439545" Sep 17 07:29:49.193956 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:49.193920 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lnpjx" event={"ID":"737e0243-b8eb-47e4-bf0f-c6b31e6ba787","Type":"ContainerStarted","Data":"9b53d1914623ef1a995314696759d9cb3f9f7c200ce50dc67a1144e54e0673b8"} Sep 17 07:29:49.194322 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:49.193976 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lnpjx" event={"ID":"737e0243-b8eb-47e4-bf0f-c6b31e6ba787","Type":"ContainerStarted","Data":"e26a06f21902cec7a0ce5331c2e31695a1ee38981f6c419d7de3e9ae68bdc449"} Sep 17 07:29:49.226933 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:49.224979 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-lnpjx" podStartSLOduration=131.870975186 podStartE2EDuration="2m13.224959068s" totalImagesPullingTime="1.353983882s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 07:27:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 07:29:49.221899488 +0000 UTC m=+133.102933841" watchObservedRunningTime="2026-09-17 07:29:49.224959068 +0000 UTC m=+133.105993432" Sep 17 07:29:50.472052 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:50.472015 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-bcc9d4dd5-bn4zl"] Sep 17 07:29:55.943969 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:55.943931 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-665f5f65f8-sdxtt" Sep 17 07:29:55.944419 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:29:55.944020 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-665f5f65f8-sdxtt" Sep 17 07:30:04.439356 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:04.439308 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-67fd756cb9-f96vt" podUID="93b13dbc-04df-4472-890b-9bd51419e06c" containerName="console" containerID="cri-o://df228ea26e695a55fa7ce1127d71efb0abdd3926436c93edecdf3fce217b60ee" gracePeriod=15 Sep 17 07:30:04.672745 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:04.672720 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67fd756cb9-f96vt" Sep 17 07:30:04.784010 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:04.783941 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/93b13dbc-04df-4472-890b-9bd51419e06c-console-serving-cert\") pod \"93b13dbc-04df-4472-890b-9bd51419e06c\" (UID: \"93b13dbc-04df-4472-890b-9bd51419e06c\") " Sep 17 07:30:04.784010 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:04.783977 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/93b13dbc-04df-4472-890b-9bd51419e06c-console-oauth-config\") pod \"93b13dbc-04df-4472-890b-9bd51419e06c\" (UID: \"93b13dbc-04df-4472-890b-9bd51419e06c\") " Sep 17 07:30:04.784010 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:04.783998 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/93b13dbc-04df-4472-890b-9bd51419e06c-console-config\") pod \"93b13dbc-04df-4472-890b-9bd51419e06c\" (UID: \"93b13dbc-04df-4472-890b-9bd51419e06c\") " Sep 17 07:30:04.784256 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:04.784053 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/93b13dbc-04df-4472-890b-9bd51419e06c-trusted-ca-bundle\") pod \"93b13dbc-04df-4472-890b-9bd51419e06c\" (UID: \"93b13dbc-04df-4472-890b-9bd51419e06c\") " Sep 17 07:30:04.784256 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:04.784085 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zpqr4\" (UniqueName: \"kubernetes.io/projected/93b13dbc-04df-4472-890b-9bd51419e06c-kube-api-access-zpqr4\") pod \"93b13dbc-04df-4472-890b-9bd51419e06c\" (UID: \"93b13dbc-04df-4472-890b-9bd51419e06c\") " Sep 17 07:30:04.784256 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:04.784125 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/93b13dbc-04df-4472-890b-9bd51419e06c-service-ca\") pod \"93b13dbc-04df-4472-890b-9bd51419e06c\" (UID: \"93b13dbc-04df-4472-890b-9bd51419e06c\") " Sep 17 07:30:04.784256 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:04.784166 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/93b13dbc-04df-4472-890b-9bd51419e06c-oauth-serving-cert\") pod \"93b13dbc-04df-4472-890b-9bd51419e06c\" (UID: \"93b13dbc-04df-4472-890b-9bd51419e06c\") " Sep 17 07:30:04.784532 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:04.784501 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93b13dbc-04df-4472-890b-9bd51419e06c-console-config" (OuterVolumeSpecName: "console-config") pod "93b13dbc-04df-4472-890b-9bd51419e06c" (UID: "93b13dbc-04df-4472-890b-9bd51419e06c"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 07:30:04.784602 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:04.784524 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93b13dbc-04df-4472-890b-9bd51419e06c-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "93b13dbc-04df-4472-890b-9bd51419e06c" (UID: "93b13dbc-04df-4472-890b-9bd51419e06c"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 07:30:04.784602 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:04.784544 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93b13dbc-04df-4472-890b-9bd51419e06c-service-ca" (OuterVolumeSpecName: "service-ca") pod "93b13dbc-04df-4472-890b-9bd51419e06c" (UID: "93b13dbc-04df-4472-890b-9bd51419e06c"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 07:30:04.784761 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:04.784729 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93b13dbc-04df-4472-890b-9bd51419e06c-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "93b13dbc-04df-4472-890b-9bd51419e06c" (UID: "93b13dbc-04df-4472-890b-9bd51419e06c"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 07:30:04.786173 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:04.786148 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93b13dbc-04df-4472-890b-9bd51419e06c-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "93b13dbc-04df-4472-890b-9bd51419e06c" (UID: "93b13dbc-04df-4472-890b-9bd51419e06c"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 07:30:04.786278 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:04.786235 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93b13dbc-04df-4472-890b-9bd51419e06c-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "93b13dbc-04df-4472-890b-9bd51419e06c" (UID: "93b13dbc-04df-4472-890b-9bd51419e06c"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 07:30:04.786278 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:04.786239 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93b13dbc-04df-4472-890b-9bd51419e06c-kube-api-access-zpqr4" (OuterVolumeSpecName: "kube-api-access-zpqr4") pod "93b13dbc-04df-4472-890b-9bd51419e06c" (UID: "93b13dbc-04df-4472-890b-9bd51419e06c"). InnerVolumeSpecName "kube-api-access-zpqr4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 07:30:04.885117 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:04.885078 2578 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/93b13dbc-04df-4472-890b-9bd51419e06c-console-serving-cert\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Sep 17 07:30:04.885117 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:04.885108 2578 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/93b13dbc-04df-4472-890b-9bd51419e06c-console-oauth-config\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Sep 17 07:30:04.885117 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:04.885117 2578 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/93b13dbc-04df-4472-890b-9bd51419e06c-console-config\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Sep 17 07:30:04.885117 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:04.885127 2578 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/93b13dbc-04df-4472-890b-9bd51419e06c-trusted-ca-bundle\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Sep 17 07:30:04.885366 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:04.885137 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpqr4\" (UniqueName: \"kubernetes.io/projected/93b13dbc-04df-4472-890b-9bd51419e06c-kube-api-access-zpqr4\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Sep 17 07:30:04.885366 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:04.885146 2578 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/93b13dbc-04df-4472-890b-9bd51419e06c-service-ca\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Sep 17 07:30:04.885366 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:04.885154 2578 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/93b13dbc-04df-4472-890b-9bd51419e06c-oauth-serving-cert\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Sep 17 07:30:05.260535 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:05.260504 2578 generic.go:356] "Generic (PLEG): container finished" podID="93b13dbc-04df-4472-890b-9bd51419e06c" containerID="df228ea26e695a55fa7ce1127d71efb0abdd3926436c93edecdf3fce217b60ee" exitCode=0 Sep 17 07:30:05.260742 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:05.260558 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67fd756cb9-f96vt" Sep 17 07:30:05.260742 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:05.260596 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67fd756cb9-f96vt" event={"ID":"93b13dbc-04df-4472-890b-9bd51419e06c","Type":"ContainerDied","Data":"df228ea26e695a55fa7ce1127d71efb0abdd3926436c93edecdf3fce217b60ee"} Sep 17 07:30:05.260742 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:05.260638 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67fd756cb9-f96vt" event={"ID":"93b13dbc-04df-4472-890b-9bd51419e06c","Type":"ContainerDied","Data":"54bf30b93c2df1ff6ef72f2178896df6c943ff60cca17a786a1bd691c7808070"} Sep 17 07:30:05.260742 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:05.260660 2578 scope.go:117] "RemoveContainer" containerID="df228ea26e695a55fa7ce1127d71efb0abdd3926436c93edecdf3fce217b60ee" Sep 17 07:30:05.271999 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:05.271974 2578 scope.go:117] "RemoveContainer" containerID="df228ea26e695a55fa7ce1127d71efb0abdd3926436c93edecdf3fce217b60ee" Sep 17 07:30:05.272285 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:30:05.272261 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df228ea26e695a55fa7ce1127d71efb0abdd3926436c93edecdf3fce217b60ee\": container with ID starting with df228ea26e695a55fa7ce1127d71efb0abdd3926436c93edecdf3fce217b60ee not found: ID does not exist" containerID="df228ea26e695a55fa7ce1127d71efb0abdd3926436c93edecdf3fce217b60ee" Sep 17 07:30:05.272368 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:05.272289 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df228ea26e695a55fa7ce1127d71efb0abdd3926436c93edecdf3fce217b60ee"} err="failed to get container status \"df228ea26e695a55fa7ce1127d71efb0abdd3926436c93edecdf3fce217b60ee\": rpc error: code = NotFound desc = could not find container \"df228ea26e695a55fa7ce1127d71efb0abdd3926436c93edecdf3fce217b60ee\": container with ID starting with df228ea26e695a55fa7ce1127d71efb0abdd3926436c93edecdf3fce217b60ee not found: ID does not exist" Sep 17 07:30:05.286103 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:05.286077 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-67fd756cb9-f96vt"] Sep 17 07:30:05.290276 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:05.290252 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-67fd756cb9-f96vt"] Sep 17 07:30:06.629852 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:06.629806 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="93b13dbc-04df-4472-890b-9bd51419e06c" path="/var/lib/kubelet/pods/93b13dbc-04df-4472-890b-9bd51419e06c/volumes" Sep 17 07:30:09.410750 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:09.410694 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-648fcb787c-kxf78" podUID="44eefc80-34c2-47a2-ab48-e7e685f7dfe3" containerName="registry" containerID="cri-o://f784bfd05b8bc20da17f6d7f822a39780f27a30bbc6044f12c87ef9acb50c943" gracePeriod=30 Sep 17 07:30:09.683346 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:09.683284 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-648fcb787c-kxf78" Sep 17 07:30:09.821123 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:09.821092 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/44eefc80-34c2-47a2-ab48-e7e685f7dfe3-installation-pull-secrets\") pod \"44eefc80-34c2-47a2-ab48-e7e685f7dfe3\" (UID: \"44eefc80-34c2-47a2-ab48-e7e685f7dfe3\") " Sep 17 07:30:09.821123 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:09.821127 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/44eefc80-34c2-47a2-ab48-e7e685f7dfe3-registry-tls\") pod \"44eefc80-34c2-47a2-ab48-e7e685f7dfe3\" (UID: \"44eefc80-34c2-47a2-ab48-e7e685f7dfe3\") " Sep 17 07:30:09.821336 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:09.821153 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/44eefc80-34c2-47a2-ab48-e7e685f7dfe3-trusted-ca\") pod \"44eefc80-34c2-47a2-ab48-e7e685f7dfe3\" (UID: \"44eefc80-34c2-47a2-ab48-e7e685f7dfe3\") " Sep 17 07:30:09.821336 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:09.821169 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dfwmj\" (UniqueName: \"kubernetes.io/projected/44eefc80-34c2-47a2-ab48-e7e685f7dfe3-kube-api-access-dfwmj\") pod \"44eefc80-34c2-47a2-ab48-e7e685f7dfe3\" (UID: \"44eefc80-34c2-47a2-ab48-e7e685f7dfe3\") " Sep 17 07:30:09.821336 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:09.821189 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/44eefc80-34c2-47a2-ab48-e7e685f7dfe3-registry-certificates\") pod \"44eefc80-34c2-47a2-ab48-e7e685f7dfe3\" (UID: \"44eefc80-34c2-47a2-ab48-e7e685f7dfe3\") " Sep 17 07:30:09.821336 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:09.821209 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/44eefc80-34c2-47a2-ab48-e7e685f7dfe3-ca-trust-extracted\") pod \"44eefc80-34c2-47a2-ab48-e7e685f7dfe3\" (UID: \"44eefc80-34c2-47a2-ab48-e7e685f7dfe3\") " Sep 17 07:30:09.821336 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:09.821246 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/44eefc80-34c2-47a2-ab48-e7e685f7dfe3-image-registry-private-configuration\") pod \"44eefc80-34c2-47a2-ab48-e7e685f7dfe3\" (UID: \"44eefc80-34c2-47a2-ab48-e7e685f7dfe3\") " Sep 17 07:30:09.821336 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:09.821323 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/44eefc80-34c2-47a2-ab48-e7e685f7dfe3-bound-sa-token\") pod \"44eefc80-34c2-47a2-ab48-e7e685f7dfe3\" (UID: \"44eefc80-34c2-47a2-ab48-e7e685f7dfe3\") " Sep 17 07:30:09.821787 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:09.821670 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44eefc80-34c2-47a2-ab48-e7e685f7dfe3-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "44eefc80-34c2-47a2-ab48-e7e685f7dfe3" (UID: "44eefc80-34c2-47a2-ab48-e7e685f7dfe3"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 07:30:09.822039 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:09.822010 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44eefc80-34c2-47a2-ab48-e7e685f7dfe3-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "44eefc80-34c2-47a2-ab48-e7e685f7dfe3" (UID: "44eefc80-34c2-47a2-ab48-e7e685f7dfe3"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 07:30:09.823668 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:09.823640 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44eefc80-34c2-47a2-ab48-e7e685f7dfe3-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "44eefc80-34c2-47a2-ab48-e7e685f7dfe3" (UID: "44eefc80-34c2-47a2-ab48-e7e685f7dfe3"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 07:30:09.823668 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:09.823652 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44eefc80-34c2-47a2-ab48-e7e685f7dfe3-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "44eefc80-34c2-47a2-ab48-e7e685f7dfe3" (UID: "44eefc80-34c2-47a2-ab48-e7e685f7dfe3"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 07:30:09.823905 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:09.823880 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44eefc80-34c2-47a2-ab48-e7e685f7dfe3-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "44eefc80-34c2-47a2-ab48-e7e685f7dfe3" (UID: "44eefc80-34c2-47a2-ab48-e7e685f7dfe3"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 07:30:09.823970 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:09.823918 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44eefc80-34c2-47a2-ab48-e7e685f7dfe3-kube-api-access-dfwmj" (OuterVolumeSpecName: "kube-api-access-dfwmj") pod "44eefc80-34c2-47a2-ab48-e7e685f7dfe3" (UID: "44eefc80-34c2-47a2-ab48-e7e685f7dfe3"). InnerVolumeSpecName "kube-api-access-dfwmj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 07:30:09.824332 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:09.824314 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44eefc80-34c2-47a2-ab48-e7e685f7dfe3-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "44eefc80-34c2-47a2-ab48-e7e685f7dfe3" (UID: "44eefc80-34c2-47a2-ab48-e7e685f7dfe3"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 07:30:09.829724 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:09.829696 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44eefc80-34c2-47a2-ab48-e7e685f7dfe3-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "44eefc80-34c2-47a2-ab48-e7e685f7dfe3" (UID: "44eefc80-34c2-47a2-ab48-e7e685f7dfe3"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 07:30:09.922753 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:09.922724 2578 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/44eefc80-34c2-47a2-ab48-e7e685f7dfe3-trusted-ca\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Sep 17 07:30:09.922753 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:09.922750 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfwmj\" (UniqueName: \"kubernetes.io/projected/44eefc80-34c2-47a2-ab48-e7e685f7dfe3-kube-api-access-dfwmj\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Sep 17 07:30:09.922753 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:09.922761 2578 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/44eefc80-34c2-47a2-ab48-e7e685f7dfe3-registry-certificates\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Sep 17 07:30:09.922974 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:09.922769 2578 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/44eefc80-34c2-47a2-ab48-e7e685f7dfe3-ca-trust-extracted\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Sep 17 07:30:09.922974 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:09.922779 2578 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/44eefc80-34c2-47a2-ab48-e7e685f7dfe3-image-registry-private-configuration\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Sep 17 07:30:09.922974 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:09.922788 2578 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/44eefc80-34c2-47a2-ab48-e7e685f7dfe3-bound-sa-token\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Sep 17 07:30:09.922974 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:09.922796 2578 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/44eefc80-34c2-47a2-ab48-e7e685f7dfe3-installation-pull-secrets\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Sep 17 07:30:09.922974 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:09.922804 2578 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/44eefc80-34c2-47a2-ab48-e7e685f7dfe3-registry-tls\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Sep 17 07:30:10.282479 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:10.282448 2578 generic.go:356] "Generic (PLEG): container finished" podID="44eefc80-34c2-47a2-ab48-e7e685f7dfe3" containerID="f784bfd05b8bc20da17f6d7f822a39780f27a30bbc6044f12c87ef9acb50c943" exitCode=0 Sep 17 07:30:10.282702 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:10.282488 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-648fcb787c-kxf78" Sep 17 07:30:10.282702 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:10.282535 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-648fcb787c-kxf78" event={"ID":"44eefc80-34c2-47a2-ab48-e7e685f7dfe3","Type":"ContainerDied","Data":"f784bfd05b8bc20da17f6d7f822a39780f27a30bbc6044f12c87ef9acb50c943"} Sep 17 07:30:10.282702 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:10.282609 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-648fcb787c-kxf78" event={"ID":"44eefc80-34c2-47a2-ab48-e7e685f7dfe3","Type":"ContainerDied","Data":"4f7ac77aa2de930155129abaa5c1d298cea21c932f882d04586750b3052538c7"} Sep 17 07:30:10.282702 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:10.282633 2578 scope.go:117] "RemoveContainer" containerID="f784bfd05b8bc20da17f6d7f822a39780f27a30bbc6044f12c87ef9acb50c943" Sep 17 07:30:10.303799 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:10.303784 2578 scope.go:117] "RemoveContainer" containerID="f784bfd05b8bc20da17f6d7f822a39780f27a30bbc6044f12c87ef9acb50c943" Sep 17 07:30:10.304453 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:30:10.304392 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f784bfd05b8bc20da17f6d7f822a39780f27a30bbc6044f12c87ef9acb50c943\": container with ID starting with f784bfd05b8bc20da17f6d7f822a39780f27a30bbc6044f12c87ef9acb50c943 not found: ID does not exist" containerID="f784bfd05b8bc20da17f6d7f822a39780f27a30bbc6044f12c87ef9acb50c943" Sep 17 07:30:10.304453 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:10.304430 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f784bfd05b8bc20da17f6d7f822a39780f27a30bbc6044f12c87ef9acb50c943"} err="failed to get container status \"f784bfd05b8bc20da17f6d7f822a39780f27a30bbc6044f12c87ef9acb50c943\": rpc error: code = NotFound desc = could not find container \"f784bfd05b8bc20da17f6d7f822a39780f27a30bbc6044f12c87ef9acb50c943\": container with ID starting with f784bfd05b8bc20da17f6d7f822a39780f27a30bbc6044f12c87ef9acb50c943 not found: ID does not exist" Sep 17 07:30:10.305523 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:10.305503 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-648fcb787c-kxf78"] Sep 17 07:30:10.309644 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:10.309624 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-648fcb787c-kxf78"] Sep 17 07:30:10.628418 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:10.628343 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44eefc80-34c2-47a2-ab48-e7e685f7dfe3" path="/var/lib/kubelet/pods/44eefc80-34c2-47a2-ab48-e7e685f7dfe3/volumes" Sep 17 07:30:15.491341 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:15.491297 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-bcc9d4dd5-bn4zl" podUID="7e500620-3439-4f1e-834c-7134a5582b50" containerName="console" containerID="cri-o://2da01d9877fd029ce1e90d543d45061080f96ea12697586a2e0b4eb79f9c1abd" gracePeriod=15 Sep 17 07:30:15.755617 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:15.755562 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bcc9d4dd5-bn4zl" Sep 17 07:30:15.870445 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:15.870417 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7e500620-3439-4f1e-834c-7134a5582b50-service-ca\") pod \"7e500620-3439-4f1e-834c-7134a5582b50\" (UID: \"7e500620-3439-4f1e-834c-7134a5582b50\") " Sep 17 07:30:15.870614 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:15.870460 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ddqxw\" (UniqueName: \"kubernetes.io/projected/7e500620-3439-4f1e-834c-7134a5582b50-kube-api-access-ddqxw\") pod \"7e500620-3439-4f1e-834c-7134a5582b50\" (UID: \"7e500620-3439-4f1e-834c-7134a5582b50\") " Sep 17 07:30:15.870614 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:15.870491 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7e500620-3439-4f1e-834c-7134a5582b50-console-config\") pod \"7e500620-3439-4f1e-834c-7134a5582b50\" (UID: \"7e500620-3439-4f1e-834c-7134a5582b50\") " Sep 17 07:30:15.870614 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:15.870519 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7e500620-3439-4f1e-834c-7134a5582b50-console-serving-cert\") pod \"7e500620-3439-4f1e-834c-7134a5582b50\" (UID: \"7e500620-3439-4f1e-834c-7134a5582b50\") " Sep 17 07:30:15.870614 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:15.870541 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7e500620-3439-4f1e-834c-7134a5582b50-oauth-serving-cert\") pod \"7e500620-3439-4f1e-834c-7134a5582b50\" (UID: \"7e500620-3439-4f1e-834c-7134a5582b50\") " Sep 17 07:30:15.870614 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:15.870560 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7e500620-3439-4f1e-834c-7134a5582b50-console-oauth-config\") pod \"7e500620-3439-4f1e-834c-7134a5582b50\" (UID: \"7e500620-3439-4f1e-834c-7134a5582b50\") " Sep 17 07:30:15.870940 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:15.870914 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e500620-3439-4f1e-834c-7134a5582b50-console-config" (OuterVolumeSpecName: "console-config") pod "7e500620-3439-4f1e-834c-7134a5582b50" (UID: "7e500620-3439-4f1e-834c-7134a5582b50"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 07:30:15.870994 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:15.870916 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e500620-3439-4f1e-834c-7134a5582b50-service-ca" (OuterVolumeSpecName: "service-ca") pod "7e500620-3439-4f1e-834c-7134a5582b50" (UID: "7e500620-3439-4f1e-834c-7134a5582b50"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 07:30:15.870994 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:15.870956 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e500620-3439-4f1e-834c-7134a5582b50-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "7e500620-3439-4f1e-834c-7134a5582b50" (UID: "7e500620-3439-4f1e-834c-7134a5582b50"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 07:30:15.872596 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:15.872568 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e500620-3439-4f1e-834c-7134a5582b50-kube-api-access-ddqxw" (OuterVolumeSpecName: "kube-api-access-ddqxw") pod "7e500620-3439-4f1e-834c-7134a5582b50" (UID: "7e500620-3439-4f1e-834c-7134a5582b50"). InnerVolumeSpecName "kube-api-access-ddqxw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 07:30:15.872693 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:15.872600 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e500620-3439-4f1e-834c-7134a5582b50-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "7e500620-3439-4f1e-834c-7134a5582b50" (UID: "7e500620-3439-4f1e-834c-7134a5582b50"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 07:30:15.872693 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:15.872655 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e500620-3439-4f1e-834c-7134a5582b50-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "7e500620-3439-4f1e-834c-7134a5582b50" (UID: "7e500620-3439-4f1e-834c-7134a5582b50"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 07:30:15.950045 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:15.950024 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-665f5f65f8-sdxtt" Sep 17 07:30:15.953803 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:15.953786 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-665f5f65f8-sdxtt" Sep 17 07:30:15.971116 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:15.971098 2578 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7e500620-3439-4f1e-834c-7134a5582b50-console-config\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Sep 17 07:30:15.971116 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:15.971116 2578 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7e500620-3439-4f1e-834c-7134a5582b50-console-serving-cert\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Sep 17 07:30:15.971234 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:15.971125 2578 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7e500620-3439-4f1e-834c-7134a5582b50-oauth-serving-cert\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Sep 17 07:30:15.971234 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:15.971135 2578 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7e500620-3439-4f1e-834c-7134a5582b50-console-oauth-config\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Sep 17 07:30:15.971234 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:15.971144 2578 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7e500620-3439-4f1e-834c-7134a5582b50-service-ca\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Sep 17 07:30:15.971234 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:15.971152 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddqxw\" (UniqueName: \"kubernetes.io/projected/7e500620-3439-4f1e-834c-7134a5582b50-kube-api-access-ddqxw\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Sep 17 07:30:16.307750 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:16.307720 2578 generic.go:356] "Generic (PLEG): container finished" podID="7e500620-3439-4f1e-834c-7134a5582b50" containerID="2da01d9877fd029ce1e90d543d45061080f96ea12697586a2e0b4eb79f9c1abd" exitCode=0 Sep 17 07:30:16.308007 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:16.307767 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bcc9d4dd5-bn4zl" Sep 17 07:30:16.308007 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:16.307761 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bcc9d4dd5-bn4zl" event={"ID":"7e500620-3439-4f1e-834c-7134a5582b50","Type":"ContainerDied","Data":"2da01d9877fd029ce1e90d543d45061080f96ea12697586a2e0b4eb79f9c1abd"} Sep 17 07:30:16.308007 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:16.307917 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bcc9d4dd5-bn4zl" event={"ID":"7e500620-3439-4f1e-834c-7134a5582b50","Type":"ContainerDied","Data":"4e0d39009d2788f29e2c33de24637ce3a18937a50a6daddadbbe2c27a17157d6"} Sep 17 07:30:16.308007 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:16.307943 2578 scope.go:117] "RemoveContainer" containerID="2da01d9877fd029ce1e90d543d45061080f96ea12697586a2e0b4eb79f9c1abd" Sep 17 07:30:16.316296 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:16.316276 2578 scope.go:117] "RemoveContainer" containerID="2da01d9877fd029ce1e90d543d45061080f96ea12697586a2e0b4eb79f9c1abd" Sep 17 07:30:16.316560 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:30:16.316543 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2da01d9877fd029ce1e90d543d45061080f96ea12697586a2e0b4eb79f9c1abd\": container with ID starting with 2da01d9877fd029ce1e90d543d45061080f96ea12697586a2e0b4eb79f9c1abd not found: ID does not exist" containerID="2da01d9877fd029ce1e90d543d45061080f96ea12697586a2e0b4eb79f9c1abd" Sep 17 07:30:16.316606 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:16.316567 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2da01d9877fd029ce1e90d543d45061080f96ea12697586a2e0b4eb79f9c1abd"} err="failed to get container status \"2da01d9877fd029ce1e90d543d45061080f96ea12697586a2e0b4eb79f9c1abd\": rpc error: code = NotFound desc = could not find container \"2da01d9877fd029ce1e90d543d45061080f96ea12697586a2e0b4eb79f9c1abd\": container with ID starting with 2da01d9877fd029ce1e90d543d45061080f96ea12697586a2e0b4eb79f9c1abd not found: ID does not exist" Sep 17 07:30:16.329509 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:16.329485 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-bcc9d4dd5-bn4zl"] Sep 17 07:30:16.333844 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:16.333823 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-bcc9d4dd5-bn4zl"] Sep 17 07:30:16.631645 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:16.631560 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e500620-3439-4f1e-834c-7134a5582b50" path="/var/lib/kubelet/pods/7e500620-3439-4f1e-834c-7134a5582b50/volumes" Sep 17 07:30:51.298619 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:51.298576 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 17 07:30:51.299207 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:51.298951 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1ccdd24e-b616-4c0f-b552-17948e1f2f87" containerName="alertmanager" containerID="cri-o://c609d57988d766825a6844048d3ff25f1e1c5b90a68656473f45b91583cbce39" gracePeriod=120 Sep 17 07:30:51.299334 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:51.299309 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1ccdd24e-b616-4c0f-b552-17948e1f2f87" containerName="prom-label-proxy" containerID="cri-o://e586e8ecfd1167d50d2df4c81847b083201da0ab9426a84e48c03891e8a5d8d6" gracePeriod=120 Sep 17 07:30:51.299433 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:51.299415 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1ccdd24e-b616-4c0f-b552-17948e1f2f87" containerName="kube-rbac-proxy-metric" containerID="cri-o://56d851184361d5da953d9dcf721653cb3982db6298c4a10ab7b216e47d1617e2" gracePeriod=120 Sep 17 07:30:51.299511 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:51.299495 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1ccdd24e-b616-4c0f-b552-17948e1f2f87" containerName="kube-rbac-proxy" containerID="cri-o://acd62b14dd106b0edd2b87eaed16042831d98ad81527b8601d099641ca13e4ef" gracePeriod=120 Sep 17 07:30:51.299585 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:51.299565 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1ccdd24e-b616-4c0f-b552-17948e1f2f87" containerName="kube-rbac-proxy-web" containerID="cri-o://a31b96d80065b73ed5005a7a277cfca8c7ff4908a4d30cc2585744fbbd9d537b" gracePeriod=120 Sep 17 07:30:51.299654 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:51.299639 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1ccdd24e-b616-4c0f-b552-17948e1f2f87" containerName="config-reloader" containerID="cri-o://10ae6ed023efe8a28596bcaa71542a640e110c377b4999cd17b94d7c42771374" gracePeriod=120 Sep 17 07:30:51.454259 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:51.454233 2578 generic.go:356] "Generic (PLEG): container finished" podID="1ccdd24e-b616-4c0f-b552-17948e1f2f87" containerID="e586e8ecfd1167d50d2df4c81847b083201da0ab9426a84e48c03891e8a5d8d6" exitCode=0 Sep 17 07:30:51.454259 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:51.454255 2578 generic.go:356] "Generic (PLEG): container finished" podID="1ccdd24e-b616-4c0f-b552-17948e1f2f87" containerID="56d851184361d5da953d9dcf721653cb3982db6298c4a10ab7b216e47d1617e2" exitCode=0 Sep 17 07:30:51.454259 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:51.454262 2578 generic.go:356] "Generic (PLEG): container finished" podID="1ccdd24e-b616-4c0f-b552-17948e1f2f87" containerID="acd62b14dd106b0edd2b87eaed16042831d98ad81527b8601d099641ca13e4ef" exitCode=0 Sep 17 07:30:51.454438 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:51.454269 2578 generic.go:356] "Generic (PLEG): container finished" podID="1ccdd24e-b616-4c0f-b552-17948e1f2f87" containerID="10ae6ed023efe8a28596bcaa71542a640e110c377b4999cd17b94d7c42771374" exitCode=0 Sep 17 07:30:51.454438 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:51.454276 2578 generic.go:356] "Generic (PLEG): container finished" podID="1ccdd24e-b616-4c0f-b552-17948e1f2f87" containerID="c609d57988d766825a6844048d3ff25f1e1c5b90a68656473f45b91583cbce39" exitCode=0 Sep 17 07:30:51.454438 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:51.454308 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1ccdd24e-b616-4c0f-b552-17948e1f2f87","Type":"ContainerDied","Data":"e586e8ecfd1167d50d2df4c81847b083201da0ab9426a84e48c03891e8a5d8d6"} Sep 17 07:30:51.454438 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:51.454366 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1ccdd24e-b616-4c0f-b552-17948e1f2f87","Type":"ContainerDied","Data":"56d851184361d5da953d9dcf721653cb3982db6298c4a10ab7b216e47d1617e2"} Sep 17 07:30:51.454438 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:51.454393 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1ccdd24e-b616-4c0f-b552-17948e1f2f87","Type":"ContainerDied","Data":"acd62b14dd106b0edd2b87eaed16042831d98ad81527b8601d099641ca13e4ef"} Sep 17 07:30:51.454438 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:51.454416 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1ccdd24e-b616-4c0f-b552-17948e1f2f87","Type":"ContainerDied","Data":"10ae6ed023efe8a28596bcaa71542a640e110c377b4999cd17b94d7c42771374"} Sep 17 07:30:51.454438 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:51.454439 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1ccdd24e-b616-4c0f-b552-17948e1f2f87","Type":"ContainerDied","Data":"c609d57988d766825a6844048d3ff25f1e1c5b90a68656473f45b91583cbce39"} Sep 17 07:30:52.462553 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:52.462524 2578 generic.go:356] "Generic (PLEG): container finished" podID="1ccdd24e-b616-4c0f-b552-17948e1f2f87" containerID="a31b96d80065b73ed5005a7a277cfca8c7ff4908a4d30cc2585744fbbd9d537b" exitCode=0 Sep 17 07:30:52.462903 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:52.462585 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1ccdd24e-b616-4c0f-b552-17948e1f2f87","Type":"ContainerDied","Data":"a31b96d80065b73ed5005a7a277cfca8c7ff4908a4d30cc2585744fbbd9d537b"} Sep 17 07:30:52.548582 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:52.548558 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 17 07:30:52.640789 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:52.640765 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pnjpw\" (UniqueName: \"kubernetes.io/projected/1ccdd24e-b616-4c0f-b552-17948e1f2f87-kube-api-access-pnjpw\") pod \"1ccdd24e-b616-4c0f-b552-17948e1f2f87\" (UID: \"1ccdd24e-b616-4c0f-b552-17948e1f2f87\") " Sep 17 07:30:52.640962 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:52.640811 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1ccdd24e-b616-4c0f-b552-17948e1f2f87-secret-alertmanager-kube-rbac-proxy-web\") pod \"1ccdd24e-b616-4c0f-b552-17948e1f2f87\" (UID: \"1ccdd24e-b616-4c0f-b552-17948e1f2f87\") " Sep 17 07:30:52.640962 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:52.640833 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1ccdd24e-b616-4c0f-b552-17948e1f2f87-metrics-client-ca\") pod \"1ccdd24e-b616-4c0f-b552-17948e1f2f87\" (UID: \"1ccdd24e-b616-4c0f-b552-17948e1f2f87\") " Sep 17 07:30:52.640962 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:52.640852 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ccdd24e-b616-4c0f-b552-17948e1f2f87-alertmanager-trusted-ca-bundle\") pod \"1ccdd24e-b616-4c0f-b552-17948e1f2f87\" (UID: \"1ccdd24e-b616-4c0f-b552-17948e1f2f87\") " Sep 17 07:30:52.640962 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:52.640885 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1ccdd24e-b616-4c0f-b552-17948e1f2f87-secret-alertmanager-kube-rbac-proxy-metric\") pod \"1ccdd24e-b616-4c0f-b552-17948e1f2f87\" (UID: \"1ccdd24e-b616-4c0f-b552-17948e1f2f87\") " Sep 17 07:30:52.640962 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:52.640917 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1ccdd24e-b616-4c0f-b552-17948e1f2f87-alertmanager-main-db\") pod \"1ccdd24e-b616-4c0f-b552-17948e1f2f87\" (UID: \"1ccdd24e-b616-4c0f-b552-17948e1f2f87\") " Sep 17 07:30:52.640962 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:52.640956 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1ccdd24e-b616-4c0f-b552-17948e1f2f87-config-out\") pod \"1ccdd24e-b616-4c0f-b552-17948e1f2f87\" (UID: \"1ccdd24e-b616-4c0f-b552-17948e1f2f87\") " Sep 17 07:30:52.641262 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:52.641116 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1ccdd24e-b616-4c0f-b552-17948e1f2f87-secret-alertmanager-main-tls\") pod \"1ccdd24e-b616-4c0f-b552-17948e1f2f87\" (UID: \"1ccdd24e-b616-4c0f-b552-17948e1f2f87\") " Sep 17 07:30:52.641262 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:52.641171 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1ccdd24e-b616-4c0f-b552-17948e1f2f87-cluster-tls-config\") pod \"1ccdd24e-b616-4c0f-b552-17948e1f2f87\" (UID: \"1ccdd24e-b616-4c0f-b552-17948e1f2f87\") " Sep 17 07:30:52.641379 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:52.641308 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ccdd24e-b616-4c0f-b552-17948e1f2f87-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "1ccdd24e-b616-4c0f-b552-17948e1f2f87" (UID: "1ccdd24e-b616-4c0f-b552-17948e1f2f87"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 07:30:52.641379 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:52.641356 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1ccdd24e-b616-4c0f-b552-17948e1f2f87-config-volume\") pod \"1ccdd24e-b616-4c0f-b552-17948e1f2f87\" (UID: \"1ccdd24e-b616-4c0f-b552-17948e1f2f87\") " Sep 17 07:30:52.641472 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:52.641430 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1ccdd24e-b616-4c0f-b552-17948e1f2f87-tls-assets\") pod \"1ccdd24e-b616-4c0f-b552-17948e1f2f87\" (UID: \"1ccdd24e-b616-4c0f-b552-17948e1f2f87\") " Sep 17 07:30:52.641524 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:52.641486 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1ccdd24e-b616-4c0f-b552-17948e1f2f87-web-config\") pod \"1ccdd24e-b616-4c0f-b552-17948e1f2f87\" (UID: \"1ccdd24e-b616-4c0f-b552-17948e1f2f87\") " Sep 17 07:30:52.641524 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:52.641503 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ccdd24e-b616-4c0f-b552-17948e1f2f87-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "1ccdd24e-b616-4c0f-b552-17948e1f2f87" (UID: "1ccdd24e-b616-4c0f-b552-17948e1f2f87"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 07:30:52.641524 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:52.641510 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1ccdd24e-b616-4c0f-b552-17948e1f2f87-secret-alertmanager-kube-rbac-proxy\") pod \"1ccdd24e-b616-4c0f-b552-17948e1f2f87\" (UID: \"1ccdd24e-b616-4c0f-b552-17948e1f2f87\") " Sep 17 07:30:52.641751 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:52.641729 2578 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1ccdd24e-b616-4c0f-b552-17948e1f2f87-metrics-client-ca\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Sep 17 07:30:52.641993 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:52.641977 2578 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1ccdd24e-b616-4c0f-b552-17948e1f2f87-alertmanager-main-db\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Sep 17 07:30:52.642079 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:52.642052 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ccdd24e-b616-4c0f-b552-17948e1f2f87-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "1ccdd24e-b616-4c0f-b552-17948e1f2f87" (UID: "1ccdd24e-b616-4c0f-b552-17948e1f2f87"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 07:30:52.643884 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:52.643604 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ccdd24e-b616-4c0f-b552-17948e1f2f87-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "1ccdd24e-b616-4c0f-b552-17948e1f2f87" (UID: "1ccdd24e-b616-4c0f-b552-17948e1f2f87"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 07:30:52.643884 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:52.643699 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ccdd24e-b616-4c0f-b552-17948e1f2f87-kube-api-access-pnjpw" (OuterVolumeSpecName: "kube-api-access-pnjpw") pod "1ccdd24e-b616-4c0f-b552-17948e1f2f87" (UID: "1ccdd24e-b616-4c0f-b552-17948e1f2f87"). InnerVolumeSpecName "kube-api-access-pnjpw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 07:30:52.644022 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:52.643913 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ccdd24e-b616-4c0f-b552-17948e1f2f87-config-out" (OuterVolumeSpecName: "config-out") pod "1ccdd24e-b616-4c0f-b552-17948e1f2f87" (UID: "1ccdd24e-b616-4c0f-b552-17948e1f2f87"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 07:30:52.644228 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:52.644194 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ccdd24e-b616-4c0f-b552-17948e1f2f87-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "1ccdd24e-b616-4c0f-b552-17948e1f2f87" (UID: "1ccdd24e-b616-4c0f-b552-17948e1f2f87"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 07:30:52.644228 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:52.644210 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ccdd24e-b616-4c0f-b552-17948e1f2f87-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "1ccdd24e-b616-4c0f-b552-17948e1f2f87" (UID: "1ccdd24e-b616-4c0f-b552-17948e1f2f87"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 07:30:52.645076 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:52.645050 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ccdd24e-b616-4c0f-b552-17948e1f2f87-config-volume" (OuterVolumeSpecName: "config-volume") pod "1ccdd24e-b616-4c0f-b552-17948e1f2f87" (UID: "1ccdd24e-b616-4c0f-b552-17948e1f2f87"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 07:30:52.645613 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:52.645593 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ccdd24e-b616-4c0f-b552-17948e1f2f87-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "1ccdd24e-b616-4c0f-b552-17948e1f2f87" (UID: "1ccdd24e-b616-4c0f-b552-17948e1f2f87"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 07:30:52.646043 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:52.646024 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ccdd24e-b616-4c0f-b552-17948e1f2f87-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "1ccdd24e-b616-4c0f-b552-17948e1f2f87" (UID: "1ccdd24e-b616-4c0f-b552-17948e1f2f87"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 07:30:52.650173 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:52.650143 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ccdd24e-b616-4c0f-b552-17948e1f2f87-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "1ccdd24e-b616-4c0f-b552-17948e1f2f87" (UID: "1ccdd24e-b616-4c0f-b552-17948e1f2f87"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 07:30:52.655271 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:52.655252 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ccdd24e-b616-4c0f-b552-17948e1f2f87-web-config" (OuterVolumeSpecName: "web-config") pod "1ccdd24e-b616-4c0f-b552-17948e1f2f87" (UID: "1ccdd24e-b616-4c0f-b552-17948e1f2f87"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 07:30:52.742666 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:52.742644 2578 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1ccdd24e-b616-4c0f-b552-17948e1f2f87-config-out\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Sep 17 07:30:52.742666 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:52.742667 2578 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1ccdd24e-b616-4c0f-b552-17948e1f2f87-secret-alertmanager-main-tls\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Sep 17 07:30:52.742799 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:52.742677 2578 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1ccdd24e-b616-4c0f-b552-17948e1f2f87-cluster-tls-config\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Sep 17 07:30:52.742799 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:52.742685 2578 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1ccdd24e-b616-4c0f-b552-17948e1f2f87-config-volume\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Sep 17 07:30:52.742799 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:52.742693 2578 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1ccdd24e-b616-4c0f-b552-17948e1f2f87-tls-assets\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Sep 17 07:30:52.742799 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:52.742701 2578 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1ccdd24e-b616-4c0f-b552-17948e1f2f87-web-config\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Sep 17 07:30:52.742799 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:52.742709 2578 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1ccdd24e-b616-4c0f-b552-17948e1f2f87-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Sep 17 07:30:52.742799 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:52.742718 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnjpw\" (UniqueName: \"kubernetes.io/projected/1ccdd24e-b616-4c0f-b552-17948e1f2f87-kube-api-access-pnjpw\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Sep 17 07:30:52.742799 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:52.742726 2578 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1ccdd24e-b616-4c0f-b552-17948e1f2f87-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Sep 17 07:30:52.742799 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:52.742734 2578 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ccdd24e-b616-4c0f-b552-17948e1f2f87-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Sep 17 07:30:52.742799 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:52.742742 2578 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1ccdd24e-b616-4c0f-b552-17948e1f2f87-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Sep 17 07:30:53.469037 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:53.469006 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1ccdd24e-b616-4c0f-b552-17948e1f2f87","Type":"ContainerDied","Data":"bca27e4dbf57239d7c5e2ac73e4f324d45d51d7c6341a41b13a9d6256e15ac99"} Sep 17 07:30:53.469037 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:53.469034 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 17 07:30:53.469402 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:53.469084 2578 scope.go:117] "RemoveContainer" containerID="e586e8ecfd1167d50d2df4c81847b083201da0ab9426a84e48c03891e8a5d8d6" Sep 17 07:30:53.476981 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:53.476963 2578 scope.go:117] "RemoveContainer" containerID="56d851184361d5da953d9dcf721653cb3982db6298c4a10ab7b216e47d1617e2" Sep 17 07:30:53.483642 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:53.483623 2578 scope.go:117] "RemoveContainer" containerID="acd62b14dd106b0edd2b87eaed16042831d98ad81527b8601d099641ca13e4ef" Sep 17 07:30:53.489997 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:53.489980 2578 scope.go:117] "RemoveContainer" containerID="a31b96d80065b73ed5005a7a277cfca8c7ff4908a4d30cc2585744fbbd9d537b" Sep 17 07:30:53.496117 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:53.496101 2578 scope.go:117] "RemoveContainer" containerID="10ae6ed023efe8a28596bcaa71542a640e110c377b4999cd17b94d7c42771374" Sep 17 07:30:53.499237 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:53.499216 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 17 07:30:53.503146 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:53.503112 2578 scope.go:117] "RemoveContainer" containerID="c609d57988d766825a6844048d3ff25f1e1c5b90a68656473f45b91583cbce39" Sep 17 07:30:53.505276 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:53.505258 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 17 07:30:53.509981 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:53.509964 2578 scope.go:117] "RemoveContainer" containerID="3d7910f4feb1a92e822e68afbf3c03f530d2efd53814295584b8d8442564f47c" Sep 17 07:30:53.533413 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:53.533389 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 17 07:30:53.533660 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:53.533648 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ccdd24e-b616-4c0f-b552-17948e1f2f87" containerName="alertmanager" Sep 17 07:30:53.533703 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:53.533662 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ccdd24e-b616-4c0f-b552-17948e1f2f87" containerName="alertmanager" Sep 17 07:30:53.533703 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:53.533672 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ccdd24e-b616-4c0f-b552-17948e1f2f87" containerName="prom-label-proxy" Sep 17 07:30:53.533703 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:53.533678 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ccdd24e-b616-4c0f-b552-17948e1f2f87" containerName="prom-label-proxy" Sep 17 07:30:53.533790 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:53.533712 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ccdd24e-b616-4c0f-b552-17948e1f2f87" containerName="config-reloader" Sep 17 07:30:53.533790 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:53.533718 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ccdd24e-b616-4c0f-b552-17948e1f2f87" containerName="config-reloader" Sep 17 07:30:53.533790 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:53.533724 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ccdd24e-b616-4c0f-b552-17948e1f2f87" containerName="kube-rbac-proxy-web" Sep 17 07:30:53.533790 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:53.533729 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ccdd24e-b616-4c0f-b552-17948e1f2f87" containerName="kube-rbac-proxy-web" Sep 17 07:30:53.533790 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:53.533737 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93b13dbc-04df-4472-890b-9bd51419e06c" containerName="console" Sep 17 07:30:53.533790 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:53.533742 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="93b13dbc-04df-4472-890b-9bd51419e06c" containerName="console" Sep 17 07:30:53.533790 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:53.533750 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ccdd24e-b616-4c0f-b552-17948e1f2f87" containerName="kube-rbac-proxy-metric" Sep 17 07:30:53.533790 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:53.533755 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ccdd24e-b616-4c0f-b552-17948e1f2f87" containerName="kube-rbac-proxy-metric" Sep 17 07:30:53.533790 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:53.533763 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ccdd24e-b616-4c0f-b552-17948e1f2f87" containerName="init-config-reloader" Sep 17 07:30:53.533790 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:53.533771 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ccdd24e-b616-4c0f-b552-17948e1f2f87" containerName="init-config-reloader" Sep 17 07:30:53.533790 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:53.533785 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e500620-3439-4f1e-834c-7134a5582b50" containerName="console" Sep 17 07:30:53.533790 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:53.533790 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e500620-3439-4f1e-834c-7134a5582b50" containerName="console" Sep 17 07:30:53.534135 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:53.533799 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ccdd24e-b616-4c0f-b552-17948e1f2f87" containerName="kube-rbac-proxy" Sep 17 07:30:53.534135 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:53.533804 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ccdd24e-b616-4c0f-b552-17948e1f2f87" containerName="kube-rbac-proxy" Sep 17 07:30:53.534135 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:53.533810 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44eefc80-34c2-47a2-ab48-e7e685f7dfe3" containerName="registry" Sep 17 07:30:53.534135 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:53.533815 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="44eefc80-34c2-47a2-ab48-e7e685f7dfe3" containerName="registry" Sep 17 07:30:53.534135 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:53.533875 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="93b13dbc-04df-4472-890b-9bd51419e06c" containerName="console" Sep 17 07:30:53.534135 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:53.533883 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ccdd24e-b616-4c0f-b552-17948e1f2f87" containerName="kube-rbac-proxy-web" Sep 17 07:30:53.534135 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:53.533890 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ccdd24e-b616-4c0f-b552-17948e1f2f87" containerName="kube-rbac-proxy" Sep 17 07:30:53.534135 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:53.533896 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ccdd24e-b616-4c0f-b552-17948e1f2f87" containerName="config-reloader" Sep 17 07:30:53.534135 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:53.533902 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ccdd24e-b616-4c0f-b552-17948e1f2f87" containerName="kube-rbac-proxy-metric" Sep 17 07:30:53.534135 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:53.533915 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e500620-3439-4f1e-834c-7134a5582b50" containerName="console" Sep 17 07:30:53.534135 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:53.533922 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ccdd24e-b616-4c0f-b552-17948e1f2f87" containerName="alertmanager" Sep 17 07:30:53.534135 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:53.533932 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ccdd24e-b616-4c0f-b552-17948e1f2f87" containerName="prom-label-proxy" Sep 17 07:30:53.534135 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:53.533941 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="44eefc80-34c2-47a2-ab48-e7e685f7dfe3" containerName="registry" Sep 17 07:30:53.539651 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:53.539635 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 17 07:30:53.543089 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:53.543069 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Sep 17 07:30:53.543197 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:53.543134 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Sep 17 07:30:53.543197 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:53.543146 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Sep 17 07:30:53.543305 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:53.543195 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Sep 17 07:30:53.543496 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:53.543479 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Sep 17 07:30:53.543703 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:53.543674 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Sep 17 07:30:53.544057 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:53.544041 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Sep 17 07:30:53.544645 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:53.544627 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-mww9g\"" Sep 17 07:30:53.545047 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:53.545032 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Sep 17 07:30:53.548791 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:53.548771 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/55b5d3f9-e42c-40be-a9a4-3cc3c81ad7e2-config-out\") pod \"alertmanager-main-0\" (UID: \"55b5d3f9-e42c-40be-a9a4-3cc3c81ad7e2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 07:30:53.548898 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:53.548798 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/55b5d3f9-e42c-40be-a9a4-3cc3c81ad7e2-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"55b5d3f9-e42c-40be-a9a4-3cc3c81ad7e2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 07:30:53.548898 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:53.548817 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/55b5d3f9-e42c-40be-a9a4-3cc3c81ad7e2-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"55b5d3f9-e42c-40be-a9a4-3cc3c81ad7e2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 07:30:53.548898 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:53.548839 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/55b5d3f9-e42c-40be-a9a4-3cc3c81ad7e2-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"55b5d3f9-e42c-40be-a9a4-3cc3c81ad7e2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 07:30:53.548898 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:53.548874 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/55b5d3f9-e42c-40be-a9a4-3cc3c81ad7e2-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"55b5d3f9-e42c-40be-a9a4-3cc3c81ad7e2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 07:30:53.548898 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:53.548897 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/55b5d3f9-e42c-40be-a9a4-3cc3c81ad7e2-tls-assets\") pod \"alertmanager-main-0\" (UID: \"55b5d3f9-e42c-40be-a9a4-3cc3c81ad7e2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 07:30:53.549129 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:53.548911 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/55b5d3f9-e42c-40be-a9a4-3cc3c81ad7e2-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"55b5d3f9-e42c-40be-a9a4-3cc3c81ad7e2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 07:30:53.549129 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:53.548971 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/55b5d3f9-e42c-40be-a9a4-3cc3c81ad7e2-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"55b5d3f9-e42c-40be-a9a4-3cc3c81ad7e2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 07:30:53.549129 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:53.549031 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/55b5d3f9-e42c-40be-a9a4-3cc3c81ad7e2-config-volume\") pod \"alertmanager-main-0\" (UID: \"55b5d3f9-e42c-40be-a9a4-3cc3c81ad7e2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 07:30:53.549129 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:53.549072 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/55b5d3f9-e42c-40be-a9a4-3cc3c81ad7e2-web-config\") pod \"alertmanager-main-0\" (UID: \"55b5d3f9-e42c-40be-a9a4-3cc3c81ad7e2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 07:30:53.549129 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:53.549100 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/55b5d3f9-e42c-40be-a9a4-3cc3c81ad7e2-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"55b5d3f9-e42c-40be-a9a4-3cc3c81ad7e2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 07:30:53.549323 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:53.549134 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/55b5d3f9-e42c-40be-a9a4-3cc3c81ad7e2-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"55b5d3f9-e42c-40be-a9a4-3cc3c81ad7e2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 07:30:53.549323 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:53.549157 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lcf9\" (UniqueName: \"kubernetes.io/projected/55b5d3f9-e42c-40be-a9a4-3cc3c81ad7e2-kube-api-access-8lcf9\") pod \"alertmanager-main-0\" (UID: \"55b5d3f9-e42c-40be-a9a4-3cc3c81ad7e2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 07:30:53.550583 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:53.550562 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Sep 17 07:30:53.554916 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:53.554892 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 17 07:30:53.650342 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:53.650314 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/55b5d3f9-e42c-40be-a9a4-3cc3c81ad7e2-web-config\") pod \"alertmanager-main-0\" (UID: \"55b5d3f9-e42c-40be-a9a4-3cc3c81ad7e2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 07:30:53.650482 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:53.650345 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/55b5d3f9-e42c-40be-a9a4-3cc3c81ad7e2-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"55b5d3f9-e42c-40be-a9a4-3cc3c81ad7e2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 07:30:53.650482 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:53.650365 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/55b5d3f9-e42c-40be-a9a4-3cc3c81ad7e2-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"55b5d3f9-e42c-40be-a9a4-3cc3c81ad7e2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 07:30:53.650482 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:53.650380 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lcf9\" (UniqueName: \"kubernetes.io/projected/55b5d3f9-e42c-40be-a9a4-3cc3c81ad7e2-kube-api-access-8lcf9\") pod \"alertmanager-main-0\" (UID: \"55b5d3f9-e42c-40be-a9a4-3cc3c81ad7e2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 07:30:53.650652 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:53.650507 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/55b5d3f9-e42c-40be-a9a4-3cc3c81ad7e2-config-out\") pod \"alertmanager-main-0\" (UID: \"55b5d3f9-e42c-40be-a9a4-3cc3c81ad7e2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 07:30:53.650652 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:53.650556 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/55b5d3f9-e42c-40be-a9a4-3cc3c81ad7e2-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"55b5d3f9-e42c-40be-a9a4-3cc3c81ad7e2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 07:30:53.650652 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:53.650595 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/55b5d3f9-e42c-40be-a9a4-3cc3c81ad7e2-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"55b5d3f9-e42c-40be-a9a4-3cc3c81ad7e2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 07:30:53.650652 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:53.650639 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/55b5d3f9-e42c-40be-a9a4-3cc3c81ad7e2-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"55b5d3f9-e42c-40be-a9a4-3cc3c81ad7e2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 07:30:53.650840 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:53.650680 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/55b5d3f9-e42c-40be-a9a4-3cc3c81ad7e2-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"55b5d3f9-e42c-40be-a9a4-3cc3c81ad7e2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 07:30:53.650840 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:53.650716 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/55b5d3f9-e42c-40be-a9a4-3cc3c81ad7e2-tls-assets\") pod \"alertmanager-main-0\" (UID: \"55b5d3f9-e42c-40be-a9a4-3cc3c81ad7e2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 07:30:53.650840 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:53.650741 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/55b5d3f9-e42c-40be-a9a4-3cc3c81ad7e2-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"55b5d3f9-e42c-40be-a9a4-3cc3c81ad7e2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 07:30:53.650840 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:53.650765 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/55b5d3f9-e42c-40be-a9a4-3cc3c81ad7e2-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"55b5d3f9-e42c-40be-a9a4-3cc3c81ad7e2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 07:30:53.650840 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:53.650797 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/55b5d3f9-e42c-40be-a9a4-3cc3c81ad7e2-config-volume\") pod \"alertmanager-main-0\" (UID: \"55b5d3f9-e42c-40be-a9a4-3cc3c81ad7e2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 07:30:53.651468 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:53.651171 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/55b5d3f9-e42c-40be-a9a4-3cc3c81ad7e2-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"55b5d3f9-e42c-40be-a9a4-3cc3c81ad7e2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 07:30:53.651625 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:53.651593 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/55b5d3f9-e42c-40be-a9a4-3cc3c81ad7e2-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"55b5d3f9-e42c-40be-a9a4-3cc3c81ad7e2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 07:30:53.653796 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:53.653223 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/55b5d3f9-e42c-40be-a9a4-3cc3c81ad7e2-config-out\") pod \"alertmanager-main-0\" (UID: \"55b5d3f9-e42c-40be-a9a4-3cc3c81ad7e2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 07:30:53.653796 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:53.653503 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/55b5d3f9-e42c-40be-a9a4-3cc3c81ad7e2-web-config\") pod \"alertmanager-main-0\" (UID: \"55b5d3f9-e42c-40be-a9a4-3cc3c81ad7e2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 07:30:53.653796 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:53.653504 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/55b5d3f9-e42c-40be-a9a4-3cc3c81ad7e2-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"55b5d3f9-e42c-40be-a9a4-3cc3c81ad7e2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 07:30:53.653796 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:53.653519 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/55b5d3f9-e42c-40be-a9a4-3cc3c81ad7e2-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"55b5d3f9-e42c-40be-a9a4-3cc3c81ad7e2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 07:30:53.653796 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:53.653669 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/55b5d3f9-e42c-40be-a9a4-3cc3c81ad7e2-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"55b5d3f9-e42c-40be-a9a4-3cc3c81ad7e2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 07:30:53.654136 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:53.653957 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/55b5d3f9-e42c-40be-a9a4-3cc3c81ad7e2-config-volume\") pod \"alertmanager-main-0\" (UID: \"55b5d3f9-e42c-40be-a9a4-3cc3c81ad7e2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 07:30:53.654626 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:53.654603 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/55b5d3f9-e42c-40be-a9a4-3cc3c81ad7e2-tls-assets\") pod \"alertmanager-main-0\" (UID: \"55b5d3f9-e42c-40be-a9a4-3cc3c81ad7e2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 07:30:53.654738 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:53.654607 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/55b5d3f9-e42c-40be-a9a4-3cc3c81ad7e2-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"55b5d3f9-e42c-40be-a9a4-3cc3c81ad7e2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 07:30:53.654738 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:53.654608 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/55b5d3f9-e42c-40be-a9a4-3cc3c81ad7e2-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"55b5d3f9-e42c-40be-a9a4-3cc3c81ad7e2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 07:30:53.655575 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:53.655553 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/55b5d3f9-e42c-40be-a9a4-3cc3c81ad7e2-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"55b5d3f9-e42c-40be-a9a4-3cc3c81ad7e2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 07:30:53.658235 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:53.658219 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lcf9\" (UniqueName: \"kubernetes.io/projected/55b5d3f9-e42c-40be-a9a4-3cc3c81ad7e2-kube-api-access-8lcf9\") pod \"alertmanager-main-0\" (UID: \"55b5d3f9-e42c-40be-a9a4-3cc3c81ad7e2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 07:30:53.849845 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:53.849816 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 17 07:30:54.133094 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:54.133068 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 17 07:30:54.133992 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:30:54.133968 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55b5d3f9_e42c_40be_a9a4_3cc3c81ad7e2.slice/crio-d92a82b07f1678e3a389dfd6398c6bd78a44c5572a9aefa7697075f2a7adefde WatchSource:0}: Error finding container d92a82b07f1678e3a389dfd6398c6bd78a44c5572a9aefa7697075f2a7adefde: Status 404 returned error can't find the container with id d92a82b07f1678e3a389dfd6398c6bd78a44c5572a9aefa7697075f2a7adefde Sep 17 07:30:54.474502 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:54.474473 2578 generic.go:356] "Generic (PLEG): container finished" podID="55b5d3f9-e42c-40be-a9a4-3cc3c81ad7e2" containerID="d3bd1ad01e28882bd688e6cf8f80aa07168b03b1a68b0e310224c3d72c7983a4" exitCode=0 Sep 17 07:30:54.474929 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:54.474550 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"55b5d3f9-e42c-40be-a9a4-3cc3c81ad7e2","Type":"ContainerDied","Data":"d3bd1ad01e28882bd688e6cf8f80aa07168b03b1a68b0e310224c3d72c7983a4"} Sep 17 07:30:54.474929 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:54.474627 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"55b5d3f9-e42c-40be-a9a4-3cc3c81ad7e2","Type":"ContainerStarted","Data":"d92a82b07f1678e3a389dfd6398c6bd78a44c5572a9aefa7697075f2a7adefde"} Sep 17 07:30:54.631766 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:54.631725 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ccdd24e-b616-4c0f-b552-17948e1f2f87" path="/var/lib/kubelet/pods/1ccdd24e-b616-4c0f-b552-17948e1f2f87/volumes" Sep 17 07:30:55.431469 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:55.431432 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-68dddff84-wggvb"] Sep 17 07:30:55.433946 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:55.433930 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-68dddff84-wggvb" Sep 17 07:30:55.437399 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:55.437377 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Sep 17 07:30:55.437541 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:55.437524 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Sep 17 07:30:55.438233 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:55.438220 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-6xqn8\"" Sep 17 07:30:55.438582 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:55.438564 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Sep 17 07:30:55.438648 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:55.438587 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Sep 17 07:30:55.438648 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:55.438621 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Sep 17 07:30:55.444336 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:55.444317 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Sep 17 07:30:55.464130 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:55.464107 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6869870-c751-48a4-8f01-bc70ae552eb4-serving-certs-ca-bundle\") pod \"telemeter-client-68dddff84-wggvb\" (UID: \"c6869870-c751-48a4-8f01-bc70ae552eb4\") " pod="openshift-monitoring/telemeter-client-68dddff84-wggvb" Sep 17 07:30:55.464313 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:55.464144 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6869870-c751-48a4-8f01-bc70ae552eb4-telemeter-trusted-ca-bundle\") pod \"telemeter-client-68dddff84-wggvb\" (UID: \"c6869870-c751-48a4-8f01-bc70ae552eb4\") " pod="openshift-monitoring/telemeter-client-68dddff84-wggvb" Sep 17 07:30:55.464313 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:55.464178 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hkt2\" (UniqueName: \"kubernetes.io/projected/c6869870-c751-48a4-8f01-bc70ae552eb4-kube-api-access-2hkt2\") pod \"telemeter-client-68dddff84-wggvb\" (UID: \"c6869870-c751-48a4-8f01-bc70ae552eb4\") " pod="openshift-monitoring/telemeter-client-68dddff84-wggvb" Sep 17 07:30:55.464313 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:55.464237 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/c6869870-c751-48a4-8f01-bc70ae552eb4-secret-telemeter-client\") pod \"telemeter-client-68dddff84-wggvb\" (UID: \"c6869870-c751-48a4-8f01-bc70ae552eb4\") " pod="openshift-monitoring/telemeter-client-68dddff84-wggvb" Sep 17 07:30:55.464313 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:55.464262 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c6869870-c751-48a4-8f01-bc70ae552eb4-metrics-client-ca\") pod \"telemeter-client-68dddff84-wggvb\" (UID: \"c6869870-c751-48a4-8f01-bc70ae552eb4\") " pod="openshift-monitoring/telemeter-client-68dddff84-wggvb" Sep 17 07:30:55.464313 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:55.464282 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/c6869870-c751-48a4-8f01-bc70ae552eb4-federate-client-tls\") pod \"telemeter-client-68dddff84-wggvb\" (UID: \"c6869870-c751-48a4-8f01-bc70ae552eb4\") " pod="openshift-monitoring/telemeter-client-68dddff84-wggvb" Sep 17 07:30:55.464313 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:55.464299 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c6869870-c751-48a4-8f01-bc70ae552eb4-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-68dddff84-wggvb\" (UID: \"c6869870-c751-48a4-8f01-bc70ae552eb4\") " pod="openshift-monitoring/telemeter-client-68dddff84-wggvb" Sep 17 07:30:55.464516 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:55.464320 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/c6869870-c751-48a4-8f01-bc70ae552eb4-telemeter-client-tls\") pod \"telemeter-client-68dddff84-wggvb\" (UID: \"c6869870-c751-48a4-8f01-bc70ae552eb4\") " pod="openshift-monitoring/telemeter-client-68dddff84-wggvb" Sep 17 07:30:55.470204 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:55.470183 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-68dddff84-wggvb"] Sep 17 07:30:55.486163 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:55.486138 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"55b5d3f9-e42c-40be-a9a4-3cc3c81ad7e2","Type":"ContainerStarted","Data":"8bce75a0147725db65f7ea5148a90218f58a71cb7d23b486582ef2a49025c82f"} Sep 17 07:30:55.486454 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:55.486207 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"55b5d3f9-e42c-40be-a9a4-3cc3c81ad7e2","Type":"ContainerStarted","Data":"9799f4903024d0f7c0e9db626139ccc7ff2274358b2e2cb1958d4a9e1bd3e498"} Sep 17 07:30:55.486454 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:55.486244 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"55b5d3f9-e42c-40be-a9a4-3cc3c81ad7e2","Type":"ContainerStarted","Data":"e2590cacadb4b596af3e712d697afb820652c5d7494002ddf0425377bb829f34"} Sep 17 07:30:55.486454 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:55.486282 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"55b5d3f9-e42c-40be-a9a4-3cc3c81ad7e2","Type":"ContainerStarted","Data":"34338da41a1b65dc24b9a102ddbce1a44f2db54ee51aa1e4090c261a55dda70f"} Sep 17 07:30:55.486454 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:55.486321 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"55b5d3f9-e42c-40be-a9a4-3cc3c81ad7e2","Type":"ContainerStarted","Data":"bf3207a1f5948530908eec8404f8de867a3f192cfc4d1b77ebadcb1f9b8b0138"} Sep 17 07:30:55.486454 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:55.486349 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"55b5d3f9-e42c-40be-a9a4-3cc3c81ad7e2","Type":"ContainerStarted","Data":"0d1ec2d8b29d2de164fa969ace68d59a115562482e624c0db7fcbba4a20a6afa"} Sep 17 07:30:55.542476 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:55.542421 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.527406516 podStartE2EDuration="2.542400595s" totalImagesPullingTime="0s" totalInitContainerRuntime="14.994079ms" isStatefulPod=false podCreationTimestamp="2026-09-17 07:30:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 07:30:55.540007986 +0000 UTC m=+199.421042344" watchObservedRunningTime="2026-09-17 07:30:55.542400595 +0000 UTC m=+199.423434958" Sep 17 07:30:55.567957 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:55.567719 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6869870-c751-48a4-8f01-bc70ae552eb4-serving-certs-ca-bundle\") pod \"telemeter-client-68dddff84-wggvb\" (UID: \"c6869870-c751-48a4-8f01-bc70ae552eb4\") " pod="openshift-monitoring/telemeter-client-68dddff84-wggvb" Sep 17 07:30:55.568201 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:55.568184 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6869870-c751-48a4-8f01-bc70ae552eb4-telemeter-trusted-ca-bundle\") pod \"telemeter-client-68dddff84-wggvb\" (UID: \"c6869870-c751-48a4-8f01-bc70ae552eb4\") " pod="openshift-monitoring/telemeter-client-68dddff84-wggvb" Sep 17 07:30:55.570489 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:55.568916 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hkt2\" (UniqueName: \"kubernetes.io/projected/c6869870-c751-48a4-8f01-bc70ae552eb4-kube-api-access-2hkt2\") pod \"telemeter-client-68dddff84-wggvb\" (UID: \"c6869870-c751-48a4-8f01-bc70ae552eb4\") " pod="openshift-monitoring/telemeter-client-68dddff84-wggvb" Sep 17 07:30:55.570489 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:55.568991 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/c6869870-c751-48a4-8f01-bc70ae552eb4-secret-telemeter-client\") pod \"telemeter-client-68dddff84-wggvb\" (UID: \"c6869870-c751-48a4-8f01-bc70ae552eb4\") " pod="openshift-monitoring/telemeter-client-68dddff84-wggvb" Sep 17 07:30:55.570489 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:55.569009 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6869870-c751-48a4-8f01-bc70ae552eb4-telemeter-trusted-ca-bundle\") pod \"telemeter-client-68dddff84-wggvb\" (UID: \"c6869870-c751-48a4-8f01-bc70ae552eb4\") " pod="openshift-monitoring/telemeter-client-68dddff84-wggvb" Sep 17 07:30:55.570489 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:55.569046 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c6869870-c751-48a4-8f01-bc70ae552eb4-metrics-client-ca\") pod \"telemeter-client-68dddff84-wggvb\" (UID: \"c6869870-c751-48a4-8f01-bc70ae552eb4\") " pod="openshift-monitoring/telemeter-client-68dddff84-wggvb" Sep 17 07:30:55.570489 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:55.569075 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/c6869870-c751-48a4-8f01-bc70ae552eb4-federate-client-tls\") pod \"telemeter-client-68dddff84-wggvb\" (UID: \"c6869870-c751-48a4-8f01-bc70ae552eb4\") " pod="openshift-monitoring/telemeter-client-68dddff84-wggvb" Sep 17 07:30:55.570489 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:55.569105 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c6869870-c751-48a4-8f01-bc70ae552eb4-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-68dddff84-wggvb\" (UID: \"c6869870-c751-48a4-8f01-bc70ae552eb4\") " pod="openshift-monitoring/telemeter-client-68dddff84-wggvb" Sep 17 07:30:55.570489 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:55.569143 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/c6869870-c751-48a4-8f01-bc70ae552eb4-telemeter-client-tls\") pod \"telemeter-client-68dddff84-wggvb\" (UID: \"c6869870-c751-48a4-8f01-bc70ae552eb4\") " pod="openshift-monitoring/telemeter-client-68dddff84-wggvb" Sep 17 07:30:55.570489 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:55.568624 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6869870-c751-48a4-8f01-bc70ae552eb4-serving-certs-ca-bundle\") pod \"telemeter-client-68dddff84-wggvb\" (UID: \"c6869870-c751-48a4-8f01-bc70ae552eb4\") " pod="openshift-monitoring/telemeter-client-68dddff84-wggvb" Sep 17 07:30:55.570489 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:55.570403 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c6869870-c751-48a4-8f01-bc70ae552eb4-metrics-client-ca\") pod \"telemeter-client-68dddff84-wggvb\" (UID: \"c6869870-c751-48a4-8f01-bc70ae552eb4\") " pod="openshift-monitoring/telemeter-client-68dddff84-wggvb" Sep 17 07:30:55.572978 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:55.572952 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/c6869870-c751-48a4-8f01-bc70ae552eb4-secret-telemeter-client\") pod \"telemeter-client-68dddff84-wggvb\" (UID: \"c6869870-c751-48a4-8f01-bc70ae552eb4\") " pod="openshift-monitoring/telemeter-client-68dddff84-wggvb" Sep 17 07:30:55.574253 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:55.574226 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c6869870-c751-48a4-8f01-bc70ae552eb4-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-68dddff84-wggvb\" (UID: \"c6869870-c751-48a4-8f01-bc70ae552eb4\") " pod="openshift-monitoring/telemeter-client-68dddff84-wggvb" Sep 17 07:30:55.574393 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:55.574370 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/c6869870-c751-48a4-8f01-bc70ae552eb4-federate-client-tls\") pod \"telemeter-client-68dddff84-wggvb\" (UID: \"c6869870-c751-48a4-8f01-bc70ae552eb4\") " pod="openshift-monitoring/telemeter-client-68dddff84-wggvb" Sep 17 07:30:55.574539 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:55.574370 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/c6869870-c751-48a4-8f01-bc70ae552eb4-telemeter-client-tls\") pod \"telemeter-client-68dddff84-wggvb\" (UID: \"c6869870-c751-48a4-8f01-bc70ae552eb4\") " pod="openshift-monitoring/telemeter-client-68dddff84-wggvb" Sep 17 07:30:55.608726 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:55.608694 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hkt2\" (UniqueName: \"kubernetes.io/projected/c6869870-c751-48a4-8f01-bc70ae552eb4-kube-api-access-2hkt2\") pod \"telemeter-client-68dddff84-wggvb\" (UID: \"c6869870-c751-48a4-8f01-bc70ae552eb4\") " pod="openshift-monitoring/telemeter-client-68dddff84-wggvb" Sep 17 07:30:55.743343 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:55.743264 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-68dddff84-wggvb" Sep 17 07:30:56.038137 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:56.038057 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-68dddff84-wggvb"] Sep 17 07:30:56.041951 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:30:56.041916 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6869870_c751_48a4_8f01_bc70ae552eb4.slice/crio-f6984fafa27c9e92b6cd08dc408e576748ecbdc9a2c22e2a43154c17424ba62a WatchSource:0}: Error finding container f6984fafa27c9e92b6cd08dc408e576748ecbdc9a2c22e2a43154c17424ba62a: Status 404 returned error can't find the container with id f6984fafa27c9e92b6cd08dc408e576748ecbdc9a2c22e2a43154c17424ba62a Sep 17 07:30:56.491714 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:56.491681 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-68dddff84-wggvb" event={"ID":"c6869870-c751-48a4-8f01-bc70ae552eb4","Type":"ContainerStarted","Data":"f6984fafa27c9e92b6cd08dc408e576748ecbdc9a2c22e2a43154c17424ba62a"} Sep 17 07:30:58.502983 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:58.502954 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-68dddff84-wggvb" event={"ID":"c6869870-c751-48a4-8f01-bc70ae552eb4","Type":"ContainerStarted","Data":"66cb336f0abb663552fb0625b35745420084cc5fc99b249be835ff8f00d7c584"} Sep 17 07:30:58.503324 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:58.503034 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-68dddff84-wggvb" event={"ID":"c6869870-c751-48a4-8f01-bc70ae552eb4","Type":"ContainerStarted","Data":"6e0d27f3767aec43f6406cddf48edc80d00ec02c814bd00023eeb7756f12bb33"} Sep 17 07:30:58.503324 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:58.503055 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-68dddff84-wggvb" event={"ID":"c6869870-c751-48a4-8f01-bc70ae552eb4","Type":"ContainerStarted","Data":"7c36f4297cc7a74b104a4008ebead9f17fc6d20da79e0bcfe57b71d593370059"} Sep 17 07:30:58.536164 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:58.536087 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-68dddff84-wggvb" podStartSLOduration=1.296912052 podStartE2EDuration="3.536072482s" totalImagesPullingTime="2.23916043s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 07:30:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 07:30:58.535498235 +0000 UTC m=+202.416532593" watchObservedRunningTime="2026-09-17 07:30:58.536072482 +0000 UTC m=+202.417106851" Sep 17 07:30:58.698139 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:58.698112 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7758967d7d-d4mjf"] Sep 17 07:30:58.700664 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:58.700648 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7758967d7d-d4mjf" Sep 17 07:30:58.703548 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:58.703528 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 17 07:30:58.703663 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:58.703530 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 17 07:30:58.703750 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:58.703734 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 17 07:30:58.703922 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:58.703909 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-4hwfz\"" Sep 17 07:30:58.704235 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:58.704218 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 17 07:30:58.704314 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:58.704298 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 17 07:30:58.710786 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:58.710762 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Sep 17 07:30:58.721008 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:58.720989 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7758967d7d-d4mjf"] Sep 17 07:30:58.796884 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:58.796796 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a839f2da-8341-4985-8fe5-db569835f6e1-console-config\") pod \"console-7758967d7d-d4mjf\" (UID: \"a839f2da-8341-4985-8fe5-db569835f6e1\") " pod="openshift-console/console-7758967d7d-d4mjf" Sep 17 07:30:58.796884 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:58.796847 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4j8l\" (UniqueName: \"kubernetes.io/projected/a839f2da-8341-4985-8fe5-db569835f6e1-kube-api-access-x4j8l\") pod \"console-7758967d7d-d4mjf\" (UID: \"a839f2da-8341-4985-8fe5-db569835f6e1\") " pod="openshift-console/console-7758967d7d-d4mjf" Sep 17 07:30:58.797021 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:58.796945 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a839f2da-8341-4985-8fe5-db569835f6e1-oauth-serving-cert\") pod \"console-7758967d7d-d4mjf\" (UID: \"a839f2da-8341-4985-8fe5-db569835f6e1\") " pod="openshift-console/console-7758967d7d-d4mjf" Sep 17 07:30:58.797021 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:58.796996 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a839f2da-8341-4985-8fe5-db569835f6e1-trusted-ca-bundle\") pod \"console-7758967d7d-d4mjf\" (UID: \"a839f2da-8341-4985-8fe5-db569835f6e1\") " pod="openshift-console/console-7758967d7d-d4mjf" Sep 17 07:30:58.797085 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:58.797032 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a839f2da-8341-4985-8fe5-db569835f6e1-service-ca\") pod \"console-7758967d7d-d4mjf\" (UID: \"a839f2da-8341-4985-8fe5-db569835f6e1\") " pod="openshift-console/console-7758967d7d-d4mjf" Sep 17 07:30:58.797085 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:58.797050 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a839f2da-8341-4985-8fe5-db569835f6e1-console-serving-cert\") pod \"console-7758967d7d-d4mjf\" (UID: \"a839f2da-8341-4985-8fe5-db569835f6e1\") " pod="openshift-console/console-7758967d7d-d4mjf" Sep 17 07:30:58.797145 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:58.797106 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a839f2da-8341-4985-8fe5-db569835f6e1-console-oauth-config\") pod \"console-7758967d7d-d4mjf\" (UID: \"a839f2da-8341-4985-8fe5-db569835f6e1\") " pod="openshift-console/console-7758967d7d-d4mjf" Sep 17 07:30:58.897801 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:58.897775 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a839f2da-8341-4985-8fe5-db569835f6e1-console-config\") pod \"console-7758967d7d-d4mjf\" (UID: \"a839f2da-8341-4985-8fe5-db569835f6e1\") " pod="openshift-console/console-7758967d7d-d4mjf" Sep 17 07:30:58.897943 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:58.897813 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4j8l\" (UniqueName: \"kubernetes.io/projected/a839f2da-8341-4985-8fe5-db569835f6e1-kube-api-access-x4j8l\") pod \"console-7758967d7d-d4mjf\" (UID: \"a839f2da-8341-4985-8fe5-db569835f6e1\") " pod="openshift-console/console-7758967d7d-d4mjf" Sep 17 07:30:58.897943 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:58.897839 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a839f2da-8341-4985-8fe5-db569835f6e1-oauth-serving-cert\") pod \"console-7758967d7d-d4mjf\" (UID: \"a839f2da-8341-4985-8fe5-db569835f6e1\") " pod="openshift-console/console-7758967d7d-d4mjf" Sep 17 07:30:58.897943 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:58.897872 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a839f2da-8341-4985-8fe5-db569835f6e1-trusted-ca-bundle\") pod \"console-7758967d7d-d4mjf\" (UID: \"a839f2da-8341-4985-8fe5-db569835f6e1\") " pod="openshift-console/console-7758967d7d-d4mjf" Sep 17 07:30:58.897943 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:58.897893 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a839f2da-8341-4985-8fe5-db569835f6e1-service-ca\") pod \"console-7758967d7d-d4mjf\" (UID: \"a839f2da-8341-4985-8fe5-db569835f6e1\") " pod="openshift-console/console-7758967d7d-d4mjf" Sep 17 07:30:58.897943 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:58.897914 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a839f2da-8341-4985-8fe5-db569835f6e1-console-serving-cert\") pod \"console-7758967d7d-d4mjf\" (UID: \"a839f2da-8341-4985-8fe5-db569835f6e1\") " pod="openshift-console/console-7758967d7d-d4mjf" Sep 17 07:30:58.898201 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:58.897949 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a839f2da-8341-4985-8fe5-db569835f6e1-console-oauth-config\") pod \"console-7758967d7d-d4mjf\" (UID: \"a839f2da-8341-4985-8fe5-db569835f6e1\") " pod="openshift-console/console-7758967d7d-d4mjf" Sep 17 07:30:58.898627 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:58.898601 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a839f2da-8341-4985-8fe5-db569835f6e1-console-config\") pod \"console-7758967d7d-d4mjf\" (UID: \"a839f2da-8341-4985-8fe5-db569835f6e1\") " pod="openshift-console/console-7758967d7d-d4mjf" Sep 17 07:30:58.898712 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:58.898629 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a839f2da-8341-4985-8fe5-db569835f6e1-service-ca\") pod \"console-7758967d7d-d4mjf\" (UID: \"a839f2da-8341-4985-8fe5-db569835f6e1\") " pod="openshift-console/console-7758967d7d-d4mjf" Sep 17 07:30:58.898712 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:58.898629 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a839f2da-8341-4985-8fe5-db569835f6e1-oauth-serving-cert\") pod \"console-7758967d7d-d4mjf\" (UID: \"a839f2da-8341-4985-8fe5-db569835f6e1\") " pod="openshift-console/console-7758967d7d-d4mjf" Sep 17 07:30:58.898712 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:58.898690 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a839f2da-8341-4985-8fe5-db569835f6e1-trusted-ca-bundle\") pod \"console-7758967d7d-d4mjf\" (UID: \"a839f2da-8341-4985-8fe5-db569835f6e1\") " pod="openshift-console/console-7758967d7d-d4mjf" Sep 17 07:30:58.900293 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:58.900273 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a839f2da-8341-4985-8fe5-db569835f6e1-console-serving-cert\") pod \"console-7758967d7d-d4mjf\" (UID: \"a839f2da-8341-4985-8fe5-db569835f6e1\") " pod="openshift-console/console-7758967d7d-d4mjf" Sep 17 07:30:58.900373 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:58.900319 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a839f2da-8341-4985-8fe5-db569835f6e1-console-oauth-config\") pod \"console-7758967d7d-d4mjf\" (UID: \"a839f2da-8341-4985-8fe5-db569835f6e1\") " pod="openshift-console/console-7758967d7d-d4mjf" Sep 17 07:30:58.909941 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:58.909918 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4j8l\" (UniqueName: \"kubernetes.io/projected/a839f2da-8341-4985-8fe5-db569835f6e1-kube-api-access-x4j8l\") pod \"console-7758967d7d-d4mjf\" (UID: \"a839f2da-8341-4985-8fe5-db569835f6e1\") " pod="openshift-console/console-7758967d7d-d4mjf" Sep 17 07:30:59.009991 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:59.009965 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7758967d7d-d4mjf" Sep 17 07:30:59.287350 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:59.287302 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7758967d7d-d4mjf"] Sep 17 07:30:59.289664 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:30:59.289639 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda839f2da_8341_4985_8fe5_db569835f6e1.slice/crio-7709dc8dadca8e43d123efe60383617ed52398ca236f480ebeb9cf21e394b061 WatchSource:0}: Error finding container 7709dc8dadca8e43d123efe60383617ed52398ca236f480ebeb9cf21e394b061: Status 404 returned error can't find the container with id 7709dc8dadca8e43d123efe60383617ed52398ca236f480ebeb9cf21e394b061 Sep 17 07:30:59.509320 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:59.509281 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7758967d7d-d4mjf" event={"ID":"a839f2da-8341-4985-8fe5-db569835f6e1","Type":"ContainerStarted","Data":"e1b1cc7e1f20fb5826d24d93d7c7163193724774f0be8dd3b7eebc1a2d5c6aca"} Sep 17 07:30:59.509707 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:59.509374 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7758967d7d-d4mjf" event={"ID":"a839f2da-8341-4985-8fe5-db569835f6e1","Type":"ContainerStarted","Data":"7709dc8dadca8e43d123efe60383617ed52398ca236f480ebeb9cf21e394b061"} Sep 17 07:30:59.538662 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:30:59.538611 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7758967d7d-d4mjf" podStartSLOduration=1.5385991639999999 podStartE2EDuration="1.538599164s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 07:30:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 07:30:59.537324527 +0000 UTC m=+203.418358912" watchObservedRunningTime="2026-09-17 07:30:59.538599164 +0000 UTC m=+203.419633522" Sep 17 07:31:09.010405 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:31:09.010358 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7758967d7d-d4mjf" Sep 17 07:31:09.010405 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:31:09.010420 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7758967d7d-d4mjf" Sep 17 07:31:09.014810 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:31:09.014788 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7758967d7d-d4mjf" Sep 17 07:31:09.552116 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:31:09.552091 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7758967d7d-d4mjf" Sep 17 07:32:13.707722 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:32:13.707655 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5d8cc66756-8kwwx"] Sep 17 07:32:13.709742 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:32:13.709726 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d8cc66756-8kwwx" Sep 17 07:32:13.723076 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:32:13.723053 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d8cc66756-8kwwx"] Sep 17 07:32:13.760321 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:32:13.760300 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7928d319-52af-43dd-bdcd-d3e4893e0219-console-oauth-config\") pod \"console-5d8cc66756-8kwwx\" (UID: \"7928d319-52af-43dd-bdcd-d3e4893e0219\") " pod="openshift-console/console-5d8cc66756-8kwwx" Sep 17 07:32:13.760438 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:32:13.760327 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7928d319-52af-43dd-bdcd-d3e4893e0219-oauth-serving-cert\") pod \"console-5d8cc66756-8kwwx\" (UID: \"7928d319-52af-43dd-bdcd-d3e4893e0219\") " pod="openshift-console/console-5d8cc66756-8kwwx" Sep 17 07:32:13.760438 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:32:13.760346 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7928d319-52af-43dd-bdcd-d3e4893e0219-service-ca\") pod \"console-5d8cc66756-8kwwx\" (UID: \"7928d319-52af-43dd-bdcd-d3e4893e0219\") " pod="openshift-console/console-5d8cc66756-8kwwx" Sep 17 07:32:13.760438 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:32:13.760361 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7928d319-52af-43dd-bdcd-d3e4893e0219-trusted-ca-bundle\") pod \"console-5d8cc66756-8kwwx\" (UID: \"7928d319-52af-43dd-bdcd-d3e4893e0219\") " pod="openshift-console/console-5d8cc66756-8kwwx" Sep 17 07:32:13.760539 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:32:13.760430 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjkwk\" (UniqueName: \"kubernetes.io/projected/7928d319-52af-43dd-bdcd-d3e4893e0219-kube-api-access-mjkwk\") pod \"console-5d8cc66756-8kwwx\" (UID: \"7928d319-52af-43dd-bdcd-d3e4893e0219\") " pod="openshift-console/console-5d8cc66756-8kwwx" Sep 17 07:32:13.760539 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:32:13.760492 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7928d319-52af-43dd-bdcd-d3e4893e0219-console-serving-cert\") pod \"console-5d8cc66756-8kwwx\" (UID: \"7928d319-52af-43dd-bdcd-d3e4893e0219\") " pod="openshift-console/console-5d8cc66756-8kwwx" Sep 17 07:32:13.760600 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:32:13.760542 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7928d319-52af-43dd-bdcd-d3e4893e0219-console-config\") pod \"console-5d8cc66756-8kwwx\" (UID: \"7928d319-52af-43dd-bdcd-d3e4893e0219\") " pod="openshift-console/console-5d8cc66756-8kwwx" Sep 17 07:32:13.861719 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:32:13.861690 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7928d319-52af-43dd-bdcd-d3e4893e0219-console-oauth-config\") pod \"console-5d8cc66756-8kwwx\" (UID: \"7928d319-52af-43dd-bdcd-d3e4893e0219\") " pod="openshift-console/console-5d8cc66756-8kwwx" Sep 17 07:32:13.861719 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:32:13.861721 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7928d319-52af-43dd-bdcd-d3e4893e0219-oauth-serving-cert\") pod \"console-5d8cc66756-8kwwx\" (UID: \"7928d319-52af-43dd-bdcd-d3e4893e0219\") " pod="openshift-console/console-5d8cc66756-8kwwx" Sep 17 07:32:13.861941 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:32:13.861739 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7928d319-52af-43dd-bdcd-d3e4893e0219-service-ca\") pod \"console-5d8cc66756-8kwwx\" (UID: \"7928d319-52af-43dd-bdcd-d3e4893e0219\") " pod="openshift-console/console-5d8cc66756-8kwwx" Sep 17 07:32:13.861941 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:32:13.861753 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7928d319-52af-43dd-bdcd-d3e4893e0219-trusted-ca-bundle\") pod \"console-5d8cc66756-8kwwx\" (UID: \"7928d319-52af-43dd-bdcd-d3e4893e0219\") " pod="openshift-console/console-5d8cc66756-8kwwx" Sep 17 07:32:13.861941 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:32:13.861777 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjkwk\" (UniqueName: \"kubernetes.io/projected/7928d319-52af-43dd-bdcd-d3e4893e0219-kube-api-access-mjkwk\") pod \"console-5d8cc66756-8kwwx\" (UID: \"7928d319-52af-43dd-bdcd-d3e4893e0219\") " pod="openshift-console/console-5d8cc66756-8kwwx" Sep 17 07:32:13.861941 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:32:13.861843 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7928d319-52af-43dd-bdcd-d3e4893e0219-console-serving-cert\") pod \"console-5d8cc66756-8kwwx\" (UID: \"7928d319-52af-43dd-bdcd-d3e4893e0219\") " pod="openshift-console/console-5d8cc66756-8kwwx" Sep 17 07:32:13.861941 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:32:13.861934 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7928d319-52af-43dd-bdcd-d3e4893e0219-console-config\") pod \"console-5d8cc66756-8kwwx\" (UID: \"7928d319-52af-43dd-bdcd-d3e4893e0219\") " pod="openshift-console/console-5d8cc66756-8kwwx" Sep 17 07:32:13.862521 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:32:13.862495 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7928d319-52af-43dd-bdcd-d3e4893e0219-oauth-serving-cert\") pod \"console-5d8cc66756-8kwwx\" (UID: \"7928d319-52af-43dd-bdcd-d3e4893e0219\") " pod="openshift-console/console-5d8cc66756-8kwwx" Sep 17 07:32:13.862722 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:32:13.862695 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7928d319-52af-43dd-bdcd-d3e4893e0219-console-config\") pod \"console-5d8cc66756-8kwwx\" (UID: \"7928d319-52af-43dd-bdcd-d3e4893e0219\") " pod="openshift-console/console-5d8cc66756-8kwwx" Sep 17 07:32:13.862795 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:32:13.862736 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7928d319-52af-43dd-bdcd-d3e4893e0219-service-ca\") pod \"console-5d8cc66756-8kwwx\" (UID: \"7928d319-52af-43dd-bdcd-d3e4893e0219\") " pod="openshift-console/console-5d8cc66756-8kwwx" Sep 17 07:32:13.862795 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:32:13.862736 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7928d319-52af-43dd-bdcd-d3e4893e0219-trusted-ca-bundle\") pod \"console-5d8cc66756-8kwwx\" (UID: \"7928d319-52af-43dd-bdcd-d3e4893e0219\") " pod="openshift-console/console-5d8cc66756-8kwwx" Sep 17 07:32:13.864264 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:32:13.864248 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7928d319-52af-43dd-bdcd-d3e4893e0219-console-serving-cert\") pod \"console-5d8cc66756-8kwwx\" (UID: \"7928d319-52af-43dd-bdcd-d3e4893e0219\") " pod="openshift-console/console-5d8cc66756-8kwwx" Sep 17 07:32:13.864419 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:32:13.864401 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7928d319-52af-43dd-bdcd-d3e4893e0219-console-oauth-config\") pod \"console-5d8cc66756-8kwwx\" (UID: \"7928d319-52af-43dd-bdcd-d3e4893e0219\") " pod="openshift-console/console-5d8cc66756-8kwwx" Sep 17 07:32:13.872335 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:32:13.872314 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjkwk\" (UniqueName: \"kubernetes.io/projected/7928d319-52af-43dd-bdcd-d3e4893e0219-kube-api-access-mjkwk\") pod \"console-5d8cc66756-8kwwx\" (UID: \"7928d319-52af-43dd-bdcd-d3e4893e0219\") " pod="openshift-console/console-5d8cc66756-8kwwx" Sep 17 07:32:14.018544 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:32:14.018452 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d8cc66756-8kwwx" Sep 17 07:32:14.291746 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:32:14.291679 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d8cc66756-8kwwx"] Sep 17 07:32:14.295010 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:32:14.294981 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7928d319_52af_43dd_bdcd_d3e4893e0219.slice/crio-320dadcddef96691b33b331190b49c3b0ab2ba523c25929fc973b0146c93d864 WatchSource:0}: Error finding container 320dadcddef96691b33b331190b49c3b0ab2ba523c25929fc973b0146c93d864: Status 404 returned error can't find the container with id 320dadcddef96691b33b331190b49c3b0ab2ba523c25929fc973b0146c93d864 Sep 17 07:32:14.793310 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:32:14.793279 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d8cc66756-8kwwx" event={"ID":"7928d319-52af-43dd-bdcd-d3e4893e0219","Type":"ContainerStarted","Data":"4325c6074eb0a6b9eeaa2773965902a2ae8d8c3675f057ece5dc7d22f7f65bbb"} Sep 17 07:32:14.793661 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:32:14.793323 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d8cc66756-8kwwx" event={"ID":"7928d319-52af-43dd-bdcd-d3e4893e0219","Type":"ContainerStarted","Data":"320dadcddef96691b33b331190b49c3b0ab2ba523c25929fc973b0146c93d864"} Sep 17 07:32:14.817418 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:32:14.817368 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5d8cc66756-8kwwx" podStartSLOduration=1.817354943 podStartE2EDuration="1.817354943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 07:32:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 07:32:14.816218635 +0000 UTC m=+278.697252995" watchObservedRunningTime="2026-09-17 07:32:14.817354943 +0000 UTC m=+278.698389302" Sep 17 07:32:24.018707 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:32:24.018669 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5d8cc66756-8kwwx" Sep 17 07:32:24.019208 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:32:24.018800 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5d8cc66756-8kwwx" Sep 17 07:32:24.023446 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:32:24.023425 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5d8cc66756-8kwwx" Sep 17 07:32:24.837473 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:32:24.837443 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5d8cc66756-8kwwx" Sep 17 07:32:24.920977 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:32:24.920943 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7758967d7d-d4mjf"] Sep 17 07:32:36.551543 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:32:36.551514 2578 kubelet.go:1648] "Image garbage collection succeeded" Sep 17 07:32:49.939484 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:32:49.939449 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7758967d7d-d4mjf" podUID="a839f2da-8341-4985-8fe5-db569835f6e1" containerName="console" containerID="cri-o://e1b1cc7e1f20fb5826d24d93d7c7163193724774f0be8dd3b7eebc1a2d5c6aca" gracePeriod=15 Sep 17 07:32:50.177672 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:32:50.177649 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7758967d7d-d4mjf" Sep 17 07:32:50.352138 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:32:50.352062 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a839f2da-8341-4985-8fe5-db569835f6e1-oauth-serving-cert\") pod \"a839f2da-8341-4985-8fe5-db569835f6e1\" (UID: \"a839f2da-8341-4985-8fe5-db569835f6e1\") " Sep 17 07:32:50.352138 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:32:50.352100 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a839f2da-8341-4985-8fe5-db569835f6e1-console-oauth-config\") pod \"a839f2da-8341-4985-8fe5-db569835f6e1\" (UID: \"a839f2da-8341-4985-8fe5-db569835f6e1\") " Sep 17 07:32:50.352138 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:32:50.352122 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a839f2da-8341-4985-8fe5-db569835f6e1-console-serving-cert\") pod \"a839f2da-8341-4985-8fe5-db569835f6e1\" (UID: \"a839f2da-8341-4985-8fe5-db569835f6e1\") " Sep 17 07:32:50.352382 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:32:50.352151 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a839f2da-8341-4985-8fe5-db569835f6e1-console-config\") pod \"a839f2da-8341-4985-8fe5-db569835f6e1\" (UID: \"a839f2da-8341-4985-8fe5-db569835f6e1\") " Sep 17 07:32:50.352382 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:32:50.352191 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a839f2da-8341-4985-8fe5-db569835f6e1-trusted-ca-bundle\") pod \"a839f2da-8341-4985-8fe5-db569835f6e1\" (UID: \"a839f2da-8341-4985-8fe5-db569835f6e1\") " Sep 17 07:32:50.352382 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:32:50.352223 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x4j8l\" (UniqueName: \"kubernetes.io/projected/a839f2da-8341-4985-8fe5-db569835f6e1-kube-api-access-x4j8l\") pod \"a839f2da-8341-4985-8fe5-db569835f6e1\" (UID: \"a839f2da-8341-4985-8fe5-db569835f6e1\") " Sep 17 07:32:50.352382 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:32:50.352271 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a839f2da-8341-4985-8fe5-db569835f6e1-service-ca\") pod \"a839f2da-8341-4985-8fe5-db569835f6e1\" (UID: \"a839f2da-8341-4985-8fe5-db569835f6e1\") " Sep 17 07:32:50.352612 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:32:50.352580 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a839f2da-8341-4985-8fe5-db569835f6e1-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "a839f2da-8341-4985-8fe5-db569835f6e1" (UID: "a839f2da-8341-4985-8fe5-db569835f6e1"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 07:32:50.352755 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:32:50.352625 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a839f2da-8341-4985-8fe5-db569835f6e1-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "a839f2da-8341-4985-8fe5-db569835f6e1" (UID: "a839f2da-8341-4985-8fe5-db569835f6e1"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 07:32:50.352755 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:32:50.352652 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a839f2da-8341-4985-8fe5-db569835f6e1-console-config" (OuterVolumeSpecName: "console-config") pod "a839f2da-8341-4985-8fe5-db569835f6e1" (UID: "a839f2da-8341-4985-8fe5-db569835f6e1"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 07:32:50.352755 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:32:50.352723 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a839f2da-8341-4985-8fe5-db569835f6e1-service-ca" (OuterVolumeSpecName: "service-ca") pod "a839f2da-8341-4985-8fe5-db569835f6e1" (UID: "a839f2da-8341-4985-8fe5-db569835f6e1"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 07:32:50.354402 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:32:50.354375 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a839f2da-8341-4985-8fe5-db569835f6e1-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "a839f2da-8341-4985-8fe5-db569835f6e1" (UID: "a839f2da-8341-4985-8fe5-db569835f6e1"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 07:32:50.354487 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:32:50.354428 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a839f2da-8341-4985-8fe5-db569835f6e1-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "a839f2da-8341-4985-8fe5-db569835f6e1" (UID: "a839f2da-8341-4985-8fe5-db569835f6e1"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 07:32:50.354487 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:32:50.354440 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a839f2da-8341-4985-8fe5-db569835f6e1-kube-api-access-x4j8l" (OuterVolumeSpecName: "kube-api-access-x4j8l") pod "a839f2da-8341-4985-8fe5-db569835f6e1" (UID: "a839f2da-8341-4985-8fe5-db569835f6e1"). InnerVolumeSpecName "kube-api-access-x4j8l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 07:32:50.452737 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:32:50.452710 2578 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a839f2da-8341-4985-8fe5-db569835f6e1-oauth-serving-cert\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Sep 17 07:32:50.452737 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:32:50.452733 2578 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a839f2da-8341-4985-8fe5-db569835f6e1-console-oauth-config\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Sep 17 07:32:50.452894 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:32:50.452742 2578 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a839f2da-8341-4985-8fe5-db569835f6e1-console-serving-cert\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Sep 17 07:32:50.452894 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:32:50.452753 2578 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a839f2da-8341-4985-8fe5-db569835f6e1-console-config\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Sep 17 07:32:50.452894 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:32:50.452762 2578 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a839f2da-8341-4985-8fe5-db569835f6e1-trusted-ca-bundle\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Sep 17 07:32:50.452894 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:32:50.452770 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4j8l\" (UniqueName: \"kubernetes.io/projected/a839f2da-8341-4985-8fe5-db569835f6e1-kube-api-access-x4j8l\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Sep 17 07:32:50.452894 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:32:50.452778 2578 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a839f2da-8341-4985-8fe5-db569835f6e1-service-ca\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Sep 17 07:32:50.934414 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:32:50.934384 2578 generic.go:356] "Generic (PLEG): container finished" podID="a839f2da-8341-4985-8fe5-db569835f6e1" containerID="e1b1cc7e1f20fb5826d24d93d7c7163193724774f0be8dd3b7eebc1a2d5c6aca" exitCode=0 Sep 17 07:32:50.934565 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:32:50.934423 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7758967d7d-d4mjf" Sep 17 07:32:50.934565 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:32:50.934466 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7758967d7d-d4mjf" event={"ID":"a839f2da-8341-4985-8fe5-db569835f6e1","Type":"ContainerDied","Data":"e1b1cc7e1f20fb5826d24d93d7c7163193724774f0be8dd3b7eebc1a2d5c6aca"} Sep 17 07:32:50.934565 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:32:50.934515 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7758967d7d-d4mjf" event={"ID":"a839f2da-8341-4985-8fe5-db569835f6e1","Type":"ContainerDied","Data":"7709dc8dadca8e43d123efe60383617ed52398ca236f480ebeb9cf21e394b061"} Sep 17 07:32:50.934565 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:32:50.934535 2578 scope.go:117] "RemoveContainer" containerID="e1b1cc7e1f20fb5826d24d93d7c7163193724774f0be8dd3b7eebc1a2d5c6aca" Sep 17 07:32:50.942916 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:32:50.942647 2578 scope.go:117] "RemoveContainer" containerID="e1b1cc7e1f20fb5826d24d93d7c7163193724774f0be8dd3b7eebc1a2d5c6aca" Sep 17 07:32:50.943224 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:32:50.942930 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1b1cc7e1f20fb5826d24d93d7c7163193724774f0be8dd3b7eebc1a2d5c6aca\": container with ID starting with e1b1cc7e1f20fb5826d24d93d7c7163193724774f0be8dd3b7eebc1a2d5c6aca not found: ID does not exist" containerID="e1b1cc7e1f20fb5826d24d93d7c7163193724774f0be8dd3b7eebc1a2d5c6aca" Sep 17 07:32:50.943224 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:32:50.942953 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1b1cc7e1f20fb5826d24d93d7c7163193724774f0be8dd3b7eebc1a2d5c6aca"} err="failed to get container status \"e1b1cc7e1f20fb5826d24d93d7c7163193724774f0be8dd3b7eebc1a2d5c6aca\": rpc error: code = NotFound desc = could not find container \"e1b1cc7e1f20fb5826d24d93d7c7163193724774f0be8dd3b7eebc1a2d5c6aca\": container with ID starting with e1b1cc7e1f20fb5826d24d93d7c7163193724774f0be8dd3b7eebc1a2d5c6aca not found: ID does not exist" Sep 17 07:32:50.961335 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:32:50.958784 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7758967d7d-d4mjf"] Sep 17 07:32:50.965164 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:32:50.965142 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7758967d7d-d4mjf"] Sep 17 07:32:52.628876 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:32:52.628834 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a839f2da-8341-4985-8fe5-db569835f6e1" path="/var/lib/kubelet/pods/a839f2da-8341-4985-8fe5-db569835f6e1/volumes" Sep 17 07:34:38.439681 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:34:38.439643 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5d8cc66756-8kwwx"] Sep 17 07:34:48.011062 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:34:48.011028 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["feast-operator-system/feast-operator-controller-manager-765754f648-2dc7h"] Sep 17 07:34:48.011467 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:34:48.011356 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a839f2da-8341-4985-8fe5-db569835f6e1" containerName="console" Sep 17 07:34:48.011467 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:34:48.011368 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a839f2da-8341-4985-8fe5-db569835f6e1" containerName="console" Sep 17 07:34:48.011467 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:34:48.011422 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="a839f2da-8341-4985-8fe5-db569835f6e1" containerName="console" Sep 17 07:34:48.014058 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:34:48.014042 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-765754f648-2dc7h" Sep 17 07:34:48.017569 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:34:48.017550 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"feast-operator-system\"/\"feast-operator-controller-manager-dockercfg-kllmn\"" Sep 17 07:34:48.018229 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:34:48.018204 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"feast-operator-system\"/\"openshift-service-ca.crt\"" Sep 17 07:34:48.018514 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:34:48.018501 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"feast-operator-system\"/\"kube-root-ca.crt\"" Sep 17 07:34:48.027466 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:34:48.027443 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-765754f648-2dc7h"] Sep 17 07:34:48.114876 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:34:48.114837 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rl89s\" (UniqueName: \"kubernetes.io/projected/8e776b6d-f611-4e71-b88f-130aa68a8300-kube-api-access-rl89s\") pod \"feast-operator-controller-manager-765754f648-2dc7h\" (UID: \"8e776b6d-f611-4e71-b88f-130aa68a8300\") " pod="feast-operator-system/feast-operator-controller-manager-765754f648-2dc7h" Sep 17 07:34:48.216162 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:34:48.216130 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rl89s\" (UniqueName: \"kubernetes.io/projected/8e776b6d-f611-4e71-b88f-130aa68a8300-kube-api-access-rl89s\") pod \"feast-operator-controller-manager-765754f648-2dc7h\" (UID: \"8e776b6d-f611-4e71-b88f-130aa68a8300\") " pod="feast-operator-system/feast-operator-controller-manager-765754f648-2dc7h" Sep 17 07:34:48.226575 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:34:48.226547 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rl89s\" (UniqueName: \"kubernetes.io/projected/8e776b6d-f611-4e71-b88f-130aa68a8300-kube-api-access-rl89s\") pod \"feast-operator-controller-manager-765754f648-2dc7h\" (UID: \"8e776b6d-f611-4e71-b88f-130aa68a8300\") " pod="feast-operator-system/feast-operator-controller-manager-765754f648-2dc7h" Sep 17 07:34:48.323706 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:34:48.323647 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-765754f648-2dc7h" Sep 17 07:34:48.596018 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:34:48.595931 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-765754f648-2dc7h"] Sep 17 07:34:48.600385 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:34:48.600359 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e776b6d_f611_4e71_b88f_130aa68a8300.slice/crio-ee1a106b0444df2937227d09d8cfa926089f842d5a842538451bd57b4b9f2227 WatchSource:0}: Error finding container ee1a106b0444df2937227d09d8cfa926089f842d5a842538451bd57b4b9f2227: Status 404 returned error can't find the container with id ee1a106b0444df2937227d09d8cfa926089f842d5a842538451bd57b4b9f2227 Sep 17 07:34:48.602178 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:34:48.602163 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 17 07:34:49.391405 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:34:49.391371 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-765754f648-2dc7h" event={"ID":"8e776b6d-f611-4e71-b88f-130aa68a8300","Type":"ContainerStarted","Data":"ee1a106b0444df2937227d09d8cfa926089f842d5a842538451bd57b4b9f2227"} Sep 17 07:34:51.401267 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:34:51.401235 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-765754f648-2dc7h" event={"ID":"8e776b6d-f611-4e71-b88f-130aa68a8300","Type":"ContainerStarted","Data":"00f366560fe52d3828fe77ff99ceb41902e9e0dfc4af9d261c106fd36593f2f5"} Sep 17 07:34:51.401667 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:34:51.401344 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="feast-operator-system/feast-operator-controller-manager-765754f648-2dc7h" Sep 17 07:34:51.424650 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:34:51.424606 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="feast-operator-system/feast-operator-controller-manager-765754f648-2dc7h" podStartSLOduration=2.055241862 podStartE2EDuration="4.424590515s" totalImagesPullingTime="2.369348653s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 07:34:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 07:34:51.422510379 +0000 UTC m=+435.303544741" watchObservedRunningTime="2026-09-17 07:34:51.424590515 +0000 UTC m=+435.305624873" Sep 17 07:35:02.408693 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:35:02.408617 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="feast-operator-system/feast-operator-controller-manager-765754f648-2dc7h" Sep 17 07:35:03.459042 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:35:03.459003 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5d8cc66756-8kwwx" podUID="7928d319-52af-43dd-bdcd-d3e4893e0219" containerName="console" containerID="cri-o://4325c6074eb0a6b9eeaa2773965902a2ae8d8c3675f057ece5dc7d22f7f65bbb" gracePeriod=15 Sep 17 07:35:03.694855 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:35:03.694829 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d8cc66756-8kwwx" Sep 17 07:35:03.723555 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:35:03.723492 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7928d319-52af-43dd-bdcd-d3e4893e0219-oauth-serving-cert\") pod \"7928d319-52af-43dd-bdcd-d3e4893e0219\" (UID: \"7928d319-52af-43dd-bdcd-d3e4893e0219\") " Sep 17 07:35:03.723555 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:35:03.723539 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7928d319-52af-43dd-bdcd-d3e4893e0219-service-ca\") pod \"7928d319-52af-43dd-bdcd-d3e4893e0219\" (UID: \"7928d319-52af-43dd-bdcd-d3e4893e0219\") " Sep 17 07:35:03.723555 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:35:03.723554 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7928d319-52af-43dd-bdcd-d3e4893e0219-console-config\") pod \"7928d319-52af-43dd-bdcd-d3e4893e0219\" (UID: \"7928d319-52af-43dd-bdcd-d3e4893e0219\") " Sep 17 07:35:03.723786 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:35:03.723590 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mjkwk\" (UniqueName: \"kubernetes.io/projected/7928d319-52af-43dd-bdcd-d3e4893e0219-kube-api-access-mjkwk\") pod \"7928d319-52af-43dd-bdcd-d3e4893e0219\" (UID: \"7928d319-52af-43dd-bdcd-d3e4893e0219\") " Sep 17 07:35:03.723786 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:35:03.723640 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7928d319-52af-43dd-bdcd-d3e4893e0219-console-serving-cert\") pod \"7928d319-52af-43dd-bdcd-d3e4893e0219\" (UID: \"7928d319-52af-43dd-bdcd-d3e4893e0219\") " Sep 17 07:35:03.723786 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:35:03.723667 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7928d319-52af-43dd-bdcd-d3e4893e0219-trusted-ca-bundle\") pod \"7928d319-52af-43dd-bdcd-d3e4893e0219\" (UID: \"7928d319-52af-43dd-bdcd-d3e4893e0219\") " Sep 17 07:35:03.723786 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:35:03.723691 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7928d319-52af-43dd-bdcd-d3e4893e0219-console-oauth-config\") pod \"7928d319-52af-43dd-bdcd-d3e4893e0219\" (UID: \"7928d319-52af-43dd-bdcd-d3e4893e0219\") " Sep 17 07:35:03.724009 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:35:03.723949 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7928d319-52af-43dd-bdcd-d3e4893e0219-console-config" (OuterVolumeSpecName: "console-config") pod "7928d319-52af-43dd-bdcd-d3e4893e0219" (UID: "7928d319-52af-43dd-bdcd-d3e4893e0219"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 07:35:03.724009 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:35:03.723970 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7928d319-52af-43dd-bdcd-d3e4893e0219-service-ca" (OuterVolumeSpecName: "service-ca") pod "7928d319-52af-43dd-bdcd-d3e4893e0219" (UID: "7928d319-52af-43dd-bdcd-d3e4893e0219"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 07:35:03.724127 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:35:03.724100 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7928d319-52af-43dd-bdcd-d3e4893e0219-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "7928d319-52af-43dd-bdcd-d3e4893e0219" (UID: "7928d319-52af-43dd-bdcd-d3e4893e0219"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 07:35:03.724419 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:35:03.724394 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7928d319-52af-43dd-bdcd-d3e4893e0219-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "7928d319-52af-43dd-bdcd-d3e4893e0219" (UID: "7928d319-52af-43dd-bdcd-d3e4893e0219"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 07:35:03.726156 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:35:03.726125 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7928d319-52af-43dd-bdcd-d3e4893e0219-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "7928d319-52af-43dd-bdcd-d3e4893e0219" (UID: "7928d319-52af-43dd-bdcd-d3e4893e0219"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 07:35:03.726156 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:35:03.726146 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7928d319-52af-43dd-bdcd-d3e4893e0219-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "7928d319-52af-43dd-bdcd-d3e4893e0219" (UID: "7928d319-52af-43dd-bdcd-d3e4893e0219"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 07:35:03.726274 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:35:03.726207 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7928d319-52af-43dd-bdcd-d3e4893e0219-kube-api-access-mjkwk" (OuterVolumeSpecName: "kube-api-access-mjkwk") pod "7928d319-52af-43dd-bdcd-d3e4893e0219" (UID: "7928d319-52af-43dd-bdcd-d3e4893e0219"). InnerVolumeSpecName "kube-api-access-mjkwk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 07:35:03.824404 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:35:03.824365 2578 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7928d319-52af-43dd-bdcd-d3e4893e0219-console-serving-cert\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Sep 17 07:35:03.824404 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:35:03.824396 2578 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7928d319-52af-43dd-bdcd-d3e4893e0219-trusted-ca-bundle\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Sep 17 07:35:03.824404 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:35:03.824406 2578 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7928d319-52af-43dd-bdcd-d3e4893e0219-console-oauth-config\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Sep 17 07:35:03.824626 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:35:03.824414 2578 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7928d319-52af-43dd-bdcd-d3e4893e0219-oauth-serving-cert\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Sep 17 07:35:03.824626 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:35:03.824424 2578 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7928d319-52af-43dd-bdcd-d3e4893e0219-service-ca\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Sep 17 07:35:03.824626 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:35:03.824432 2578 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7928d319-52af-43dd-bdcd-d3e4893e0219-console-config\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Sep 17 07:35:03.824626 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:35:03.824440 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjkwk\" (UniqueName: \"kubernetes.io/projected/7928d319-52af-43dd-bdcd-d3e4893e0219-kube-api-access-mjkwk\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Sep 17 07:35:04.452579 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:35:04.452540 2578 generic.go:356] "Generic (PLEG): container finished" podID="7928d319-52af-43dd-bdcd-d3e4893e0219" containerID="4325c6074eb0a6b9eeaa2773965902a2ae8d8c3675f057ece5dc7d22f7f65bbb" exitCode=0 Sep 17 07:35:04.452579 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:35:04.452581 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d8cc66756-8kwwx" Sep 17 07:35:04.452777 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:35:04.452622 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d8cc66756-8kwwx" event={"ID":"7928d319-52af-43dd-bdcd-d3e4893e0219","Type":"ContainerDied","Data":"4325c6074eb0a6b9eeaa2773965902a2ae8d8c3675f057ece5dc7d22f7f65bbb"} Sep 17 07:35:04.452777 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:35:04.452673 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d8cc66756-8kwwx" event={"ID":"7928d319-52af-43dd-bdcd-d3e4893e0219","Type":"ContainerDied","Data":"320dadcddef96691b33b331190b49c3b0ab2ba523c25929fc973b0146c93d864"} Sep 17 07:35:04.452777 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:35:04.452694 2578 scope.go:117] "RemoveContainer" containerID="4325c6074eb0a6b9eeaa2773965902a2ae8d8c3675f057ece5dc7d22f7f65bbb" Sep 17 07:35:04.461481 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:35:04.461301 2578 scope.go:117] "RemoveContainer" containerID="4325c6074eb0a6b9eeaa2773965902a2ae8d8c3675f057ece5dc7d22f7f65bbb" Sep 17 07:35:04.461692 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:35:04.461558 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4325c6074eb0a6b9eeaa2773965902a2ae8d8c3675f057ece5dc7d22f7f65bbb\": container with ID starting with 4325c6074eb0a6b9eeaa2773965902a2ae8d8c3675f057ece5dc7d22f7f65bbb not found: ID does not exist" containerID="4325c6074eb0a6b9eeaa2773965902a2ae8d8c3675f057ece5dc7d22f7f65bbb" Sep 17 07:35:04.461692 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:35:04.461581 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4325c6074eb0a6b9eeaa2773965902a2ae8d8c3675f057ece5dc7d22f7f65bbb"} err="failed to get container status \"4325c6074eb0a6b9eeaa2773965902a2ae8d8c3675f057ece5dc7d22f7f65bbb\": rpc error: code = NotFound desc = could not find container \"4325c6074eb0a6b9eeaa2773965902a2ae8d8c3675f057ece5dc7d22f7f65bbb\": container with ID starting with 4325c6074eb0a6b9eeaa2773965902a2ae8d8c3675f057ece5dc7d22f7f65bbb not found: ID does not exist" Sep 17 07:35:04.480741 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:35:04.480716 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5d8cc66756-8kwwx"] Sep 17 07:35:04.494537 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:35:04.494514 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5d8cc66756-8kwwx"] Sep 17 07:35:04.628419 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:35:04.628389 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7928d319-52af-43dd-bdcd-d3e4893e0219" path="/var/lib/kubelet/pods/7928d319-52af-43dd-bdcd-d3e4893e0219/volumes" Sep 17 07:35:28.062479 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:35:28.062440 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-7fbc49b5df-hhkwm"] Sep 17 07:35:28.063205 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:35:28.063183 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7928d319-52af-43dd-bdcd-d3e4893e0219" containerName="console" Sep 17 07:35:28.063293 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:35:28.063210 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7928d319-52af-43dd-bdcd-d3e4893e0219" containerName="console" Sep 17 07:35:28.063387 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:35:28.063374 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="7928d319-52af-43dd-bdcd-d3e4893e0219" containerName="console" Sep 17 07:35:28.073060 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:35:28.073031 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-hhkwm" Sep 17 07:35:28.079167 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:35:28.079135 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-offline-tls\"" Sep 17 07:35:28.079167 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:35:28.079149 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-intra-comm\"" Sep 17 07:35:28.079354 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:35:28.079210 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\"" Sep 17 07:35:28.079354 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:35:28.079239 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-online-tls\"" Sep 17 07:35:28.079354 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:35:28.079287 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\"" Sep 17 07:35:28.079557 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:35:28.079538 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-wvvwp\"" Sep 17 07:35:28.079827 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:35:28.079808 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-ui-tls\"" Sep 17 07:35:28.079935 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:35:28.079841 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-registry-tls\"" Sep 17 07:35:28.086530 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:35:28.086510 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-7fbc49b5df-hhkwm"] Sep 17 07:35:28.210368 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:35:28.210333 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/4ea2b430-adec-4cb3-94ed-12b3e3626d11-registry-tls\") pod \"feast-simple-feast-setup-7fbc49b5df-hhkwm\" (UID: \"4ea2b430-adec-4cb3-94ed-12b3e3626d11\") " pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-hhkwm" Sep 17 07:35:28.210520 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:35:28.210378 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gzp7\" (UniqueName: \"kubernetes.io/projected/4ea2b430-adec-4cb3-94ed-12b3e3626d11-kube-api-access-5gzp7\") pod \"feast-simple-feast-setup-7fbc49b5df-hhkwm\" (UID: \"4ea2b430-adec-4cb3-94ed-12b3e3626d11\") " pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-hhkwm" Sep 17 07:35:28.210520 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:35:28.210435 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/4ea2b430-adec-4cb3-94ed-12b3e3626d11-online-tls\") pod \"feast-simple-feast-setup-7fbc49b5df-hhkwm\" (UID: \"4ea2b430-adec-4cb3-94ed-12b3e3626d11\") " pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-hhkwm" Sep 17 07:35:28.210520 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:35:28.210458 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/4ea2b430-adec-4cb3-94ed-12b3e3626d11-ui-tls\") pod \"feast-simple-feast-setup-7fbc49b5df-hhkwm\" (UID: \"4ea2b430-adec-4cb3-94ed-12b3e3626d11\") " pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-hhkwm" Sep 17 07:35:28.210630 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:35:28.210539 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/4ea2b430-adec-4cb3-94ed-12b3e3626d11-offline-tls\") pod \"feast-simple-feast-setup-7fbc49b5df-hhkwm\" (UID: \"4ea2b430-adec-4cb3-94ed-12b3e3626d11\") " pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-hhkwm" Sep 17 07:35:28.210630 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:35:28.210573 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/4ea2b430-adec-4cb3-94ed-12b3e3626d11-feast-data\") pod \"feast-simple-feast-setup-7fbc49b5df-hhkwm\" (UID: \"4ea2b430-adec-4cb3-94ed-12b3e3626d11\") " pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-hhkwm" Sep 17 07:35:28.311737 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:35:28.311700 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/4ea2b430-adec-4cb3-94ed-12b3e3626d11-offline-tls\") pod \"feast-simple-feast-setup-7fbc49b5df-hhkwm\" (UID: \"4ea2b430-adec-4cb3-94ed-12b3e3626d11\") " pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-hhkwm" Sep 17 07:35:28.311737 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:35:28.311740 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/4ea2b430-adec-4cb3-94ed-12b3e3626d11-feast-data\") pod \"feast-simple-feast-setup-7fbc49b5df-hhkwm\" (UID: \"4ea2b430-adec-4cb3-94ed-12b3e3626d11\") " pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-hhkwm" Sep 17 07:35:28.311982 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:35:28.311789 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/4ea2b430-adec-4cb3-94ed-12b3e3626d11-registry-tls\") pod \"feast-simple-feast-setup-7fbc49b5df-hhkwm\" (UID: \"4ea2b430-adec-4cb3-94ed-12b3e3626d11\") " pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-hhkwm" Sep 17 07:35:28.311982 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:35:28.311911 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gzp7\" (UniqueName: \"kubernetes.io/projected/4ea2b430-adec-4cb3-94ed-12b3e3626d11-kube-api-access-5gzp7\") pod \"feast-simple-feast-setup-7fbc49b5df-hhkwm\" (UID: \"4ea2b430-adec-4cb3-94ed-12b3e3626d11\") " pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-hhkwm" Sep 17 07:35:28.311982 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:35:28.311975 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/4ea2b430-adec-4cb3-94ed-12b3e3626d11-online-tls\") pod \"feast-simple-feast-setup-7fbc49b5df-hhkwm\" (UID: \"4ea2b430-adec-4cb3-94ed-12b3e3626d11\") " pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-hhkwm" Sep 17 07:35:28.312135 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:35:28.312001 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/4ea2b430-adec-4cb3-94ed-12b3e3626d11-ui-tls\") pod \"feast-simple-feast-setup-7fbc49b5df-hhkwm\" (UID: \"4ea2b430-adec-4cb3-94ed-12b3e3626d11\") " pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-hhkwm" Sep 17 07:35:28.312256 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:35:28.312234 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/4ea2b430-adec-4cb3-94ed-12b3e3626d11-feast-data\") pod \"feast-simple-feast-setup-7fbc49b5df-hhkwm\" (UID: \"4ea2b430-adec-4cb3-94ed-12b3e3626d11\") " pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-hhkwm" Sep 17 07:35:28.314359 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:35:28.314302 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/4ea2b430-adec-4cb3-94ed-12b3e3626d11-offline-tls\") pod \"feast-simple-feast-setup-7fbc49b5df-hhkwm\" (UID: \"4ea2b430-adec-4cb3-94ed-12b3e3626d11\") " pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-hhkwm" Sep 17 07:35:28.314359 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:35:28.314321 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/4ea2b430-adec-4cb3-94ed-12b3e3626d11-online-tls\") pod \"feast-simple-feast-setup-7fbc49b5df-hhkwm\" (UID: \"4ea2b430-adec-4cb3-94ed-12b3e3626d11\") " pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-hhkwm" Sep 17 07:35:28.314482 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:35:28.314367 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/4ea2b430-adec-4cb3-94ed-12b3e3626d11-registry-tls\") pod \"feast-simple-feast-setup-7fbc49b5df-hhkwm\" (UID: \"4ea2b430-adec-4cb3-94ed-12b3e3626d11\") " pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-hhkwm" Sep 17 07:35:28.314482 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:35:28.314415 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/4ea2b430-adec-4cb3-94ed-12b3e3626d11-ui-tls\") pod \"feast-simple-feast-setup-7fbc49b5df-hhkwm\" (UID: \"4ea2b430-adec-4cb3-94ed-12b3e3626d11\") " pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-hhkwm" Sep 17 07:35:28.325056 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:35:28.325030 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gzp7\" (UniqueName: \"kubernetes.io/projected/4ea2b430-adec-4cb3-94ed-12b3e3626d11-kube-api-access-5gzp7\") pod \"feast-simple-feast-setup-7fbc49b5df-hhkwm\" (UID: \"4ea2b430-adec-4cb3-94ed-12b3e3626d11\") " pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-hhkwm" Sep 17 07:35:28.393332 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:35:28.393304 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-hhkwm" Sep 17 07:35:28.675471 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:35:28.675447 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-7fbc49b5df-hhkwm"] Sep 17 07:35:28.679097 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:35:28.678996 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4ea2b430_adec_4cb3_94ed_12b3e3626d11.slice/crio-677466e13e475048399333bdde1736f5b435fa6459c9747aa91850031f5fc940 WatchSource:0}: Error finding container 677466e13e475048399333bdde1736f5b435fa6459c9747aa91850031f5fc940: Status 404 returned error can't find the container with id 677466e13e475048399333bdde1736f5b435fa6459c9747aa91850031f5fc940 Sep 17 07:35:29.550996 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:35:29.550916 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-hhkwm" event={"ID":"4ea2b430-adec-4cb3-94ed-12b3e3626d11","Type":"ContainerStarted","Data":"677466e13e475048399333bdde1736f5b435fa6459c9747aa91850031f5fc940"} Sep 17 07:35:57.677580 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:35:57.677545 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-hhkwm" event={"ID":"4ea2b430-adec-4cb3-94ed-12b3e3626d11","Type":"ContainerStarted","Data":"82770d059a4f271ced26f664afe8713cfb72baf3f3900bd61f5ead9284f52b2f"} Sep 17 07:36:02.701604 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:36:02.701561 2578 generic.go:356] "Generic (PLEG): container finished" podID="4ea2b430-adec-4cb3-94ed-12b3e3626d11" containerID="82770d059a4f271ced26f664afe8713cfb72baf3f3900bd61f5ead9284f52b2f" exitCode=0 Sep 17 07:36:02.702102 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:36:02.701610 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-hhkwm" event={"ID":"4ea2b430-adec-4cb3-94ed-12b3e3626d11","Type":"ContainerDied","Data":"82770d059a4f271ced26f664afe8713cfb72baf3f3900bd61f5ead9284f52b2f"} Sep 17 07:36:03.708848 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:36:03.708813 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-hhkwm" event={"ID":"4ea2b430-adec-4cb3-94ed-12b3e3626d11","Type":"ContainerStarted","Data":"bde6e5492e29ced6eb89199051bc5b8aa154787b8c61ec6ccb0e13618b2b034f"} Sep 17 07:36:08.730383 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:36:08.730295 2578 generic.go:356] "Generic (PLEG): container finished" podID="4ea2b430-adec-4cb3-94ed-12b3e3626d11" containerID="bde6e5492e29ced6eb89199051bc5b8aa154787b8c61ec6ccb0e13618b2b034f" exitCode=0 Sep 17 07:36:08.730383 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:36:08.730337 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-hhkwm" event={"ID":"4ea2b430-adec-4cb3-94ed-12b3e3626d11","Type":"ContainerDied","Data":"bde6e5492e29ced6eb89199051bc5b8aa154787b8c61ec6ccb0e13618b2b034f"} Sep 17 07:36:08.746786 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:36:08.746764 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-intra-comm\"" Sep 17 07:36:09.740390 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:36:09.740352 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-hhkwm" event={"ID":"4ea2b430-adec-4cb3-94ed-12b3e3626d11","Type":"ContainerStarted","Data":"bfd3ea7f32bb0a8b8341ff3b1d2128816f7674c871d4025749dc2d034965b3d7"} Sep 17 07:36:09.740881 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:36:09.740412 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-hhkwm" event={"ID":"4ea2b430-adec-4cb3-94ed-12b3e3626d11","Type":"ContainerStarted","Data":"23761a4d1f372a38bd2478744254f1850e5773e16072906d0065cf60f1cb2d32"} Sep 17 07:36:09.740881 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:36:09.740432 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-hhkwm" event={"ID":"4ea2b430-adec-4cb3-94ed-12b3e3626d11","Type":"ContainerStarted","Data":"1dab51b681aceb2a14df9142fb60a6c1ff0a8a4bb5e95d0662726097af7ce8bd"} Sep 17 07:36:09.740881 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:36:09.740449 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-hhkwm" event={"ID":"4ea2b430-adec-4cb3-94ed-12b3e3626d11","Type":"ContainerStarted","Data":"32f4e92cce74952f71a93994f3c1e8db1719939ee818a9d7000c497bbd2d2911"} Sep 17 07:36:09.782631 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:36:09.782569 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-hhkwm" podStartSLOduration=12.972210414 podStartE2EDuration="41.78254693s" totalImagesPullingTime="28.810336516s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 07:35:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 07:36:09.782150922 +0000 UTC m=+513.663185273" watchObservedRunningTime="2026-09-17 07:36:09.78254693 +0000 UTC m=+513.663581290" Sep 17 07:36:10.393881 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:36:10.393824 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-hhkwm" Sep 17 07:36:10.394086 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:36:10.393917 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-hhkwm" Sep 17 07:36:10.394086 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:36:10.393937 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-hhkwm" Sep 17 07:36:10.394086 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:36:10.393955 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-hhkwm" Sep 17 07:36:10.395889 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:36:10.395823 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-hhkwm" podUID="4ea2b430-adec-4cb3-94ed-12b3e3626d11" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.27:8816: connect: connection refused" Sep 17 07:36:10.395889 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:36:10.395827 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-hhkwm" podUID="4ea2b430-adec-4cb3-94ed-12b3e3626d11" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.27:6571: connect: connection refused" Sep 17 07:36:10.396088 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:36:10.395906 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-hhkwm" podUID="4ea2b430-adec-4cb3-94ed-12b3e3626d11" containerName="online" probeResult="failure" output="Get \"https://10.132.0.27:6567/health\": dial tcp 10.132.0.27:6567: connect: connection refused" Sep 17 07:36:10.396088 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:36:10.395960 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-hhkwm" podUID="4ea2b430-adec-4cb3-94ed-12b3e3626d11" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.27:8443: connect: connection refused" Sep 17 07:36:13.394342 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:36:13.394283 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-hhkwm" podUID="4ea2b430-adec-4cb3-94ed-12b3e3626d11" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.27:8816: connect: connection refused" Sep 17 07:36:13.394826 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:36:13.394283 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-hhkwm" podUID="4ea2b430-adec-4cb3-94ed-12b3e3626d11" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.27:6571: connect: connection refused" Sep 17 07:36:13.394826 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:36:13.394283 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-hhkwm" podUID="4ea2b430-adec-4cb3-94ed-12b3e3626d11" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.27:8443: connect: connection refused" Sep 17 07:36:13.394826 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:36:13.394576 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-hhkwm" podUID="4ea2b430-adec-4cb3-94ed-12b3e3626d11" containerName="online" probeResult="failure" output="Get \"https://10.132.0.27:6567/health\": dial tcp 10.132.0.27:6567: connect: connection refused" Sep 17 07:36:16.394106 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:36:16.394063 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-hhkwm" podUID="4ea2b430-adec-4cb3-94ed-12b3e3626d11" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.27:8443: connect: connection refused" Sep 17 07:36:16.394552 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:36:16.394193 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-hhkwm" podUID="4ea2b430-adec-4cb3-94ed-12b3e3626d11" containerName="online" probeResult="failure" output="Get \"https://10.132.0.27:6567/health\": dial tcp 10.132.0.27:6567: connect: connection refused" Sep 17 07:36:16.395026 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:36:16.395002 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-hhkwm" Sep 17 07:36:16.395157 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:36:16.395117 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-hhkwm" Sep 17 07:36:16.395783 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:36:16.395758 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-hhkwm" Sep 17 07:36:16.395914 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:36:16.395802 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-hhkwm" Sep 17 07:36:16.396026 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:36:16.396009 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-hhkwm" Sep 17 07:36:16.396347 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:36:16.396327 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-hhkwm" Sep 17 07:36:18.394236 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:36:18.394204 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-hhkwm" Sep 17 07:36:18.394600 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:36:18.394253 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-hhkwm" Sep 17 07:36:19.395273 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:36:19.395243 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-hhkwm" Sep 17 07:36:19.400424 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:36:19.400398 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-hhkwm" Sep 17 07:36:19.793017 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:36:19.792947 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-hhkwm" Sep 17 07:36:19.795511 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:36:19.795489 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-hhkwm" Sep 17 07:37:22.474468 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:37:22.474435 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-7fbc49b5df-hhkwm"] Sep 17 07:37:22.475038 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:37:22.474701 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-hhkwm" podUID="4ea2b430-adec-4cb3-94ed-12b3e3626d11" containerName="registry" containerID="cri-o://32f4e92cce74952f71a93994f3c1e8db1719939ee818a9d7000c497bbd2d2911" gracePeriod=30 Sep 17 07:37:22.475038 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:37:22.474784 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-hhkwm" podUID="4ea2b430-adec-4cb3-94ed-12b3e3626d11" containerName="offline" containerID="cri-o://23761a4d1f372a38bd2478744254f1850e5773e16072906d0065cf60f1cb2d32" gracePeriod=30 Sep 17 07:37:22.475038 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:37:22.474827 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-hhkwm" podUID="4ea2b430-adec-4cb3-94ed-12b3e3626d11" containerName="online" containerID="cri-o://1dab51b681aceb2a14df9142fb60a6c1ff0a8a4bb5e95d0662726097af7ce8bd" gracePeriod=30 Sep 17 07:37:22.475038 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:37:22.474784 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-hhkwm" podUID="4ea2b430-adec-4cb3-94ed-12b3e3626d11" containerName="ui" containerID="cri-o://bfd3ea7f32bb0a8b8341ff3b1d2128816f7674c871d4025749dc2d034965b3d7" gracePeriod=30 Sep 17 07:37:23.315158 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:37:23.315126 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-7fbc49b5df-sllcc"] Sep 17 07:37:23.318730 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:37:23.318715 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-sllcc" Sep 17 07:37:23.322703 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:37:23.322685 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-8q8xq\"" Sep 17 07:37:23.343416 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:37:23.343388 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-7fbc49b5df-sllcc"] Sep 17 07:37:23.469751 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:37:23.469724 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/1feb527a-10d8-4bc8-be30-52a9a4634c33-online-tls\") pod \"feast-simple-feast-setup-7fbc49b5df-sllcc\" (UID: \"1feb527a-10d8-4bc8-be30-52a9a4634c33\") " pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-sllcc" Sep 17 07:37:23.469751 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:37:23.469751 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/1feb527a-10d8-4bc8-be30-52a9a4634c33-feast-data\") pod \"feast-simple-feast-setup-7fbc49b5df-sllcc\" (UID: \"1feb527a-10d8-4bc8-be30-52a9a4634c33\") " pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-sllcc" Sep 17 07:37:23.469961 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:37:23.469800 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/1feb527a-10d8-4bc8-be30-52a9a4634c33-registry-tls\") pod \"feast-simple-feast-setup-7fbc49b5df-sllcc\" (UID: \"1feb527a-10d8-4bc8-be30-52a9a4634c33\") " pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-sllcc" Sep 17 07:37:23.469961 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:37:23.469815 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/1feb527a-10d8-4bc8-be30-52a9a4634c33-ui-tls\") pod \"feast-simple-feast-setup-7fbc49b5df-sllcc\" (UID: \"1feb527a-10d8-4bc8-be30-52a9a4634c33\") " pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-sllcc" Sep 17 07:37:23.469961 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:37:23.469840 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxqst\" (UniqueName: \"kubernetes.io/projected/1feb527a-10d8-4bc8-be30-52a9a4634c33-kube-api-access-pxqst\") pod \"feast-simple-feast-setup-7fbc49b5df-sllcc\" (UID: \"1feb527a-10d8-4bc8-be30-52a9a4634c33\") " pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-sllcc" Sep 17 07:37:23.469961 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:37:23.469869 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/1feb527a-10d8-4bc8-be30-52a9a4634c33-offline-tls\") pod \"feast-simple-feast-setup-7fbc49b5df-sllcc\" (UID: \"1feb527a-10d8-4bc8-be30-52a9a4634c33\") " pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-sllcc" Sep 17 07:37:23.570692 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:37:23.570623 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/1feb527a-10d8-4bc8-be30-52a9a4634c33-registry-tls\") pod \"feast-simple-feast-setup-7fbc49b5df-sllcc\" (UID: \"1feb527a-10d8-4bc8-be30-52a9a4634c33\") " pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-sllcc" Sep 17 07:37:23.570692 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:37:23.570650 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/1feb527a-10d8-4bc8-be30-52a9a4634c33-ui-tls\") pod \"feast-simple-feast-setup-7fbc49b5df-sllcc\" (UID: \"1feb527a-10d8-4bc8-be30-52a9a4634c33\") " pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-sllcc" Sep 17 07:37:23.570692 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:37:23.570668 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxqst\" (UniqueName: \"kubernetes.io/projected/1feb527a-10d8-4bc8-be30-52a9a4634c33-kube-api-access-pxqst\") pod \"feast-simple-feast-setup-7fbc49b5df-sllcc\" (UID: \"1feb527a-10d8-4bc8-be30-52a9a4634c33\") " pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-sllcc" Sep 17 07:37:23.570692 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:37:23.570685 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/1feb527a-10d8-4bc8-be30-52a9a4634c33-offline-tls\") pod \"feast-simple-feast-setup-7fbc49b5df-sllcc\" (UID: \"1feb527a-10d8-4bc8-be30-52a9a4634c33\") " pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-sllcc" Sep 17 07:37:23.571222 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:37:23.570734 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/1feb527a-10d8-4bc8-be30-52a9a4634c33-online-tls\") pod \"feast-simple-feast-setup-7fbc49b5df-sllcc\" (UID: \"1feb527a-10d8-4bc8-be30-52a9a4634c33\") " pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-sllcc" Sep 17 07:37:23.571222 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:37:23.570757 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/1feb527a-10d8-4bc8-be30-52a9a4634c33-feast-data\") pod \"feast-simple-feast-setup-7fbc49b5df-sllcc\" (UID: \"1feb527a-10d8-4bc8-be30-52a9a4634c33\") " pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-sllcc" Sep 17 07:37:23.571222 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:37:23.571200 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/1feb527a-10d8-4bc8-be30-52a9a4634c33-feast-data\") pod \"feast-simple-feast-setup-7fbc49b5df-sllcc\" (UID: \"1feb527a-10d8-4bc8-be30-52a9a4634c33\") " pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-sllcc" Sep 17 07:37:23.573290 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:37:23.573262 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/1feb527a-10d8-4bc8-be30-52a9a4634c33-offline-tls\") pod \"feast-simple-feast-setup-7fbc49b5df-sllcc\" (UID: \"1feb527a-10d8-4bc8-be30-52a9a4634c33\") " pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-sllcc" Sep 17 07:37:23.573290 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:37:23.573272 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/1feb527a-10d8-4bc8-be30-52a9a4634c33-ui-tls\") pod \"feast-simple-feast-setup-7fbc49b5df-sllcc\" (UID: \"1feb527a-10d8-4bc8-be30-52a9a4634c33\") " pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-sllcc" Sep 17 07:37:23.573457 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:37:23.573343 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/1feb527a-10d8-4bc8-be30-52a9a4634c33-online-tls\") pod \"feast-simple-feast-setup-7fbc49b5df-sllcc\" (UID: \"1feb527a-10d8-4bc8-be30-52a9a4634c33\") " pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-sllcc" Sep 17 07:37:23.573457 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:37:23.573371 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/1feb527a-10d8-4bc8-be30-52a9a4634c33-registry-tls\") pod \"feast-simple-feast-setup-7fbc49b5df-sllcc\" (UID: \"1feb527a-10d8-4bc8-be30-52a9a4634c33\") " pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-sllcc" Sep 17 07:37:23.582308 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:37:23.582285 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxqst\" (UniqueName: \"kubernetes.io/projected/1feb527a-10d8-4bc8-be30-52a9a4634c33-kube-api-access-pxqst\") pod \"feast-simple-feast-setup-7fbc49b5df-sllcc\" (UID: \"1feb527a-10d8-4bc8-be30-52a9a4634c33\") " pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-sllcc" Sep 17 07:37:23.627497 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:37:23.627459 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-sllcc" Sep 17 07:37:23.936719 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:37:23.936679 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-7fbc49b5df-sllcc"] Sep 17 07:37:23.937241 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:37:23.937214 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1feb527a_10d8_4bc8_be30_52a9a4634c33.slice/crio-d0fab2448fafbe46b24cc9466a789a854faa53d06291c36c8f3bc7abee73abd6 WatchSource:0}: Error finding container d0fab2448fafbe46b24cc9466a789a854faa53d06291c36c8f3bc7abee73abd6: Status 404 returned error can't find the container with id d0fab2448fafbe46b24cc9466a789a854faa53d06291c36c8f3bc7abee73abd6 Sep 17 07:37:24.051087 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:37:24.051056 2578 generic.go:356] "Generic (PLEG): container finished" podID="4ea2b430-adec-4cb3-94ed-12b3e3626d11" containerID="bfd3ea7f32bb0a8b8341ff3b1d2128816f7674c871d4025749dc2d034965b3d7" exitCode=0 Sep 17 07:37:24.051087 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:37:24.051080 2578 generic.go:356] "Generic (PLEG): container finished" podID="4ea2b430-adec-4cb3-94ed-12b3e3626d11" containerID="1dab51b681aceb2a14df9142fb60a6c1ff0a8a4bb5e95d0662726097af7ce8bd" exitCode=0 Sep 17 07:37:24.051281 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:37:24.051124 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-hhkwm" event={"ID":"4ea2b430-adec-4cb3-94ed-12b3e3626d11","Type":"ContainerDied","Data":"bfd3ea7f32bb0a8b8341ff3b1d2128816f7674c871d4025749dc2d034965b3d7"} Sep 17 07:37:24.051281 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:37:24.051164 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-hhkwm" event={"ID":"4ea2b430-adec-4cb3-94ed-12b3e3626d11","Type":"ContainerDied","Data":"1dab51b681aceb2a14df9142fb60a6c1ff0a8a4bb5e95d0662726097af7ce8bd"} Sep 17 07:37:24.052546 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:37:24.052522 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-sllcc" event={"ID":"1feb527a-10d8-4bc8-be30-52a9a4634c33","Type":"ContainerStarted","Data":"549416d35482a8540070cb0eb67659efb9e4e92c4f44286b0d8964720329d750"} Sep 17 07:37:24.052654 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:37:24.052563 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-sllcc" event={"ID":"1feb527a-10d8-4bc8-be30-52a9a4634c33","Type":"ContainerStarted","Data":"d0fab2448fafbe46b24cc9466a789a854faa53d06291c36c8f3bc7abee73abd6"} Sep 17 07:37:29.074466 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:37:29.074384 2578 generic.go:356] "Generic (PLEG): container finished" podID="1feb527a-10d8-4bc8-be30-52a9a4634c33" containerID="549416d35482a8540070cb0eb67659efb9e4e92c4f44286b0d8964720329d750" exitCode=0 Sep 17 07:37:29.074916 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:37:29.074464 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-sllcc" event={"ID":"1feb527a-10d8-4bc8-be30-52a9a4634c33","Type":"ContainerDied","Data":"549416d35482a8540070cb0eb67659efb9e4e92c4f44286b0d8964720329d750"} Sep 17 07:37:29.792613 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:37:29.792564 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-hhkwm" podUID="4ea2b430-adec-4cb3-94ed-12b3e3626d11" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.27:8443: connect: connection refused" Sep 17 07:37:29.792772 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:37:29.792587 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-hhkwm" podUID="4ea2b430-adec-4cb3-94ed-12b3e3626d11" containerName="online" probeResult="failure" output="Get \"https://10.132.0.27:6567/health\": dial tcp 10.132.0.27:6567: connect: connection refused" Sep 17 07:37:30.081267 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:37:30.081192 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-sllcc" event={"ID":"1feb527a-10d8-4bc8-be30-52a9a4634c33","Type":"ContainerStarted","Data":"af0a84299eb5b4e5924777dca47d3e315d31a9567d26e793c27fd6ba666c3a4a"} Sep 17 07:37:35.104142 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:37:35.104064 2578 generic.go:356] "Generic (PLEG): container finished" podID="1feb527a-10d8-4bc8-be30-52a9a4634c33" containerID="af0a84299eb5b4e5924777dca47d3e315d31a9567d26e793c27fd6ba666c3a4a" exitCode=0 Sep 17 07:37:35.104142 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:37:35.104099 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-sllcc" event={"ID":"1feb527a-10d8-4bc8-be30-52a9a4634c33","Type":"ContainerDied","Data":"af0a84299eb5b4e5924777dca47d3e315d31a9567d26e793c27fd6ba666c3a4a"} Sep 17 07:37:36.115378 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:37:36.115334 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-sllcc" event={"ID":"1feb527a-10d8-4bc8-be30-52a9a4634c33","Type":"ContainerStarted","Data":"422adaa09dd13d416aa68bc192459f8c7e521870ed33b34ad6560953a1378b94"} Sep 17 07:37:36.115849 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:37:36.115401 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-sllcc" event={"ID":"1feb527a-10d8-4bc8-be30-52a9a4634c33","Type":"ContainerStarted","Data":"b5dff01f644fcde1c0230dfc52529290b7e809b8ed5b740cec4e022423549cb9"} Sep 17 07:37:36.115849 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:37:36.115423 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-sllcc" event={"ID":"1feb527a-10d8-4bc8-be30-52a9a4634c33","Type":"ContainerStarted","Data":"b4bb042dd05c3651e1163c94999ebab47df20f05f61e5a36ab8b3056dbc2ec38"} Sep 17 07:37:36.115849 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:37:36.115440 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-sllcc" event={"ID":"1feb527a-10d8-4bc8-be30-52a9a4634c33","Type":"ContainerStarted","Data":"2309e2fa5d7c715a45900d9fa64c731a066425b5bb3558c09c935da79dd962a7"} Sep 17 07:37:36.146976 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:37:36.146918 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-sllcc" podStartSLOduration=13.146897094 podStartE2EDuration="13.146897094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 07:37:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 07:37:36.143739458 +0000 UTC m=+600.024773818" watchObservedRunningTime="2026-09-17 07:37:36.146897094 +0000 UTC m=+600.027931453" Sep 17 07:37:38.630532 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:37:38.630477 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-sllcc" podUID="1feb527a-10d8-4bc8-be30-52a9a4634c33" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.28:8443: connect: connection refused" Sep 17 07:37:38.631040 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:37:38.630525 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-sllcc" podUID="1feb527a-10d8-4bc8-be30-52a9a4634c33" containerName="online" probeResult="failure" output="Get \"https://10.132.0.28:6567/health\": dial tcp 10.132.0.28:6567: connect: connection refused" Sep 17 07:37:38.631040 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:37:38.630552 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-sllcc" podUID="1feb527a-10d8-4bc8-be30-52a9a4634c33" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.28:6571: connect: connection refused" Sep 17 07:37:38.631040 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:37:38.630710 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-sllcc" podUID="1feb527a-10d8-4bc8-be30-52a9a4634c33" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.28:8816: connect: connection refused" Sep 17 07:37:38.632233 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:37:38.632207 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-sllcc" Sep 17 07:37:38.632356 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:37:38.632255 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-sllcc" Sep 17 07:37:38.632356 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:37:38.632276 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-sllcc" Sep 17 07:37:38.632356 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:37:38.632293 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-sllcc" Sep 17 07:37:39.793073 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:37:39.793019 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-hhkwm" podUID="4ea2b430-adec-4cb3-94ed-12b3e3626d11" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.27:8443: connect: connection refused" Sep 17 07:37:39.793578 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:37:39.793069 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-hhkwm" podUID="4ea2b430-adec-4cb3-94ed-12b3e3626d11" containerName="online" probeResult="failure" output="Get \"https://10.132.0.27:6567/health\": dial tcp 10.132.0.27:6567: connect: connection refused" Sep 17 07:37:41.628439 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:37:41.628390 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-sllcc" podUID="1feb527a-10d8-4bc8-be30-52a9a4634c33" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.28:8816: connect: connection refused" Sep 17 07:37:41.628439 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:37:41.628403 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-sllcc" podUID="1feb527a-10d8-4bc8-be30-52a9a4634c33" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.28:8443: connect: connection refused" Sep 17 07:37:41.629009 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:37:41.628390 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-sllcc" podUID="1feb527a-10d8-4bc8-be30-52a9a4634c33" containerName="online" probeResult="failure" output="Get \"https://10.132.0.28:6567/health\": dial tcp 10.132.0.28:6567: connect: connection refused" Sep 17 07:37:41.629009 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:37:41.628914 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-sllcc" podUID="1feb527a-10d8-4bc8-be30-52a9a4634c33" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.28:6571: connect: connection refused" Sep 17 07:37:43.627815 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:37:43.627771 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-sllcc" Sep 17 07:37:43.628216 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:37:43.627840 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-sllcc" Sep 17 07:37:43.628216 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:37:43.627885 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-sllcc" Sep 17 07:37:43.628216 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:37:43.627903 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-sllcc" Sep 17 07:37:44.629291 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:37:44.629263 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-sllcc" Sep 17 07:37:44.629663 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:37:44.629360 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-sllcc" Sep 17 07:37:44.629663 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:37:44.629580 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-sllcc" Sep 17 07:37:44.633909 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:37:44.633896 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-sllcc" Sep 17 07:37:45.170055 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:37:45.170022 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-sllcc" Sep 17 07:37:45.170243 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:37:45.170159 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-sllcc" Sep 17 07:37:45.170352 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:37:45.170336 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-sllcc" Sep 17 07:37:45.172903 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:37:45.172885 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-sllcc" Sep 17 07:37:49.792944 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:37:49.792891 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-hhkwm" podUID="4ea2b430-adec-4cb3-94ed-12b3e3626d11" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.27:8443: connect: connection refused" Sep 17 07:37:49.793388 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:37:49.792960 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-hhkwm" podUID="4ea2b430-adec-4cb3-94ed-12b3e3626d11" containerName="online" probeResult="failure" output="Get \"https://10.132.0.27:6567/health\": dial tcp 10.132.0.27:6567: connect: connection refused" Sep 17 07:37:49.793388 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:37:49.793031 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-hhkwm" Sep 17 07:37:49.793388 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:37:49.793090 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-hhkwm" Sep 17 07:37:53.114623 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:37:53.114594 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-hhkwm" Sep 17 07:37:53.205144 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:37:53.205112 2578 generic.go:356] "Generic (PLEG): container finished" podID="4ea2b430-adec-4cb3-94ed-12b3e3626d11" containerID="23761a4d1f372a38bd2478744254f1850e5773e16072906d0065cf60f1cb2d32" exitCode=137 Sep 17 07:37:53.205144 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:37:53.205137 2578 generic.go:356] "Generic (PLEG): container finished" podID="4ea2b430-adec-4cb3-94ed-12b3e3626d11" containerID="32f4e92cce74952f71a93994f3c1e8db1719939ee818a9d7000c497bbd2d2911" exitCode=137 Sep 17 07:37:53.205404 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:37:53.205185 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-hhkwm" Sep 17 07:37:53.205404 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:37:53.205184 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-hhkwm" event={"ID":"4ea2b430-adec-4cb3-94ed-12b3e3626d11","Type":"ContainerDied","Data":"23761a4d1f372a38bd2478744254f1850e5773e16072906d0065cf60f1cb2d32"} Sep 17 07:37:53.205404 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:37:53.205235 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-hhkwm" event={"ID":"4ea2b430-adec-4cb3-94ed-12b3e3626d11","Type":"ContainerDied","Data":"32f4e92cce74952f71a93994f3c1e8db1719939ee818a9d7000c497bbd2d2911"} Sep 17 07:37:53.205404 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:37:53.205249 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-hhkwm" event={"ID":"4ea2b430-adec-4cb3-94ed-12b3e3626d11","Type":"ContainerDied","Data":"677466e13e475048399333bdde1736f5b435fa6459c9747aa91850031f5fc940"} Sep 17 07:37:53.205404 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:37:53.205267 2578 scope.go:117] "RemoveContainer" containerID="bfd3ea7f32bb0a8b8341ff3b1d2128816f7674c871d4025749dc2d034965b3d7" Sep 17 07:37:53.213111 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:37:53.213087 2578 scope.go:117] "RemoveContainer" containerID="23761a4d1f372a38bd2478744254f1850e5773e16072906d0065cf60f1cb2d32" Sep 17 07:37:53.215091 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:37:53.215073 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/4ea2b430-adec-4cb3-94ed-12b3e3626d11-feast-data\") pod \"4ea2b430-adec-4cb3-94ed-12b3e3626d11\" (UID: \"4ea2b430-adec-4cb3-94ed-12b3e3626d11\") " Sep 17 07:37:53.215200 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:37:53.215110 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/4ea2b430-adec-4cb3-94ed-12b3e3626d11-offline-tls\") pod \"4ea2b430-adec-4cb3-94ed-12b3e3626d11\" (UID: \"4ea2b430-adec-4cb3-94ed-12b3e3626d11\") " Sep 17 07:37:53.215200 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:37:53.215135 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/4ea2b430-adec-4cb3-94ed-12b3e3626d11-online-tls\") pod \"4ea2b430-adec-4cb3-94ed-12b3e3626d11\" (UID: \"4ea2b430-adec-4cb3-94ed-12b3e3626d11\") " Sep 17 07:37:53.215200 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:37:53.215181 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/4ea2b430-adec-4cb3-94ed-12b3e3626d11-registry-tls\") pod \"4ea2b430-adec-4cb3-94ed-12b3e3626d11\" (UID: \"4ea2b430-adec-4cb3-94ed-12b3e3626d11\") " Sep 17 07:37:53.215444 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:37:53.215211 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/4ea2b430-adec-4cb3-94ed-12b3e3626d11-ui-tls\") pod \"4ea2b430-adec-4cb3-94ed-12b3e3626d11\" (UID: \"4ea2b430-adec-4cb3-94ed-12b3e3626d11\") " Sep 17 07:37:53.215444 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:37:53.215244 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5gzp7\" (UniqueName: \"kubernetes.io/projected/4ea2b430-adec-4cb3-94ed-12b3e3626d11-kube-api-access-5gzp7\") pod \"4ea2b430-adec-4cb3-94ed-12b3e3626d11\" (UID: \"4ea2b430-adec-4cb3-94ed-12b3e3626d11\") " Sep 17 07:37:53.215692 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:37:53.215667 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ea2b430-adec-4cb3-94ed-12b3e3626d11-feast-data" (OuterVolumeSpecName: "feast-data") pod "4ea2b430-adec-4cb3-94ed-12b3e3626d11" (UID: "4ea2b430-adec-4cb3-94ed-12b3e3626d11"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 07:37:53.217893 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:37:53.217737 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ea2b430-adec-4cb3-94ed-12b3e3626d11-online-tls" (OuterVolumeSpecName: "online-tls") pod "4ea2b430-adec-4cb3-94ed-12b3e3626d11" (UID: "4ea2b430-adec-4cb3-94ed-12b3e3626d11"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 07:37:53.217893 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:37:53.217756 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ea2b430-adec-4cb3-94ed-12b3e3626d11-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "4ea2b430-adec-4cb3-94ed-12b3e3626d11" (UID: "4ea2b430-adec-4cb3-94ed-12b3e3626d11"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 07:37:53.217893 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:37:53.217826 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ea2b430-adec-4cb3-94ed-12b3e3626d11-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "4ea2b430-adec-4cb3-94ed-12b3e3626d11" (UID: "4ea2b430-adec-4cb3-94ed-12b3e3626d11"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 07:37:53.217893 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:37:53.217822 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ea2b430-adec-4cb3-94ed-12b3e3626d11-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "4ea2b430-adec-4cb3-94ed-12b3e3626d11" (UID: "4ea2b430-adec-4cb3-94ed-12b3e3626d11"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 07:37:53.217893 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:37:53.217844 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ea2b430-adec-4cb3-94ed-12b3e3626d11-kube-api-access-5gzp7" (OuterVolumeSpecName: "kube-api-access-5gzp7") pod "4ea2b430-adec-4cb3-94ed-12b3e3626d11" (UID: "4ea2b430-adec-4cb3-94ed-12b3e3626d11"). InnerVolumeSpecName "kube-api-access-5gzp7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 07:37:53.221425 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:37:53.221408 2578 scope.go:117] "RemoveContainer" containerID="1dab51b681aceb2a14df9142fb60a6c1ff0a8a4bb5e95d0662726097af7ce8bd" Sep 17 07:37:53.229732 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:37:53.229713 2578 scope.go:117] "RemoveContainer" containerID="32f4e92cce74952f71a93994f3c1e8db1719939ee818a9d7000c497bbd2d2911" Sep 17 07:37:53.236906 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:37:53.236887 2578 scope.go:117] "RemoveContainer" containerID="bde6e5492e29ced6eb89199051bc5b8aa154787b8c61ec6ccb0e13618b2b034f" Sep 17 07:37:53.243398 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:37:53.243380 2578 scope.go:117] "RemoveContainer" containerID="82770d059a4f271ced26f664afe8713cfb72baf3f3900bd61f5ead9284f52b2f" Sep 17 07:37:53.250064 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:37:53.250049 2578 scope.go:117] "RemoveContainer" containerID="bfd3ea7f32bb0a8b8341ff3b1d2128816f7674c871d4025749dc2d034965b3d7" Sep 17 07:37:53.250337 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:37:53.250319 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfd3ea7f32bb0a8b8341ff3b1d2128816f7674c871d4025749dc2d034965b3d7\": container with ID starting with bfd3ea7f32bb0a8b8341ff3b1d2128816f7674c871d4025749dc2d034965b3d7 not found: ID does not exist" containerID="bfd3ea7f32bb0a8b8341ff3b1d2128816f7674c871d4025749dc2d034965b3d7" Sep 17 07:37:53.250404 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:37:53.250349 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfd3ea7f32bb0a8b8341ff3b1d2128816f7674c871d4025749dc2d034965b3d7"} err="failed to get container status \"bfd3ea7f32bb0a8b8341ff3b1d2128816f7674c871d4025749dc2d034965b3d7\": rpc error: code = NotFound desc = could not find container \"bfd3ea7f32bb0a8b8341ff3b1d2128816f7674c871d4025749dc2d034965b3d7\": container with ID starting with bfd3ea7f32bb0a8b8341ff3b1d2128816f7674c871d4025749dc2d034965b3d7 not found: ID does not exist" Sep 17 07:37:53.250404 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:37:53.250375 2578 scope.go:117] "RemoveContainer" containerID="23761a4d1f372a38bd2478744254f1850e5773e16072906d0065cf60f1cb2d32" Sep 17 07:37:53.250635 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:37:53.250619 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23761a4d1f372a38bd2478744254f1850e5773e16072906d0065cf60f1cb2d32\": container with ID starting with 23761a4d1f372a38bd2478744254f1850e5773e16072906d0065cf60f1cb2d32 not found: ID does not exist" containerID="23761a4d1f372a38bd2478744254f1850e5773e16072906d0065cf60f1cb2d32" Sep 17 07:37:53.250676 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:37:53.250642 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23761a4d1f372a38bd2478744254f1850e5773e16072906d0065cf60f1cb2d32"} err="failed to get container status \"23761a4d1f372a38bd2478744254f1850e5773e16072906d0065cf60f1cb2d32\": rpc error: code = NotFound desc = could not find container \"23761a4d1f372a38bd2478744254f1850e5773e16072906d0065cf60f1cb2d32\": container with ID starting with 23761a4d1f372a38bd2478744254f1850e5773e16072906d0065cf60f1cb2d32 not found: ID does not exist" Sep 17 07:37:53.250676 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:37:53.250658 2578 scope.go:117] "RemoveContainer" containerID="1dab51b681aceb2a14df9142fb60a6c1ff0a8a4bb5e95d0662726097af7ce8bd" Sep 17 07:37:53.250896 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:37:53.250855 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1dab51b681aceb2a14df9142fb60a6c1ff0a8a4bb5e95d0662726097af7ce8bd\": container with ID starting with 1dab51b681aceb2a14df9142fb60a6c1ff0a8a4bb5e95d0662726097af7ce8bd not found: ID does not exist" containerID="1dab51b681aceb2a14df9142fb60a6c1ff0a8a4bb5e95d0662726097af7ce8bd" Sep 17 07:37:53.250979 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:37:53.250905 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1dab51b681aceb2a14df9142fb60a6c1ff0a8a4bb5e95d0662726097af7ce8bd"} err="failed to get container status \"1dab51b681aceb2a14df9142fb60a6c1ff0a8a4bb5e95d0662726097af7ce8bd\": rpc error: code = NotFound desc = could not find container \"1dab51b681aceb2a14df9142fb60a6c1ff0a8a4bb5e95d0662726097af7ce8bd\": container with ID starting with 1dab51b681aceb2a14df9142fb60a6c1ff0a8a4bb5e95d0662726097af7ce8bd not found: ID does not exist" Sep 17 07:37:53.250979 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:37:53.250927 2578 scope.go:117] "RemoveContainer" containerID="32f4e92cce74952f71a93994f3c1e8db1719939ee818a9d7000c497bbd2d2911" Sep 17 07:37:53.251161 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:37:53.251145 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32f4e92cce74952f71a93994f3c1e8db1719939ee818a9d7000c497bbd2d2911\": container with ID starting with 32f4e92cce74952f71a93994f3c1e8db1719939ee818a9d7000c497bbd2d2911 not found: ID does not exist" containerID="32f4e92cce74952f71a93994f3c1e8db1719939ee818a9d7000c497bbd2d2911" Sep 17 07:37:53.251203 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:37:53.251164 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32f4e92cce74952f71a93994f3c1e8db1719939ee818a9d7000c497bbd2d2911"} err="failed to get container status \"32f4e92cce74952f71a93994f3c1e8db1719939ee818a9d7000c497bbd2d2911\": rpc error: code = NotFound desc = could not find container \"32f4e92cce74952f71a93994f3c1e8db1719939ee818a9d7000c497bbd2d2911\": container with ID starting with 32f4e92cce74952f71a93994f3c1e8db1719939ee818a9d7000c497bbd2d2911 not found: ID does not exist" Sep 17 07:37:53.251203 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:37:53.251177 2578 scope.go:117] "RemoveContainer" containerID="bde6e5492e29ced6eb89199051bc5b8aa154787b8c61ec6ccb0e13618b2b034f" Sep 17 07:37:53.251418 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:37:53.251403 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bde6e5492e29ced6eb89199051bc5b8aa154787b8c61ec6ccb0e13618b2b034f\": container with ID starting with bde6e5492e29ced6eb89199051bc5b8aa154787b8c61ec6ccb0e13618b2b034f not found: ID does not exist" containerID="bde6e5492e29ced6eb89199051bc5b8aa154787b8c61ec6ccb0e13618b2b034f" Sep 17 07:37:53.251522 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:37:53.251425 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bde6e5492e29ced6eb89199051bc5b8aa154787b8c61ec6ccb0e13618b2b034f"} err="failed to get container status \"bde6e5492e29ced6eb89199051bc5b8aa154787b8c61ec6ccb0e13618b2b034f\": rpc error: code = NotFound desc = could not find container \"bde6e5492e29ced6eb89199051bc5b8aa154787b8c61ec6ccb0e13618b2b034f\": container with ID starting with bde6e5492e29ced6eb89199051bc5b8aa154787b8c61ec6ccb0e13618b2b034f not found: ID does not exist" Sep 17 07:37:53.251522 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:37:53.251444 2578 scope.go:117] "RemoveContainer" containerID="82770d059a4f271ced26f664afe8713cfb72baf3f3900bd61f5ead9284f52b2f" Sep 17 07:37:53.251709 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:37:53.251690 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82770d059a4f271ced26f664afe8713cfb72baf3f3900bd61f5ead9284f52b2f\": container with ID starting with 82770d059a4f271ced26f664afe8713cfb72baf3f3900bd61f5ead9284f52b2f not found: ID does not exist" containerID="82770d059a4f271ced26f664afe8713cfb72baf3f3900bd61f5ead9284f52b2f" Sep 17 07:37:53.251754 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:37:53.251714 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82770d059a4f271ced26f664afe8713cfb72baf3f3900bd61f5ead9284f52b2f"} err="failed to get container status \"82770d059a4f271ced26f664afe8713cfb72baf3f3900bd61f5ead9284f52b2f\": rpc error: code = NotFound desc = could not find container \"82770d059a4f271ced26f664afe8713cfb72baf3f3900bd61f5ead9284f52b2f\": container with ID starting with 82770d059a4f271ced26f664afe8713cfb72baf3f3900bd61f5ead9284f52b2f not found: ID does not exist" Sep 17 07:37:53.251754 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:37:53.251726 2578 scope.go:117] "RemoveContainer" containerID="bfd3ea7f32bb0a8b8341ff3b1d2128816f7674c871d4025749dc2d034965b3d7" Sep 17 07:37:53.251980 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:37:53.251961 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfd3ea7f32bb0a8b8341ff3b1d2128816f7674c871d4025749dc2d034965b3d7"} err="failed to get container status \"bfd3ea7f32bb0a8b8341ff3b1d2128816f7674c871d4025749dc2d034965b3d7\": rpc error: code = NotFound desc = could not find container \"bfd3ea7f32bb0a8b8341ff3b1d2128816f7674c871d4025749dc2d034965b3d7\": container with ID starting with bfd3ea7f32bb0a8b8341ff3b1d2128816f7674c871d4025749dc2d034965b3d7 not found: ID does not exist" Sep 17 07:37:53.252056 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:37:53.251981 2578 scope.go:117] "RemoveContainer" containerID="23761a4d1f372a38bd2478744254f1850e5773e16072906d0065cf60f1cb2d32" Sep 17 07:37:53.252210 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:37:53.252193 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23761a4d1f372a38bd2478744254f1850e5773e16072906d0065cf60f1cb2d32"} err="failed to get container status \"23761a4d1f372a38bd2478744254f1850e5773e16072906d0065cf60f1cb2d32\": rpc error: code = NotFound desc = could not find container \"23761a4d1f372a38bd2478744254f1850e5773e16072906d0065cf60f1cb2d32\": container with ID starting with 23761a4d1f372a38bd2478744254f1850e5773e16072906d0065cf60f1cb2d32 not found: ID does not exist" Sep 17 07:37:53.252274 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:37:53.252211 2578 scope.go:117] "RemoveContainer" containerID="1dab51b681aceb2a14df9142fb60a6c1ff0a8a4bb5e95d0662726097af7ce8bd" Sep 17 07:37:53.252440 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:37:53.252424 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1dab51b681aceb2a14df9142fb60a6c1ff0a8a4bb5e95d0662726097af7ce8bd"} err="failed to get container status \"1dab51b681aceb2a14df9142fb60a6c1ff0a8a4bb5e95d0662726097af7ce8bd\": rpc error: code = NotFound desc = could not find container \"1dab51b681aceb2a14df9142fb60a6c1ff0a8a4bb5e95d0662726097af7ce8bd\": container with ID starting with 1dab51b681aceb2a14df9142fb60a6c1ff0a8a4bb5e95d0662726097af7ce8bd not found: ID does not exist" Sep 17 07:37:53.252504 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:37:53.252441 2578 scope.go:117] "RemoveContainer" containerID="32f4e92cce74952f71a93994f3c1e8db1719939ee818a9d7000c497bbd2d2911" Sep 17 07:37:53.252704 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:37:53.252685 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32f4e92cce74952f71a93994f3c1e8db1719939ee818a9d7000c497bbd2d2911"} err="failed to get container status \"32f4e92cce74952f71a93994f3c1e8db1719939ee818a9d7000c497bbd2d2911\": rpc error: code = NotFound desc = could not find container \"32f4e92cce74952f71a93994f3c1e8db1719939ee818a9d7000c497bbd2d2911\": container with ID starting with 32f4e92cce74952f71a93994f3c1e8db1719939ee818a9d7000c497bbd2d2911 not found: ID does not exist" Sep 17 07:37:53.252755 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:37:53.252706 2578 scope.go:117] "RemoveContainer" containerID="bde6e5492e29ced6eb89199051bc5b8aa154787b8c61ec6ccb0e13618b2b034f" Sep 17 07:37:53.252953 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:37:53.252931 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bde6e5492e29ced6eb89199051bc5b8aa154787b8c61ec6ccb0e13618b2b034f"} err="failed to get container status \"bde6e5492e29ced6eb89199051bc5b8aa154787b8c61ec6ccb0e13618b2b034f\": rpc error: code = NotFound desc = could not find container \"bde6e5492e29ced6eb89199051bc5b8aa154787b8c61ec6ccb0e13618b2b034f\": container with ID starting with bde6e5492e29ced6eb89199051bc5b8aa154787b8c61ec6ccb0e13618b2b034f not found: ID does not exist" Sep 17 07:37:53.253010 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:37:53.252953 2578 scope.go:117] "RemoveContainer" containerID="82770d059a4f271ced26f664afe8713cfb72baf3f3900bd61f5ead9284f52b2f" Sep 17 07:37:53.253194 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:37:53.253176 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82770d059a4f271ced26f664afe8713cfb72baf3f3900bd61f5ead9284f52b2f"} err="failed to get container status \"82770d059a4f271ced26f664afe8713cfb72baf3f3900bd61f5ead9284f52b2f\": rpc error: code = NotFound desc = could not find container \"82770d059a4f271ced26f664afe8713cfb72baf3f3900bd61f5ead9284f52b2f\": container with ID starting with 82770d059a4f271ced26f664afe8713cfb72baf3f3900bd61f5ead9284f52b2f not found: ID does not exist" Sep 17 07:37:53.316704 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:37:53.316671 2578 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/4ea2b430-adec-4cb3-94ed-12b3e3626d11-registry-tls\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Sep 17 07:37:53.316704 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:37:53.316699 2578 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/4ea2b430-adec-4cb3-94ed-12b3e3626d11-ui-tls\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Sep 17 07:37:53.316704 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:37:53.316709 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5gzp7\" (UniqueName: \"kubernetes.io/projected/4ea2b430-adec-4cb3-94ed-12b3e3626d11-kube-api-access-5gzp7\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Sep 17 07:37:53.316909 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:37:53.316718 2578 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/4ea2b430-adec-4cb3-94ed-12b3e3626d11-feast-data\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Sep 17 07:37:53.316909 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:37:53.316726 2578 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/4ea2b430-adec-4cb3-94ed-12b3e3626d11-offline-tls\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Sep 17 07:37:53.316909 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:37:53.316734 2578 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/4ea2b430-adec-4cb3-94ed-12b3e3626d11-online-tls\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Sep 17 07:37:53.530225 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:37:53.530189 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-7fbc49b5df-hhkwm"] Sep 17 07:37:53.534397 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:37:53.534372 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-7fbc49b5df-hhkwm"] Sep 17 07:37:54.628803 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:37:54.628769 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ea2b430-adec-4cb3-94ed-12b3e3626d11" path="/var/lib/kubelet/pods/4ea2b430-adec-4cb3-94ed-12b3e3626d11/volumes" Sep 17 07:38:48.894944 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:38:48.894910 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-7bdf59746-6rn92"] Sep 17 07:38:48.895370 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:38:48.895213 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ea2b430-adec-4cb3-94ed-12b3e3626d11" containerName="offline" Sep 17 07:38:48.895370 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:38:48.895224 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ea2b430-adec-4cb3-94ed-12b3e3626d11" containerName="offline" Sep 17 07:38:48.895370 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:38:48.895233 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ea2b430-adec-4cb3-94ed-12b3e3626d11" containerName="feast-apply" Sep 17 07:38:48.895370 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:38:48.895239 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ea2b430-adec-4cb3-94ed-12b3e3626d11" containerName="feast-apply" Sep 17 07:38:48.895370 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:38:48.895249 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ea2b430-adec-4cb3-94ed-12b3e3626d11" containerName="ui" Sep 17 07:38:48.895370 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:38:48.895255 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ea2b430-adec-4cb3-94ed-12b3e3626d11" containerName="ui" Sep 17 07:38:48.895370 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:38:48.895268 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ea2b430-adec-4cb3-94ed-12b3e3626d11" containerName="feast-init" Sep 17 07:38:48.895370 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:38:48.895273 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ea2b430-adec-4cb3-94ed-12b3e3626d11" containerName="feast-init" Sep 17 07:38:48.895370 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:38:48.895280 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ea2b430-adec-4cb3-94ed-12b3e3626d11" containerName="registry" Sep 17 07:38:48.895370 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:38:48.895285 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ea2b430-adec-4cb3-94ed-12b3e3626d11" containerName="registry" Sep 17 07:38:48.895370 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:38:48.895294 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ea2b430-adec-4cb3-94ed-12b3e3626d11" containerName="online" Sep 17 07:38:48.895370 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:38:48.895300 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ea2b430-adec-4cb3-94ed-12b3e3626d11" containerName="online" Sep 17 07:38:48.895370 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:38:48.895363 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ea2b430-adec-4cb3-94ed-12b3e3626d11" containerName="ui" Sep 17 07:38:48.895370 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:38:48.895370 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ea2b430-adec-4cb3-94ed-12b3e3626d11" containerName="online" Sep 17 07:38:48.895370 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:38:48.895378 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ea2b430-adec-4cb3-94ed-12b3e3626d11" containerName="registry" Sep 17 07:38:48.895836 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:38:48.895386 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ea2b430-adec-4cb3-94ed-12b3e3626d11" containerName="offline" Sep 17 07:38:48.900037 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:38:48.900018 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7bdf59746-6rn92" Sep 17 07:38:48.912307 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:38:48.912289 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-ui-tls\"" Sep 17 07:38:48.912532 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:38:48.912519 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-client-ca\"" Sep 17 07:38:48.912618 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:38:48.912595 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-offline-tls\"" Sep 17 07:38:48.913167 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:38:48.913153 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-intra-comm\"" Sep 17 07:38:48.913224 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:38:48.913154 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-dockercfg-2n7xx\"" Sep 17 07:38:48.913519 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:38:48.913504 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-online-tls\"" Sep 17 07:38:48.913617 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:38:48.913604 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"kube-root-ca.crt\"" Sep 17 07:38:48.913651 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:38:48.913644 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"openshift-service-ca.crt\"" Sep 17 07:38:48.938685 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:38:48.938662 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-7bdf59746-6rn92"] Sep 17 07:38:48.965172 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:38:48.965149 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/6ebb0937-dac3-40c9-a155-24e756265163-ui-tls\") pod \"feast-simple-feast-remote-setup-7bdf59746-6rn92\" (UID: \"6ebb0937-dac3-40c9-a155-24e756265163\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7bdf59746-6rn92" Sep 17 07:38:48.965285 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:38:48.965178 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/6ebb0937-dac3-40c9-a155-24e756265163-offline-tls\") pod \"feast-simple-feast-remote-setup-7bdf59746-6rn92\" (UID: \"6ebb0937-dac3-40c9-a155-24e756265163\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7bdf59746-6rn92" Sep 17 07:38:48.965285 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:38:48.965195 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/6ebb0937-dac3-40c9-a155-24e756265163-feast-data\") pod \"feast-simple-feast-remote-setup-7bdf59746-6rn92\" (UID: \"6ebb0937-dac3-40c9-a155-24e756265163\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7bdf59746-6rn92" Sep 17 07:38:48.965285 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:38:48.965215 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46tsk\" (UniqueName: \"kubernetes.io/projected/6ebb0937-dac3-40c9-a155-24e756265163-kube-api-access-46tsk\") pod \"feast-simple-feast-remote-setup-7bdf59746-6rn92\" (UID: \"6ebb0937-dac3-40c9-a155-24e756265163\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7bdf59746-6rn92" Sep 17 07:38:48.965389 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:38:48.965296 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/6ebb0937-dac3-40c9-a155-24e756265163-registry-tls\") pod \"feast-simple-feast-remote-setup-7bdf59746-6rn92\" (UID: \"6ebb0937-dac3-40c9-a155-24e756265163\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7bdf59746-6rn92" Sep 17 07:38:48.965389 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:38:48.965337 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/6ebb0937-dac3-40c9-a155-24e756265163-online-tls\") pod \"feast-simple-feast-remote-setup-7bdf59746-6rn92\" (UID: \"6ebb0937-dac3-40c9-a155-24e756265163\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7bdf59746-6rn92" Sep 17 07:38:49.066163 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:38:49.066137 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/6ebb0937-dac3-40c9-a155-24e756265163-ui-tls\") pod \"feast-simple-feast-remote-setup-7bdf59746-6rn92\" (UID: \"6ebb0937-dac3-40c9-a155-24e756265163\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7bdf59746-6rn92" Sep 17 07:38:49.066163 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:38:49.066168 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/6ebb0937-dac3-40c9-a155-24e756265163-offline-tls\") pod \"feast-simple-feast-remote-setup-7bdf59746-6rn92\" (UID: \"6ebb0937-dac3-40c9-a155-24e756265163\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7bdf59746-6rn92" Sep 17 07:38:49.066379 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:38:49.066183 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/6ebb0937-dac3-40c9-a155-24e756265163-feast-data\") pod \"feast-simple-feast-remote-setup-7bdf59746-6rn92\" (UID: \"6ebb0937-dac3-40c9-a155-24e756265163\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7bdf59746-6rn92" Sep 17 07:38:49.066379 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:38:49.066204 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46tsk\" (UniqueName: \"kubernetes.io/projected/6ebb0937-dac3-40c9-a155-24e756265163-kube-api-access-46tsk\") pod \"feast-simple-feast-remote-setup-7bdf59746-6rn92\" (UID: \"6ebb0937-dac3-40c9-a155-24e756265163\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7bdf59746-6rn92" Sep 17 07:38:49.066379 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:38:49.066248 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/6ebb0937-dac3-40c9-a155-24e756265163-registry-tls\") pod \"feast-simple-feast-remote-setup-7bdf59746-6rn92\" (UID: \"6ebb0937-dac3-40c9-a155-24e756265163\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7bdf59746-6rn92" Sep 17 07:38:49.066379 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:38:49.066281 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/6ebb0937-dac3-40c9-a155-24e756265163-online-tls\") pod \"feast-simple-feast-remote-setup-7bdf59746-6rn92\" (UID: \"6ebb0937-dac3-40c9-a155-24e756265163\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7bdf59746-6rn92" Sep 17 07:38:49.066379 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:38:49.066299 2578 secret.go:189] Couldn't get secret test-ns-remote-registry/feast-simple-feast-remote-setup-ui-tls: secret "feast-simple-feast-remote-setup-ui-tls" not found Sep 17 07:38:49.066616 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:38:49.066384 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ebb0937-dac3-40c9-a155-24e756265163-ui-tls podName:6ebb0937-dac3-40c9-a155-24e756265163 nodeName:}" failed. No retries permitted until 2026-09-17 07:38:49.566361637 +0000 UTC m=+673.447395989 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "ui-tls" (UniqueName: "kubernetes.io/secret/6ebb0937-dac3-40c9-a155-24e756265163-ui-tls") pod "feast-simple-feast-remote-setup-7bdf59746-6rn92" (UID: "6ebb0937-dac3-40c9-a155-24e756265163") : secret "feast-simple-feast-remote-setup-ui-tls" not found Sep 17 07:38:49.066616 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:38:49.066563 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/6ebb0937-dac3-40c9-a155-24e756265163-feast-data\") pod \"feast-simple-feast-remote-setup-7bdf59746-6rn92\" (UID: \"6ebb0937-dac3-40c9-a155-24e756265163\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7bdf59746-6rn92" Sep 17 07:38:49.066978 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:38:49.066958 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/6ebb0937-dac3-40c9-a155-24e756265163-registry-tls\") pod \"feast-simple-feast-remote-setup-7bdf59746-6rn92\" (UID: \"6ebb0937-dac3-40c9-a155-24e756265163\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7bdf59746-6rn92" Sep 17 07:38:49.068734 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:38:49.068715 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/6ebb0937-dac3-40c9-a155-24e756265163-online-tls\") pod \"feast-simple-feast-remote-setup-7bdf59746-6rn92\" (UID: \"6ebb0937-dac3-40c9-a155-24e756265163\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7bdf59746-6rn92" Sep 17 07:38:49.068841 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:38:49.068796 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/6ebb0937-dac3-40c9-a155-24e756265163-offline-tls\") pod \"feast-simple-feast-remote-setup-7bdf59746-6rn92\" (UID: \"6ebb0937-dac3-40c9-a155-24e756265163\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7bdf59746-6rn92" Sep 17 07:38:49.089614 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:38:49.089594 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-46tsk\" (UniqueName: \"kubernetes.io/projected/6ebb0937-dac3-40c9-a155-24e756265163-kube-api-access-46tsk\") pod \"feast-simple-feast-remote-setup-7bdf59746-6rn92\" (UID: \"6ebb0937-dac3-40c9-a155-24e756265163\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7bdf59746-6rn92" Sep 17 07:38:49.568946 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:38:49.568916 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/6ebb0937-dac3-40c9-a155-24e756265163-ui-tls\") pod \"feast-simple-feast-remote-setup-7bdf59746-6rn92\" (UID: \"6ebb0937-dac3-40c9-a155-24e756265163\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7bdf59746-6rn92" Sep 17 07:38:49.571313 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:38:49.571283 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/6ebb0937-dac3-40c9-a155-24e756265163-ui-tls\") pod \"feast-simple-feast-remote-setup-7bdf59746-6rn92\" (UID: \"6ebb0937-dac3-40c9-a155-24e756265163\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7bdf59746-6rn92" Sep 17 07:38:49.808666 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:38:49.808629 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7bdf59746-6rn92" Sep 17 07:38:50.104062 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:38:50.103981 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-7bdf59746-6rn92"] Sep 17 07:38:50.109035 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:38:50.109010 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6ebb0937_dac3_40c9_a155_24e756265163.slice/crio-93b847e85a0f5f2e78dc9f4897e82ad746b280fd0b1b76f296ed168542d0355b WatchSource:0}: Error finding container 93b847e85a0f5f2e78dc9f4897e82ad746b280fd0b1b76f296ed168542d0355b: Status 404 returned error can't find the container with id 93b847e85a0f5f2e78dc9f4897e82ad746b280fd0b1b76f296ed168542d0355b Sep 17 07:38:50.434418 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:38:50.434380 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7bdf59746-6rn92" event={"ID":"6ebb0937-dac3-40c9-a155-24e756265163","Type":"ContainerStarted","Data":"17281ebb9d8b07d6ab5c84322677b680b158745cb4635ea4d1c8626e965cde0a"} Sep 17 07:38:50.434615 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:38:50.434434 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7bdf59746-6rn92" event={"ID":"6ebb0937-dac3-40c9-a155-24e756265163","Type":"ContainerStarted","Data":"93b847e85a0f5f2e78dc9f4897e82ad746b280fd0b1b76f296ed168542d0355b"} Sep 17 07:38:55.459422 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:38:55.459389 2578 generic.go:356] "Generic (PLEG): container finished" podID="6ebb0937-dac3-40c9-a155-24e756265163" containerID="17281ebb9d8b07d6ab5c84322677b680b158745cb4635ea4d1c8626e965cde0a" exitCode=0 Sep 17 07:38:55.459796 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:38:55.459464 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7bdf59746-6rn92" event={"ID":"6ebb0937-dac3-40c9-a155-24e756265163","Type":"ContainerDied","Data":"17281ebb9d8b07d6ab5c84322677b680b158745cb4635ea4d1c8626e965cde0a"} Sep 17 07:38:56.465720 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:38:56.465684 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7bdf59746-6rn92" event={"ID":"6ebb0937-dac3-40c9-a155-24e756265163","Type":"ContainerStarted","Data":"2b0ebe9a2ad58f1e4fe0b268b7bc908ba8f57ef2e2700e95c25e502b4f165e3e"} Sep 17 07:39:24.590724 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:39:24.590693 2578 generic.go:356] "Generic (PLEG): container finished" podID="6ebb0937-dac3-40c9-a155-24e756265163" containerID="2b0ebe9a2ad58f1e4fe0b268b7bc908ba8f57ef2e2700e95c25e502b4f165e3e" exitCode=0 Sep 17 07:39:24.591131 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:39:24.590769 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7bdf59746-6rn92" event={"ID":"6ebb0937-dac3-40c9-a155-24e756265163","Type":"ContainerDied","Data":"2b0ebe9a2ad58f1e4fe0b268b7bc908ba8f57ef2e2700e95c25e502b4f165e3e"} Sep 17 07:39:25.599381 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:39:25.599343 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7bdf59746-6rn92" event={"ID":"6ebb0937-dac3-40c9-a155-24e756265163","Type":"ContainerStarted","Data":"1aeff61b4d2feddeda663236fb0ae8a9d3d0cf281c8d84315be1aefde847231b"} Sep 17 07:39:25.599752 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:39:25.599403 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7bdf59746-6rn92" event={"ID":"6ebb0937-dac3-40c9-a155-24e756265163","Type":"ContainerStarted","Data":"2fc0900fb9d40d5db9161c58818df233870c604d9e5103a6952cb9a1d4d78218"} Sep 17 07:39:25.599752 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:39:25.599425 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7bdf59746-6rn92" event={"ID":"6ebb0937-dac3-40c9-a155-24e756265163","Type":"ContainerStarted","Data":"6a1ab0fdefd0dff488fc3b6cc58a855bd43afc2d4c202ab2bd0cccdf13be17cb"} Sep 17 07:39:25.639449 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:39:25.639396 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7bdf59746-6rn92" podStartSLOduration=37.639379701 podStartE2EDuration="37.639379701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 07:38:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 07:39:25.637095184 +0000 UTC m=+709.518129565" watchObservedRunningTime="2026-09-17 07:39:25.639379701 +0000 UTC m=+709.520414059" Sep 17 07:39:25.809407 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:39:25.809299 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7bdf59746-6rn92" Sep 17 07:39:25.809407 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:39:25.809359 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7bdf59746-6rn92" Sep 17 07:39:25.809407 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:39:25.809379 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7bdf59746-6rn92" Sep 17 07:39:25.810803 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:39:25.810762 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7bdf59746-6rn92" podUID="6ebb0937-dac3-40c9-a155-24e756265163" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.29:8443: connect: connection refused" Sep 17 07:39:25.810974 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:39:25.810802 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7bdf59746-6rn92" podUID="6ebb0937-dac3-40c9-a155-24e756265163" containerName="online" probeResult="failure" output="Get \"https://10.132.0.29:6567/health\": dial tcp 10.132.0.29:6567: connect: connection refused" Sep 17 07:39:25.810974 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:39:25.810821 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7bdf59746-6rn92" podUID="6ebb0937-dac3-40c9-a155-24e756265163" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.29:8816: connect: connection refused" Sep 17 07:39:28.809485 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:39:28.809434 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7bdf59746-6rn92" podUID="6ebb0937-dac3-40c9-a155-24e756265163" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.29:8443: connect: connection refused" Sep 17 07:39:28.809997 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:39:28.809488 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7bdf59746-6rn92" podUID="6ebb0937-dac3-40c9-a155-24e756265163" containerName="online" probeResult="failure" output="Get \"https://10.132.0.29:6567/health\": dial tcp 10.132.0.29:6567: connect: connection refused" Sep 17 07:39:28.809997 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:39:28.809441 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7bdf59746-6rn92" podUID="6ebb0937-dac3-40c9-a155-24e756265163" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.29:8816: connect: connection refused" Sep 17 07:39:29.809705 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:39:29.809665 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7bdf59746-6rn92" Sep 17 07:39:29.810301 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:39:29.809735 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7bdf59746-6rn92" Sep 17 07:39:29.810301 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:39:29.809757 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7bdf59746-6rn92" Sep 17 07:39:31.809294 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:39:31.809249 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7bdf59746-6rn92" podUID="6ebb0937-dac3-40c9-a155-24e756265163" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.29:8443: connect: connection refused" Sep 17 07:39:31.809788 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:39:31.809250 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7bdf59746-6rn92" podUID="6ebb0937-dac3-40c9-a155-24e756265163" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.29:8816: connect: connection refused" Sep 17 07:39:31.809788 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:39:31.809252 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7bdf59746-6rn92" podUID="6ebb0937-dac3-40c9-a155-24e756265163" containerName="online" probeResult="failure" output="Get \"https://10.132.0.29:6567/health\": dial tcp 10.132.0.29:6567: connect: connection refused" Sep 17 07:39:34.810384 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:39:34.810353 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7bdf59746-6rn92" Sep 17 07:39:34.810918 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:39:34.810896 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7bdf59746-6rn92" Sep 17 07:39:34.811018 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:39:34.810957 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7bdf59746-6rn92" Sep 17 07:39:34.838308 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:39:34.838281 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7bdf59746-6rn92" Sep 17 07:39:35.647953 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:39:35.647928 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7bdf59746-6rn92" Sep 17 07:39:35.674914 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:39:35.674889 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7bdf59746-6rn92" Sep 17 07:40:38.976021 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:40:38.975989 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-7bdf59746-6rn92"] Sep 17 07:40:38.976565 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:40:38.976198 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7bdf59746-6rn92" podUID="6ebb0937-dac3-40c9-a155-24e756265163" containerName="online" containerID="cri-o://6a1ab0fdefd0dff488fc3b6cc58a855bd43afc2d4c202ab2bd0cccdf13be17cb" gracePeriod=30 Sep 17 07:40:38.976565 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:40:38.976262 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7bdf59746-6rn92" podUID="6ebb0937-dac3-40c9-a155-24e756265163" containerName="offline" containerID="cri-o://2fc0900fb9d40d5db9161c58818df233870c604d9e5103a6952cb9a1d4d78218" gracePeriod=30 Sep 17 07:40:38.976565 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:40:38.976340 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7bdf59746-6rn92" podUID="6ebb0937-dac3-40c9-a155-24e756265163" containerName="ui" containerID="cri-o://1aeff61b4d2feddeda663236fb0ae8a9d3d0cf281c8d84315be1aefde847231b" gracePeriod=30 Sep 17 07:40:39.017073 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:40:39.017043 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-7fbc49b5df-sllcc"] Sep 17 07:40:39.017282 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:40:39.017252 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-sllcc" podUID="1feb527a-10d8-4bc8-be30-52a9a4634c33" containerName="registry" containerID="cri-o://2309e2fa5d7c715a45900d9fa64c731a066425b5bb3558c09c935da79dd962a7" gracePeriod=30 Sep 17 07:40:39.017415 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:40:39.017316 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-sllcc" podUID="1feb527a-10d8-4bc8-be30-52a9a4634c33" containerName="offline" containerID="cri-o://b5dff01f644fcde1c0230dfc52529290b7e809b8ed5b740cec4e022423549cb9" gracePeriod=30 Sep 17 07:40:39.017415 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:40:39.017325 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-sllcc" podUID="1feb527a-10d8-4bc8-be30-52a9a4634c33" containerName="online" containerID="cri-o://b4bb042dd05c3651e1163c94999ebab47df20f05f61e5a36ab8b3056dbc2ec38" gracePeriod=30 Sep 17 07:40:39.017415 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:40:39.017323 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-sllcc" podUID="1feb527a-10d8-4bc8-be30-52a9a4634c33" containerName="ui" containerID="cri-o://422adaa09dd13d416aa68bc192459f8c7e521870ed33b34ad6560953a1378b94" gracePeriod=30 Sep 17 07:40:39.907034 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:40:39.907008 2578 generic.go:356] "Generic (PLEG): container finished" podID="6ebb0937-dac3-40c9-a155-24e756265163" containerID="1aeff61b4d2feddeda663236fb0ae8a9d3d0cf281c8d84315be1aefde847231b" exitCode=0 Sep 17 07:40:39.907034 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:40:39.907031 2578 generic.go:356] "Generic (PLEG): container finished" podID="6ebb0937-dac3-40c9-a155-24e756265163" containerID="6a1ab0fdefd0dff488fc3b6cc58a855bd43afc2d4c202ab2bd0cccdf13be17cb" exitCode=0 Sep 17 07:40:39.907180 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:40:39.907078 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7bdf59746-6rn92" event={"ID":"6ebb0937-dac3-40c9-a155-24e756265163","Type":"ContainerDied","Data":"1aeff61b4d2feddeda663236fb0ae8a9d3d0cf281c8d84315be1aefde847231b"} Sep 17 07:40:39.907180 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:40:39.907126 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7bdf59746-6rn92" event={"ID":"6ebb0937-dac3-40c9-a155-24e756265163","Type":"ContainerDied","Data":"6a1ab0fdefd0dff488fc3b6cc58a855bd43afc2d4c202ab2bd0cccdf13be17cb"} Sep 17 07:40:39.909934 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:40:39.909914 2578 generic.go:356] "Generic (PLEG): container finished" podID="1feb527a-10d8-4bc8-be30-52a9a4634c33" containerID="422adaa09dd13d416aa68bc192459f8c7e521870ed33b34ad6560953a1378b94" exitCode=0 Sep 17 07:40:39.910030 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:40:39.909957 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-sllcc" event={"ID":"1feb527a-10d8-4bc8-be30-52a9a4634c33","Type":"ContainerDied","Data":"422adaa09dd13d416aa68bc192459f8c7e521870ed33b34ad6560953a1378b94"} Sep 17 07:40:40.917370 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:40:40.917334 2578 generic.go:356] "Generic (PLEG): container finished" podID="1feb527a-10d8-4bc8-be30-52a9a4634c33" containerID="b4bb042dd05c3651e1163c94999ebab47df20f05f61e5a36ab8b3056dbc2ec38" exitCode=0 Sep 17 07:40:40.917724 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:40:40.917407 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-sllcc" event={"ID":"1feb527a-10d8-4bc8-be30-52a9a4634c33","Type":"ContainerDied","Data":"b4bb042dd05c3651e1163c94999ebab47df20f05f61e5a36ab8b3056dbc2ec38"} Sep 17 07:40:45.169999 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:40:45.169951 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-sllcc" podUID="1feb527a-10d8-4bc8-be30-52a9a4634c33" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.28:8443: connect: connection refused" Sep 17 07:40:45.170538 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:40:45.170033 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-sllcc" podUID="1feb527a-10d8-4bc8-be30-52a9a4634c33" containerName="online" probeResult="failure" output="Get \"https://10.132.0.28:6567/health\": dial tcp 10.132.0.28:6567: connect: connection refused" Sep 17 07:40:45.648367 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:40:45.648328 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7bdf59746-6rn92" podUID="6ebb0937-dac3-40c9-a155-24e756265163" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.29:8443: connect: connection refused" Sep 17 07:40:45.648522 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:40:45.648370 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7bdf59746-6rn92" podUID="6ebb0937-dac3-40c9-a155-24e756265163" containerName="online" probeResult="failure" output="Get \"https://10.132.0.29:6567/health\": dial tcp 10.132.0.29:6567: connect: connection refused" Sep 17 07:40:55.170491 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:40:55.170446 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-sllcc" podUID="1feb527a-10d8-4bc8-be30-52a9a4634c33" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.28:8443: connect: connection refused" Sep 17 07:40:55.170919 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:40:55.170534 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-sllcc" podUID="1feb527a-10d8-4bc8-be30-52a9a4634c33" containerName="online" probeResult="failure" output="Get \"https://10.132.0.28:6567/health\": dial tcp 10.132.0.28:6567: connect: connection refused" Sep 17 07:40:55.647807 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:40:55.647769 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7bdf59746-6rn92" podUID="6ebb0937-dac3-40c9-a155-24e756265163" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.29:8443: connect: connection refused" Sep 17 07:40:55.647995 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:40:55.647809 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7bdf59746-6rn92" podUID="6ebb0937-dac3-40c9-a155-24e756265163" containerName="online" probeResult="failure" output="Get \"https://10.132.0.29:6567/health\": dial tcp 10.132.0.29:6567: connect: connection refused" Sep 17 07:41:05.170190 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:05.170097 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-sllcc" podUID="1feb527a-10d8-4bc8-be30-52a9a4634c33" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.28:8443: connect: connection refused" Sep 17 07:41:05.170631 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:05.170117 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-sllcc" podUID="1feb527a-10d8-4bc8-be30-52a9a4634c33" containerName="online" probeResult="failure" output="Get \"https://10.132.0.28:6567/health\": dial tcp 10.132.0.28:6567: connect: connection refused" Sep 17 07:41:05.170631 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:05.170270 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-sllcc" Sep 17 07:41:05.170631 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:05.170362 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-sllcc" Sep 17 07:41:05.647934 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:05.647894 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7bdf59746-6rn92" podUID="6ebb0937-dac3-40c9-a155-24e756265163" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.29:8443: connect: connection refused" Sep 17 07:41:05.648085 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:05.647938 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7bdf59746-6rn92" podUID="6ebb0937-dac3-40c9-a155-24e756265163" containerName="online" probeResult="failure" output="Get \"https://10.132.0.29:6567/health\": dial tcp 10.132.0.29:6567: connect: connection refused" Sep 17 07:41:05.648085 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:05.648015 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7bdf59746-6rn92" Sep 17 07:41:05.648085 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:05.648062 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7bdf59746-6rn92" Sep 17 07:41:09.030302 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:09.030265 2578 generic.go:356] "Generic (PLEG): container finished" podID="6ebb0937-dac3-40c9-a155-24e756265163" containerID="2fc0900fb9d40d5db9161c58818df233870c604d9e5103a6952cb9a1d4d78218" exitCode=137 Sep 17 07:41:09.030599 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:09.030295 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7bdf59746-6rn92" event={"ID":"6ebb0937-dac3-40c9-a155-24e756265163","Type":"ContainerDied","Data":"2fc0900fb9d40d5db9161c58818df233870c604d9e5103a6952cb9a1d4d78218"} Sep 17 07:41:09.149342 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:09.149319 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7bdf59746-6rn92" Sep 17 07:41:09.195086 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:09.195060 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/6ebb0937-dac3-40c9-a155-24e756265163-online-tls\") pod \"6ebb0937-dac3-40c9-a155-24e756265163\" (UID: \"6ebb0937-dac3-40c9-a155-24e756265163\") " Sep 17 07:41:09.195251 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:09.195129 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/6ebb0937-dac3-40c9-a155-24e756265163-ui-tls\") pod \"6ebb0937-dac3-40c9-a155-24e756265163\" (UID: \"6ebb0937-dac3-40c9-a155-24e756265163\") " Sep 17 07:41:09.195251 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:09.195157 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-46tsk\" (UniqueName: \"kubernetes.io/projected/6ebb0937-dac3-40c9-a155-24e756265163-kube-api-access-46tsk\") pod \"6ebb0937-dac3-40c9-a155-24e756265163\" (UID: \"6ebb0937-dac3-40c9-a155-24e756265163\") " Sep 17 07:41:09.195251 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:09.195199 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/6ebb0937-dac3-40c9-a155-24e756265163-offline-tls\") pod \"6ebb0937-dac3-40c9-a155-24e756265163\" (UID: \"6ebb0937-dac3-40c9-a155-24e756265163\") " Sep 17 07:41:09.195251 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:09.195223 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/6ebb0937-dac3-40c9-a155-24e756265163-registry-tls\") pod \"6ebb0937-dac3-40c9-a155-24e756265163\" (UID: \"6ebb0937-dac3-40c9-a155-24e756265163\") " Sep 17 07:41:09.195456 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:09.195255 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/6ebb0937-dac3-40c9-a155-24e756265163-feast-data\") pod \"6ebb0937-dac3-40c9-a155-24e756265163\" (UID: \"6ebb0937-dac3-40c9-a155-24e756265163\") " Sep 17 07:41:09.195661 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:09.195631 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ebb0937-dac3-40c9-a155-24e756265163-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "6ebb0937-dac3-40c9-a155-24e756265163" (UID: "6ebb0937-dac3-40c9-a155-24e756265163"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 07:41:09.195875 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:09.195833 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ebb0937-dac3-40c9-a155-24e756265163-feast-data" (OuterVolumeSpecName: "feast-data") pod "6ebb0937-dac3-40c9-a155-24e756265163" (UID: "6ebb0937-dac3-40c9-a155-24e756265163"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 07:41:09.197305 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:09.197282 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ebb0937-dac3-40c9-a155-24e756265163-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "6ebb0937-dac3-40c9-a155-24e756265163" (UID: "6ebb0937-dac3-40c9-a155-24e756265163"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 07:41:09.197403 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:09.197339 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ebb0937-dac3-40c9-a155-24e756265163-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "6ebb0937-dac3-40c9-a155-24e756265163" (UID: "6ebb0937-dac3-40c9-a155-24e756265163"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 07:41:09.197403 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:09.197370 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ebb0937-dac3-40c9-a155-24e756265163-kube-api-access-46tsk" (OuterVolumeSpecName: "kube-api-access-46tsk") pod "6ebb0937-dac3-40c9-a155-24e756265163" (UID: "6ebb0937-dac3-40c9-a155-24e756265163"). InnerVolumeSpecName "kube-api-access-46tsk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 07:41:09.197403 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:09.197391 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ebb0937-dac3-40c9-a155-24e756265163-online-tls" (OuterVolumeSpecName: "online-tls") pod "6ebb0937-dac3-40c9-a155-24e756265163" (UID: "6ebb0937-dac3-40c9-a155-24e756265163"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 07:41:09.296534 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:09.296480 2578 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/6ebb0937-dac3-40c9-a155-24e756265163-online-tls\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Sep 17 07:41:09.296534 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:09.296501 2578 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/6ebb0937-dac3-40c9-a155-24e756265163-ui-tls\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Sep 17 07:41:09.296534 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:09.296511 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46tsk\" (UniqueName: \"kubernetes.io/projected/6ebb0937-dac3-40c9-a155-24e756265163-kube-api-access-46tsk\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Sep 17 07:41:09.296534 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:09.296520 2578 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/6ebb0937-dac3-40c9-a155-24e756265163-offline-tls\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Sep 17 07:41:09.296534 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:09.296528 2578 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/6ebb0937-dac3-40c9-a155-24e756265163-registry-tls\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Sep 17 07:41:09.296534 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:09.296536 2578 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/6ebb0937-dac3-40c9-a155-24e756265163-feast-data\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Sep 17 07:41:09.660193 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:09.660170 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-sllcc" Sep 17 07:41:09.699054 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:09.699030 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/1feb527a-10d8-4bc8-be30-52a9a4634c33-offline-tls\") pod \"1feb527a-10d8-4bc8-be30-52a9a4634c33\" (UID: \"1feb527a-10d8-4bc8-be30-52a9a4634c33\") " Sep 17 07:41:09.699171 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:09.699093 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/1feb527a-10d8-4bc8-be30-52a9a4634c33-ui-tls\") pod \"1feb527a-10d8-4bc8-be30-52a9a4634c33\" (UID: \"1feb527a-10d8-4bc8-be30-52a9a4634c33\") " Sep 17 07:41:09.699171 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:09.699113 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/1feb527a-10d8-4bc8-be30-52a9a4634c33-feast-data\") pod \"1feb527a-10d8-4bc8-be30-52a9a4634c33\" (UID: \"1feb527a-10d8-4bc8-be30-52a9a4634c33\") " Sep 17 07:41:09.699171 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:09.699131 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pxqst\" (UniqueName: \"kubernetes.io/projected/1feb527a-10d8-4bc8-be30-52a9a4634c33-kube-api-access-pxqst\") pod \"1feb527a-10d8-4bc8-be30-52a9a4634c33\" (UID: \"1feb527a-10d8-4bc8-be30-52a9a4634c33\") " Sep 17 07:41:09.699328 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:09.699171 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/1feb527a-10d8-4bc8-be30-52a9a4634c33-registry-tls\") pod \"1feb527a-10d8-4bc8-be30-52a9a4634c33\" (UID: \"1feb527a-10d8-4bc8-be30-52a9a4634c33\") " Sep 17 07:41:09.699328 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:09.699203 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/1feb527a-10d8-4bc8-be30-52a9a4634c33-online-tls\") pod \"1feb527a-10d8-4bc8-be30-52a9a4634c33\" (UID: \"1feb527a-10d8-4bc8-be30-52a9a4634c33\") " Sep 17 07:41:09.699713 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:09.699680 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1feb527a-10d8-4bc8-be30-52a9a4634c33-feast-data" (OuterVolumeSpecName: "feast-data") pod "1feb527a-10d8-4bc8-be30-52a9a4634c33" (UID: "1feb527a-10d8-4bc8-be30-52a9a4634c33"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 07:41:09.701322 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:09.701290 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1feb527a-10d8-4bc8-be30-52a9a4634c33-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "1feb527a-10d8-4bc8-be30-52a9a4634c33" (UID: "1feb527a-10d8-4bc8-be30-52a9a4634c33"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 07:41:09.701436 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:09.701391 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1feb527a-10d8-4bc8-be30-52a9a4634c33-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "1feb527a-10d8-4bc8-be30-52a9a4634c33" (UID: "1feb527a-10d8-4bc8-be30-52a9a4634c33"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 07:41:09.701619 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:09.701598 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1feb527a-10d8-4bc8-be30-52a9a4634c33-kube-api-access-pxqst" (OuterVolumeSpecName: "kube-api-access-pxqst") pod "1feb527a-10d8-4bc8-be30-52a9a4634c33" (UID: "1feb527a-10d8-4bc8-be30-52a9a4634c33"). InnerVolumeSpecName "kube-api-access-pxqst". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 07:41:09.701690 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:09.701649 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1feb527a-10d8-4bc8-be30-52a9a4634c33-online-tls" (OuterVolumeSpecName: "online-tls") pod "1feb527a-10d8-4bc8-be30-52a9a4634c33" (UID: "1feb527a-10d8-4bc8-be30-52a9a4634c33"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 07:41:09.701690 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:09.701673 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1feb527a-10d8-4bc8-be30-52a9a4634c33-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "1feb527a-10d8-4bc8-be30-52a9a4634c33" (UID: "1feb527a-10d8-4bc8-be30-52a9a4634c33"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 07:41:09.799655 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:09.799632 2578 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/1feb527a-10d8-4bc8-be30-52a9a4634c33-ui-tls\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Sep 17 07:41:09.799655 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:09.799657 2578 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/1feb527a-10d8-4bc8-be30-52a9a4634c33-feast-data\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Sep 17 07:41:09.799655 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:09.799670 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxqst\" (UniqueName: \"kubernetes.io/projected/1feb527a-10d8-4bc8-be30-52a9a4634c33-kube-api-access-pxqst\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Sep 17 07:41:09.799856 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:09.799683 2578 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/1feb527a-10d8-4bc8-be30-52a9a4634c33-registry-tls\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Sep 17 07:41:09.799856 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:09.799694 2578 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/1feb527a-10d8-4bc8-be30-52a9a4634c33-online-tls\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Sep 17 07:41:09.799856 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:09.799704 2578 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/1feb527a-10d8-4bc8-be30-52a9a4634c33-offline-tls\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Sep 17 07:41:10.039313 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:10.039289 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7bdf59746-6rn92" Sep 17 07:41:10.039697 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:10.039288 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7bdf59746-6rn92" event={"ID":"6ebb0937-dac3-40c9-a155-24e756265163","Type":"ContainerDied","Data":"93b847e85a0f5f2e78dc9f4897e82ad746b280fd0b1b76f296ed168542d0355b"} Sep 17 07:41:10.039697 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:10.039430 2578 scope.go:117] "RemoveContainer" containerID="1aeff61b4d2feddeda663236fb0ae8a9d3d0cf281c8d84315be1aefde847231b" Sep 17 07:41:10.042440 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:10.042418 2578 generic.go:356] "Generic (PLEG): container finished" podID="1feb527a-10d8-4bc8-be30-52a9a4634c33" containerID="b5dff01f644fcde1c0230dfc52529290b7e809b8ed5b740cec4e022423549cb9" exitCode=137 Sep 17 07:41:10.042526 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:10.042443 2578 generic.go:356] "Generic (PLEG): container finished" podID="1feb527a-10d8-4bc8-be30-52a9a4634c33" containerID="2309e2fa5d7c715a45900d9fa64c731a066425b5bb3558c09c935da79dd962a7" exitCode=137 Sep 17 07:41:10.042526 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:10.042475 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-sllcc" event={"ID":"1feb527a-10d8-4bc8-be30-52a9a4634c33","Type":"ContainerDied","Data":"b5dff01f644fcde1c0230dfc52529290b7e809b8ed5b740cec4e022423549cb9"} Sep 17 07:41:10.042526 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:10.042500 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-sllcc" Sep 17 07:41:10.042526 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:10.042508 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-sllcc" event={"ID":"1feb527a-10d8-4bc8-be30-52a9a4634c33","Type":"ContainerDied","Data":"2309e2fa5d7c715a45900d9fa64c731a066425b5bb3558c09c935da79dd962a7"} Sep 17 07:41:10.042670 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:10.042528 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7fbc49b5df-sllcc" event={"ID":"1feb527a-10d8-4bc8-be30-52a9a4634c33","Type":"ContainerDied","Data":"d0fab2448fafbe46b24cc9466a789a854faa53d06291c36c8f3bc7abee73abd6"} Sep 17 07:41:10.048447 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:10.048422 2578 scope.go:117] "RemoveContainer" containerID="2fc0900fb9d40d5db9161c58818df233870c604d9e5103a6952cb9a1d4d78218" Sep 17 07:41:10.055881 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:10.055846 2578 scope.go:117] "RemoveContainer" containerID="6a1ab0fdefd0dff488fc3b6cc58a855bd43afc2d4c202ab2bd0cccdf13be17cb" Sep 17 07:41:10.063089 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:10.063072 2578 scope.go:117] "RemoveContainer" containerID="2b0ebe9a2ad58f1e4fe0b268b7bc908ba8f57ef2e2700e95c25e502b4f165e3e" Sep 17 07:41:10.069953 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:10.069913 2578 scope.go:117] "RemoveContainer" containerID="17281ebb9d8b07d6ab5c84322677b680b158745cb4635ea4d1c8626e965cde0a" Sep 17 07:41:10.072292 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:10.072271 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-7bdf59746-6rn92"] Sep 17 07:41:10.074009 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:10.073990 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-7bdf59746-6rn92"] Sep 17 07:41:10.077701 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:10.077684 2578 scope.go:117] "RemoveContainer" containerID="422adaa09dd13d416aa68bc192459f8c7e521870ed33b34ad6560953a1378b94" Sep 17 07:41:10.084482 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:10.084468 2578 scope.go:117] "RemoveContainer" containerID="b5dff01f644fcde1c0230dfc52529290b7e809b8ed5b740cec4e022423549cb9" Sep 17 07:41:10.088174 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:10.088153 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-7fbc49b5df-sllcc"] Sep 17 07:41:10.091425 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:10.091407 2578 scope.go:117] "RemoveContainer" containerID="b4bb042dd05c3651e1163c94999ebab47df20f05f61e5a36ab8b3056dbc2ec38" Sep 17 07:41:10.093728 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:10.093710 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-7fbc49b5df-sllcc"] Sep 17 07:41:10.098718 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:10.098697 2578 scope.go:117] "RemoveContainer" containerID="2309e2fa5d7c715a45900d9fa64c731a066425b5bb3558c09c935da79dd962a7" Sep 17 07:41:10.105518 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:10.105501 2578 scope.go:117] "RemoveContainer" containerID="af0a84299eb5b4e5924777dca47d3e315d31a9567d26e793c27fd6ba666c3a4a" Sep 17 07:41:10.112218 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:10.112205 2578 scope.go:117] "RemoveContainer" containerID="549416d35482a8540070cb0eb67659efb9e4e92c4f44286b0d8964720329d750" Sep 17 07:41:10.118830 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:10.118811 2578 scope.go:117] "RemoveContainer" containerID="422adaa09dd13d416aa68bc192459f8c7e521870ed33b34ad6560953a1378b94" Sep 17 07:41:10.119112 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:41:10.119095 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"422adaa09dd13d416aa68bc192459f8c7e521870ed33b34ad6560953a1378b94\": container with ID starting with 422adaa09dd13d416aa68bc192459f8c7e521870ed33b34ad6560953a1378b94 not found: ID does not exist" containerID="422adaa09dd13d416aa68bc192459f8c7e521870ed33b34ad6560953a1378b94" Sep 17 07:41:10.119153 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:10.119117 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"422adaa09dd13d416aa68bc192459f8c7e521870ed33b34ad6560953a1378b94"} err="failed to get container status \"422adaa09dd13d416aa68bc192459f8c7e521870ed33b34ad6560953a1378b94\": rpc error: code = NotFound desc = could not find container \"422adaa09dd13d416aa68bc192459f8c7e521870ed33b34ad6560953a1378b94\": container with ID starting with 422adaa09dd13d416aa68bc192459f8c7e521870ed33b34ad6560953a1378b94 not found: ID does not exist" Sep 17 07:41:10.119153 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:10.119133 2578 scope.go:117] "RemoveContainer" containerID="b5dff01f644fcde1c0230dfc52529290b7e809b8ed5b740cec4e022423549cb9" Sep 17 07:41:10.119365 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:41:10.119348 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5dff01f644fcde1c0230dfc52529290b7e809b8ed5b740cec4e022423549cb9\": container with ID starting with b5dff01f644fcde1c0230dfc52529290b7e809b8ed5b740cec4e022423549cb9 not found: ID does not exist" containerID="b5dff01f644fcde1c0230dfc52529290b7e809b8ed5b740cec4e022423549cb9" Sep 17 07:41:10.119407 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:10.119370 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5dff01f644fcde1c0230dfc52529290b7e809b8ed5b740cec4e022423549cb9"} err="failed to get container status \"b5dff01f644fcde1c0230dfc52529290b7e809b8ed5b740cec4e022423549cb9\": rpc error: code = NotFound desc = could not find container \"b5dff01f644fcde1c0230dfc52529290b7e809b8ed5b740cec4e022423549cb9\": container with ID starting with b5dff01f644fcde1c0230dfc52529290b7e809b8ed5b740cec4e022423549cb9 not found: ID does not exist" Sep 17 07:41:10.119407 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:10.119386 2578 scope.go:117] "RemoveContainer" containerID="b4bb042dd05c3651e1163c94999ebab47df20f05f61e5a36ab8b3056dbc2ec38" Sep 17 07:41:10.119603 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:41:10.119588 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4bb042dd05c3651e1163c94999ebab47df20f05f61e5a36ab8b3056dbc2ec38\": container with ID starting with b4bb042dd05c3651e1163c94999ebab47df20f05f61e5a36ab8b3056dbc2ec38 not found: ID does not exist" containerID="b4bb042dd05c3651e1163c94999ebab47df20f05f61e5a36ab8b3056dbc2ec38" Sep 17 07:41:10.119649 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:10.119607 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4bb042dd05c3651e1163c94999ebab47df20f05f61e5a36ab8b3056dbc2ec38"} err="failed to get container status \"b4bb042dd05c3651e1163c94999ebab47df20f05f61e5a36ab8b3056dbc2ec38\": rpc error: code = NotFound desc = could not find container \"b4bb042dd05c3651e1163c94999ebab47df20f05f61e5a36ab8b3056dbc2ec38\": container with ID starting with b4bb042dd05c3651e1163c94999ebab47df20f05f61e5a36ab8b3056dbc2ec38 not found: ID does not exist" Sep 17 07:41:10.119649 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:10.119620 2578 scope.go:117] "RemoveContainer" containerID="2309e2fa5d7c715a45900d9fa64c731a066425b5bb3558c09c935da79dd962a7" Sep 17 07:41:10.119810 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:41:10.119795 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2309e2fa5d7c715a45900d9fa64c731a066425b5bb3558c09c935da79dd962a7\": container with ID starting with 2309e2fa5d7c715a45900d9fa64c731a066425b5bb3558c09c935da79dd962a7 not found: ID does not exist" containerID="2309e2fa5d7c715a45900d9fa64c731a066425b5bb3558c09c935da79dd962a7" Sep 17 07:41:10.119846 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:10.119814 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2309e2fa5d7c715a45900d9fa64c731a066425b5bb3558c09c935da79dd962a7"} err="failed to get container status \"2309e2fa5d7c715a45900d9fa64c731a066425b5bb3558c09c935da79dd962a7\": rpc error: code = NotFound desc = could not find container \"2309e2fa5d7c715a45900d9fa64c731a066425b5bb3558c09c935da79dd962a7\": container with ID starting with 2309e2fa5d7c715a45900d9fa64c731a066425b5bb3558c09c935da79dd962a7 not found: ID does not exist" Sep 17 07:41:10.119846 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:10.119827 2578 scope.go:117] "RemoveContainer" containerID="af0a84299eb5b4e5924777dca47d3e315d31a9567d26e793c27fd6ba666c3a4a" Sep 17 07:41:10.120103 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:41:10.120086 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af0a84299eb5b4e5924777dca47d3e315d31a9567d26e793c27fd6ba666c3a4a\": container with ID starting with af0a84299eb5b4e5924777dca47d3e315d31a9567d26e793c27fd6ba666c3a4a not found: ID does not exist" containerID="af0a84299eb5b4e5924777dca47d3e315d31a9567d26e793c27fd6ba666c3a4a" Sep 17 07:41:10.120165 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:10.120107 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af0a84299eb5b4e5924777dca47d3e315d31a9567d26e793c27fd6ba666c3a4a"} err="failed to get container status \"af0a84299eb5b4e5924777dca47d3e315d31a9567d26e793c27fd6ba666c3a4a\": rpc error: code = NotFound desc = could not find container \"af0a84299eb5b4e5924777dca47d3e315d31a9567d26e793c27fd6ba666c3a4a\": container with ID starting with af0a84299eb5b4e5924777dca47d3e315d31a9567d26e793c27fd6ba666c3a4a not found: ID does not exist" Sep 17 07:41:10.120165 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:10.120121 2578 scope.go:117] "RemoveContainer" containerID="549416d35482a8540070cb0eb67659efb9e4e92c4f44286b0d8964720329d750" Sep 17 07:41:10.120362 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:41:10.120345 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"549416d35482a8540070cb0eb67659efb9e4e92c4f44286b0d8964720329d750\": container with ID starting with 549416d35482a8540070cb0eb67659efb9e4e92c4f44286b0d8964720329d750 not found: ID does not exist" containerID="549416d35482a8540070cb0eb67659efb9e4e92c4f44286b0d8964720329d750" Sep 17 07:41:10.120405 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:10.120367 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"549416d35482a8540070cb0eb67659efb9e4e92c4f44286b0d8964720329d750"} err="failed to get container status \"549416d35482a8540070cb0eb67659efb9e4e92c4f44286b0d8964720329d750\": rpc error: code = NotFound desc = could not find container \"549416d35482a8540070cb0eb67659efb9e4e92c4f44286b0d8964720329d750\": container with ID starting with 549416d35482a8540070cb0eb67659efb9e4e92c4f44286b0d8964720329d750 not found: ID does not exist" Sep 17 07:41:10.120405 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:10.120381 2578 scope.go:117] "RemoveContainer" containerID="422adaa09dd13d416aa68bc192459f8c7e521870ed33b34ad6560953a1378b94" Sep 17 07:41:10.120616 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:10.120600 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"422adaa09dd13d416aa68bc192459f8c7e521870ed33b34ad6560953a1378b94"} err="failed to get container status \"422adaa09dd13d416aa68bc192459f8c7e521870ed33b34ad6560953a1378b94\": rpc error: code = NotFound desc = could not find container \"422adaa09dd13d416aa68bc192459f8c7e521870ed33b34ad6560953a1378b94\": container with ID starting with 422adaa09dd13d416aa68bc192459f8c7e521870ed33b34ad6560953a1378b94 not found: ID does not exist" Sep 17 07:41:10.120657 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:10.120616 2578 scope.go:117] "RemoveContainer" containerID="b5dff01f644fcde1c0230dfc52529290b7e809b8ed5b740cec4e022423549cb9" Sep 17 07:41:10.120842 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:10.120822 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5dff01f644fcde1c0230dfc52529290b7e809b8ed5b740cec4e022423549cb9"} err="failed to get container status \"b5dff01f644fcde1c0230dfc52529290b7e809b8ed5b740cec4e022423549cb9\": rpc error: code = NotFound desc = could not find container \"b5dff01f644fcde1c0230dfc52529290b7e809b8ed5b740cec4e022423549cb9\": container with ID starting with b5dff01f644fcde1c0230dfc52529290b7e809b8ed5b740cec4e022423549cb9 not found: ID does not exist" Sep 17 07:41:10.120976 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:10.120844 2578 scope.go:117] "RemoveContainer" containerID="b4bb042dd05c3651e1163c94999ebab47df20f05f61e5a36ab8b3056dbc2ec38" Sep 17 07:41:10.121106 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:10.121087 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4bb042dd05c3651e1163c94999ebab47df20f05f61e5a36ab8b3056dbc2ec38"} err="failed to get container status \"b4bb042dd05c3651e1163c94999ebab47df20f05f61e5a36ab8b3056dbc2ec38\": rpc error: code = NotFound desc = could not find container \"b4bb042dd05c3651e1163c94999ebab47df20f05f61e5a36ab8b3056dbc2ec38\": container with ID starting with b4bb042dd05c3651e1163c94999ebab47df20f05f61e5a36ab8b3056dbc2ec38 not found: ID does not exist" Sep 17 07:41:10.121155 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:10.121107 2578 scope.go:117] "RemoveContainer" containerID="2309e2fa5d7c715a45900d9fa64c731a066425b5bb3558c09c935da79dd962a7" Sep 17 07:41:10.121321 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:10.121302 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2309e2fa5d7c715a45900d9fa64c731a066425b5bb3558c09c935da79dd962a7"} err="failed to get container status \"2309e2fa5d7c715a45900d9fa64c731a066425b5bb3558c09c935da79dd962a7\": rpc error: code = NotFound desc = could not find container \"2309e2fa5d7c715a45900d9fa64c731a066425b5bb3558c09c935da79dd962a7\": container with ID starting with 2309e2fa5d7c715a45900d9fa64c731a066425b5bb3558c09c935da79dd962a7 not found: ID does not exist" Sep 17 07:41:10.121387 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:10.121323 2578 scope.go:117] "RemoveContainer" containerID="af0a84299eb5b4e5924777dca47d3e315d31a9567d26e793c27fd6ba666c3a4a" Sep 17 07:41:10.121538 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:10.121516 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af0a84299eb5b4e5924777dca47d3e315d31a9567d26e793c27fd6ba666c3a4a"} err="failed to get container status \"af0a84299eb5b4e5924777dca47d3e315d31a9567d26e793c27fd6ba666c3a4a\": rpc error: code = NotFound desc = could not find container \"af0a84299eb5b4e5924777dca47d3e315d31a9567d26e793c27fd6ba666c3a4a\": container with ID starting with af0a84299eb5b4e5924777dca47d3e315d31a9567d26e793c27fd6ba666c3a4a not found: ID does not exist" Sep 17 07:41:10.121583 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:10.121539 2578 scope.go:117] "RemoveContainer" containerID="549416d35482a8540070cb0eb67659efb9e4e92c4f44286b0d8964720329d750" Sep 17 07:41:10.121733 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:10.121716 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"549416d35482a8540070cb0eb67659efb9e4e92c4f44286b0d8964720329d750"} err="failed to get container status \"549416d35482a8540070cb0eb67659efb9e4e92c4f44286b0d8964720329d750\": rpc error: code = NotFound desc = could not find container \"549416d35482a8540070cb0eb67659efb9e4e92c4f44286b0d8964720329d750\": container with ID starting with 549416d35482a8540070cb0eb67659efb9e4e92c4f44286b0d8964720329d750 not found: ID does not exist" Sep 17 07:41:10.628951 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:10.628916 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1feb527a-10d8-4bc8-be30-52a9a4634c33" path="/var/lib/kubelet/pods/1feb527a-10d8-4bc8-be30-52a9a4634c33/volumes" Sep 17 07:41:10.629520 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:10.629506 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ebb0937-dac3-40c9-a155-24e756265163" path="/var/lib/kubelet/pods/6ebb0937-dac3-40c9-a155-24e756265163/volumes" Sep 17 07:41:22.545737 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:22.545703 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/postgres-864846d4f5-phl98"] Sep 17 07:41:22.546316 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:22.546087 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1feb527a-10d8-4bc8-be30-52a9a4634c33" containerName="ui" Sep 17 07:41:22.546316 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:22.546104 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1feb527a-10d8-4bc8-be30-52a9a4634c33" containerName="ui" Sep 17 07:41:22.546316 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:22.546115 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ebb0937-dac3-40c9-a155-24e756265163" containerName="ui" Sep 17 07:41:22.546316 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:22.546120 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ebb0937-dac3-40c9-a155-24e756265163" containerName="ui" Sep 17 07:41:22.546316 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:22.546128 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1feb527a-10d8-4bc8-be30-52a9a4634c33" containerName="offline" Sep 17 07:41:22.546316 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:22.546134 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1feb527a-10d8-4bc8-be30-52a9a4634c33" containerName="offline" Sep 17 07:41:22.546316 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:22.546141 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ebb0937-dac3-40c9-a155-24e756265163" containerName="online" Sep 17 07:41:22.546316 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:22.546146 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ebb0937-dac3-40c9-a155-24e756265163" containerName="online" Sep 17 07:41:22.546316 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:22.546152 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1feb527a-10d8-4bc8-be30-52a9a4634c33" containerName="feast-apply" Sep 17 07:41:22.546316 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:22.546158 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1feb527a-10d8-4bc8-be30-52a9a4634c33" containerName="feast-apply" Sep 17 07:41:22.546316 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:22.546167 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1feb527a-10d8-4bc8-be30-52a9a4634c33" containerName="feast-init" Sep 17 07:41:22.546316 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:22.546172 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1feb527a-10d8-4bc8-be30-52a9a4634c33" containerName="feast-init" Sep 17 07:41:22.546316 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:22.546178 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ebb0937-dac3-40c9-a155-24e756265163" containerName="offline" Sep 17 07:41:22.546316 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:22.546183 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ebb0937-dac3-40c9-a155-24e756265163" containerName="offline" Sep 17 07:41:22.546316 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:22.546191 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1feb527a-10d8-4bc8-be30-52a9a4634c33" containerName="online" Sep 17 07:41:22.546316 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:22.546196 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1feb527a-10d8-4bc8-be30-52a9a4634c33" containerName="online" Sep 17 07:41:22.546316 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:22.546206 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ebb0937-dac3-40c9-a155-24e756265163" containerName="feast-apply" Sep 17 07:41:22.546316 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:22.546211 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ebb0937-dac3-40c9-a155-24e756265163" containerName="feast-apply" Sep 17 07:41:22.546316 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:22.546218 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1feb527a-10d8-4bc8-be30-52a9a4634c33" containerName="registry" Sep 17 07:41:22.546316 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:22.546223 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1feb527a-10d8-4bc8-be30-52a9a4634c33" containerName="registry" Sep 17 07:41:22.546316 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:22.546229 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ebb0937-dac3-40c9-a155-24e756265163" containerName="feast-init" Sep 17 07:41:22.546316 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:22.546235 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ebb0937-dac3-40c9-a155-24e756265163" containerName="feast-init" Sep 17 07:41:22.546316 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:22.546278 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ebb0937-dac3-40c9-a155-24e756265163" containerName="online" Sep 17 07:41:22.546316 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:22.546287 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="1feb527a-10d8-4bc8-be30-52a9a4634c33" containerName="online" Sep 17 07:41:22.546316 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:22.546293 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ebb0937-dac3-40c9-a155-24e756265163" containerName="ui" Sep 17 07:41:22.546316 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:22.546299 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="1feb527a-10d8-4bc8-be30-52a9a4634c33" containerName="offline" Sep 17 07:41:22.546316 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:22.546304 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="1feb527a-10d8-4bc8-be30-52a9a4634c33" containerName="registry" Sep 17 07:41:22.546316 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:22.546310 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="1feb527a-10d8-4bc8-be30-52a9a4634c33" containerName="ui" Sep 17 07:41:22.546316 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:22.546316 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ebb0937-dac3-40c9-a155-24e756265163" containerName="offline" Sep 17 07:41:22.550826 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:22.550804 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-864846d4f5-phl98" Sep 17 07:41:22.553618 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:22.553592 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\"" Sep 17 07:41:22.554615 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:22.554595 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"default-dockercfg-rf4jl\"" Sep 17 07:41:22.554685 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:22.554595 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"postgres-secret\"" Sep 17 07:41:22.554762 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:22.554597 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\"" Sep 17 07:41:22.561902 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:22.561879 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/postgres-864846d4f5-phl98"] Sep 17 07:41:22.587540 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:22.587514 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/redis-69bbd798b6-n2qsf"] Sep 17 07:41:22.590365 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:22.590342 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/9107f72a-8b2b-4616-b8ca-0e689bd449de-postgresdata\") pod \"postgres-864846d4f5-phl98\" (UID: \"9107f72a-8b2b-4616-b8ca-0e689bd449de\") " pod="test-ns-feast/postgres-864846d4f5-phl98" Sep 17 07:41:22.590447 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:22.590377 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4l4vs\" (UniqueName: \"kubernetes.io/projected/9107f72a-8b2b-4616-b8ca-0e689bd449de-kube-api-access-4l4vs\") pod \"postgres-864846d4f5-phl98\" (UID: \"9107f72a-8b2b-4616-b8ca-0e689bd449de\") " pod="test-ns-feast/postgres-864846d4f5-phl98" Sep 17 07:41:22.590670 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:22.590657 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/redis-69bbd798b6-n2qsf" Sep 17 07:41:22.603717 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:22.603695 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/redis-69bbd798b6-n2qsf"] Sep 17 07:41:22.691646 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:22.691620 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/9107f72a-8b2b-4616-b8ca-0e689bd449de-postgresdata\") pod \"postgres-864846d4f5-phl98\" (UID: \"9107f72a-8b2b-4616-b8ca-0e689bd449de\") " pod="test-ns-feast/postgres-864846d4f5-phl98" Sep 17 07:41:22.691770 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:22.691653 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4l4vs\" (UniqueName: \"kubernetes.io/projected/9107f72a-8b2b-4616-b8ca-0e689bd449de-kube-api-access-4l4vs\") pod \"postgres-864846d4f5-phl98\" (UID: \"9107f72a-8b2b-4616-b8ca-0e689bd449de\") " pod="test-ns-feast/postgres-864846d4f5-phl98" Sep 17 07:41:22.691770 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:22.691714 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stf7s\" (UniqueName: \"kubernetes.io/projected/89d5c464-c239-487a-9bcf-6046b90900db-kube-api-access-stf7s\") pod \"redis-69bbd798b6-n2qsf\" (UID: \"89d5c464-c239-487a-9bcf-6046b90900db\") " pod="test-ns-feast/redis-69bbd798b6-n2qsf" Sep 17 07:41:22.692038 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:22.692021 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/9107f72a-8b2b-4616-b8ca-0e689bd449de-postgresdata\") pod \"postgres-864846d4f5-phl98\" (UID: \"9107f72a-8b2b-4616-b8ca-0e689bd449de\") " pod="test-ns-feast/postgres-864846d4f5-phl98" Sep 17 07:41:22.700735 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:22.700719 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4l4vs\" (UniqueName: \"kubernetes.io/projected/9107f72a-8b2b-4616-b8ca-0e689bd449de-kube-api-access-4l4vs\") pod \"postgres-864846d4f5-phl98\" (UID: \"9107f72a-8b2b-4616-b8ca-0e689bd449de\") " pod="test-ns-feast/postgres-864846d4f5-phl98" Sep 17 07:41:22.792992 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:22.792969 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stf7s\" (UniqueName: \"kubernetes.io/projected/89d5c464-c239-487a-9bcf-6046b90900db-kube-api-access-stf7s\") pod \"redis-69bbd798b6-n2qsf\" (UID: \"89d5c464-c239-487a-9bcf-6046b90900db\") " pod="test-ns-feast/redis-69bbd798b6-n2qsf" Sep 17 07:41:22.804177 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:22.804127 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-stf7s\" (UniqueName: \"kubernetes.io/projected/89d5c464-c239-487a-9bcf-6046b90900db-kube-api-access-stf7s\") pod \"redis-69bbd798b6-n2qsf\" (UID: \"89d5c464-c239-487a-9bcf-6046b90900db\") " pod="test-ns-feast/redis-69bbd798b6-n2qsf" Sep 17 07:41:22.866484 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:22.866461 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-864846d4f5-phl98" Sep 17 07:41:22.899671 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:22.899646 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/redis-69bbd798b6-n2qsf" Sep 17 07:41:23.140807 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:23.140780 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/postgres-864846d4f5-phl98"] Sep 17 07:41:23.142111 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:41:23.142086 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9107f72a_8b2b_4616_b8ca_0e689bd449de.slice/crio-98442abfcecd0a4fb8f39a937959a5801fd193d418484ac944b7f8cc84474e97 WatchSource:0}: Error finding container 98442abfcecd0a4fb8f39a937959a5801fd193d418484ac944b7f8cc84474e97: Status 404 returned error can't find the container with id 98442abfcecd0a4fb8f39a937959a5801fd193d418484ac944b7f8cc84474e97 Sep 17 07:41:23.143804 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:23.143789 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 17 07:41:23.178341 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:23.178318 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/redis-69bbd798b6-n2qsf"] Sep 17 07:41:23.179574 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:41:23.179553 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod89d5c464_c239_487a_9bcf_6046b90900db.slice/crio-6b643e8b0d276892f3860e96847b32bc4ab54b3d7bedae4b67f3de75a49bc6d1 WatchSource:0}: Error finding container 6b643e8b0d276892f3860e96847b32bc4ab54b3d7bedae4b67f3de75a49bc6d1: Status 404 returned error can't find the container with id 6b643e8b0d276892f3860e96847b32bc4ab54b3d7bedae4b67f3de75a49bc6d1 Sep 17 07:41:24.102945 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:24.102898 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/redis-69bbd798b6-n2qsf" event={"ID":"89d5c464-c239-487a-9bcf-6046b90900db","Type":"ContainerStarted","Data":"6b643e8b0d276892f3860e96847b32bc4ab54b3d7bedae4b67f3de75a49bc6d1"} Sep 17 07:41:24.104501 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:24.104471 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-864846d4f5-phl98" event={"ID":"9107f72a-8b2b-4616-b8ca-0e689bd449de","Type":"ContainerStarted","Data":"98442abfcecd0a4fb8f39a937959a5801fd193d418484ac944b7f8cc84474e97"} Sep 17 07:41:25.111162 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:25.111080 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/redis-69bbd798b6-n2qsf" event={"ID":"89d5c464-c239-487a-9bcf-6046b90900db","Type":"ContainerStarted","Data":"3f07a7a715e68502c8e31d4c39bdd2512ea98d8644bb7d4bd50c2cb46e08fff0"} Sep 17 07:41:25.135601 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:25.135551 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/redis-69bbd798b6-n2qsf" podStartSLOduration=1.599606574 podStartE2EDuration="3.135537526s" totalImagesPullingTime="1.535930952s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 07:41:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 07:41:25.135162448 +0000 UTC m=+829.016196807" watchObservedRunningTime="2026-09-17 07:41:25.135537526 +0000 UTC m=+829.016571884" Sep 17 07:41:28.125394 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:28.125353 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-864846d4f5-phl98" event={"ID":"9107f72a-8b2b-4616-b8ca-0e689bd449de","Type":"ContainerStarted","Data":"d4d85dc90007febfd6bfb8fb6afbd3654e4820d4c272985f66f616a54a3c31c0"} Sep 17 07:41:28.156687 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:28.156637 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/postgres-864846d4f5-phl98" podStartSLOduration=1.743095946 podStartE2EDuration="6.156621965s" totalImagesPullingTime="4.413526019s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 07:41:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 07:41:28.155768165 +0000 UTC m=+832.036802525" watchObservedRunningTime="2026-09-17 07:41:28.156621965 +0000 UTC m=+832.037656323" Sep 17 07:41:31.533377 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:31.533344 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-credit-scoring-d4cdcd4c9-4ztck"] Sep 17 07:41:31.539555 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:31.539533 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-d4cdcd4c9-4ztck" Sep 17 07:41:31.548950 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:31.548930 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-offline-tls\"" Sep 17 07:41:31.549061 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:31.548950 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-registry-tls\"" Sep 17 07:41:31.549167 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:31.549146 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-intra-comm\"" Sep 17 07:41:31.549378 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:31.549365 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-online-tls\"" Sep 17 07:41:31.549606 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:31.549418 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-dockercfg-htz7h\"" Sep 17 07:41:31.563549 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:31.563528 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-credit-scoring-d4cdcd4c9-4ztck"] Sep 17 07:41:31.668547 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:31.668522 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/cb6324c3-f17e-4c82-88f7-aadd168dd79a-registry-tls\") pod \"feast-credit-scoring-d4cdcd4c9-4ztck\" (UID: \"cb6324c3-f17e-4c82-88f7-aadd168dd79a\") " pod="test-ns-feast/feast-credit-scoring-d4cdcd4c9-4ztck" Sep 17 07:41:31.668547 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:31.668550 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/cb6324c3-f17e-4c82-88f7-aadd168dd79a-feast-data\") pod \"feast-credit-scoring-d4cdcd4c9-4ztck\" (UID: \"cb6324c3-f17e-4c82-88f7-aadd168dd79a\") " pod="test-ns-feast/feast-credit-scoring-d4cdcd4c9-4ztck" Sep 17 07:41:31.668729 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:31.668569 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2qbp\" (UniqueName: \"kubernetes.io/projected/cb6324c3-f17e-4c82-88f7-aadd168dd79a-kube-api-access-n2qbp\") pod \"feast-credit-scoring-d4cdcd4c9-4ztck\" (UID: \"cb6324c3-f17e-4c82-88f7-aadd168dd79a\") " pod="test-ns-feast/feast-credit-scoring-d4cdcd4c9-4ztck" Sep 17 07:41:31.668729 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:31.668588 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/cb6324c3-f17e-4c82-88f7-aadd168dd79a-offline-tls\") pod \"feast-credit-scoring-d4cdcd4c9-4ztck\" (UID: \"cb6324c3-f17e-4c82-88f7-aadd168dd79a\") " pod="test-ns-feast/feast-credit-scoring-d4cdcd4c9-4ztck" Sep 17 07:41:31.668729 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:31.668695 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/cb6324c3-f17e-4c82-88f7-aadd168dd79a-online-tls\") pod \"feast-credit-scoring-d4cdcd4c9-4ztck\" (UID: \"cb6324c3-f17e-4c82-88f7-aadd168dd79a\") " pod="test-ns-feast/feast-credit-scoring-d4cdcd4c9-4ztck" Sep 17 07:41:31.769185 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:31.769156 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/cb6324c3-f17e-4c82-88f7-aadd168dd79a-online-tls\") pod \"feast-credit-scoring-d4cdcd4c9-4ztck\" (UID: \"cb6324c3-f17e-4c82-88f7-aadd168dd79a\") " pod="test-ns-feast/feast-credit-scoring-d4cdcd4c9-4ztck" Sep 17 07:41:31.769341 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:31.769195 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/cb6324c3-f17e-4c82-88f7-aadd168dd79a-registry-tls\") pod \"feast-credit-scoring-d4cdcd4c9-4ztck\" (UID: \"cb6324c3-f17e-4c82-88f7-aadd168dd79a\") " pod="test-ns-feast/feast-credit-scoring-d4cdcd4c9-4ztck" Sep 17 07:41:31.769341 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:31.769212 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/cb6324c3-f17e-4c82-88f7-aadd168dd79a-feast-data\") pod \"feast-credit-scoring-d4cdcd4c9-4ztck\" (UID: \"cb6324c3-f17e-4c82-88f7-aadd168dd79a\") " pod="test-ns-feast/feast-credit-scoring-d4cdcd4c9-4ztck" Sep 17 07:41:31.769341 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:31.769228 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2qbp\" (UniqueName: \"kubernetes.io/projected/cb6324c3-f17e-4c82-88f7-aadd168dd79a-kube-api-access-n2qbp\") pod \"feast-credit-scoring-d4cdcd4c9-4ztck\" (UID: \"cb6324c3-f17e-4c82-88f7-aadd168dd79a\") " pod="test-ns-feast/feast-credit-scoring-d4cdcd4c9-4ztck" Sep 17 07:41:31.769341 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:31.769250 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/cb6324c3-f17e-4c82-88f7-aadd168dd79a-offline-tls\") pod \"feast-credit-scoring-d4cdcd4c9-4ztck\" (UID: \"cb6324c3-f17e-4c82-88f7-aadd168dd79a\") " pod="test-ns-feast/feast-credit-scoring-d4cdcd4c9-4ztck" Sep 17 07:41:31.769341 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:41:31.769320 2578 secret.go:189] Couldn't get secret test-ns-feast/feast-credit-scoring-registry-tls: secret "feast-credit-scoring-registry-tls" not found Sep 17 07:41:31.769696 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:41:31.769349 2578 secret.go:189] Couldn't get secret test-ns-feast/feast-credit-scoring-offline-tls: secret "feast-credit-scoring-offline-tls" not found Sep 17 07:41:31.769696 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:41:31.769388 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb6324c3-f17e-4c82-88f7-aadd168dd79a-registry-tls podName:cb6324c3-f17e-4c82-88f7-aadd168dd79a nodeName:}" failed. No retries permitted until 2026-09-17 07:41:32.26936759 +0000 UTC m=+836.150401941 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/cb6324c3-f17e-4c82-88f7-aadd168dd79a-registry-tls") pod "feast-credit-scoring-d4cdcd4c9-4ztck" (UID: "cb6324c3-f17e-4c82-88f7-aadd168dd79a") : secret "feast-credit-scoring-registry-tls" not found Sep 17 07:41:31.769696 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:41:31.769408 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb6324c3-f17e-4c82-88f7-aadd168dd79a-offline-tls podName:cb6324c3-f17e-4c82-88f7-aadd168dd79a nodeName:}" failed. No retries permitted until 2026-09-17 07:41:32.269399668 +0000 UTC m=+836.150434013 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/cb6324c3-f17e-4c82-88f7-aadd168dd79a-offline-tls") pod "feast-credit-scoring-d4cdcd4c9-4ztck" (UID: "cb6324c3-f17e-4c82-88f7-aadd168dd79a") : secret "feast-credit-scoring-offline-tls" not found Sep 17 07:41:31.769696 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:31.769605 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/cb6324c3-f17e-4c82-88f7-aadd168dd79a-feast-data\") pod \"feast-credit-scoring-d4cdcd4c9-4ztck\" (UID: \"cb6324c3-f17e-4c82-88f7-aadd168dd79a\") " pod="test-ns-feast/feast-credit-scoring-d4cdcd4c9-4ztck" Sep 17 07:41:31.771636 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:31.771620 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/cb6324c3-f17e-4c82-88f7-aadd168dd79a-online-tls\") pod \"feast-credit-scoring-d4cdcd4c9-4ztck\" (UID: \"cb6324c3-f17e-4c82-88f7-aadd168dd79a\") " pod="test-ns-feast/feast-credit-scoring-d4cdcd4c9-4ztck" Sep 17 07:41:31.784048 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:31.783997 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2qbp\" (UniqueName: \"kubernetes.io/projected/cb6324c3-f17e-4c82-88f7-aadd168dd79a-kube-api-access-n2qbp\") pod \"feast-credit-scoring-d4cdcd4c9-4ztck\" (UID: \"cb6324c3-f17e-4c82-88f7-aadd168dd79a\") " pod="test-ns-feast/feast-credit-scoring-d4cdcd4c9-4ztck" Sep 17 07:41:32.274583 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:32.274556 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/cb6324c3-f17e-4c82-88f7-aadd168dd79a-registry-tls\") pod \"feast-credit-scoring-d4cdcd4c9-4ztck\" (UID: \"cb6324c3-f17e-4c82-88f7-aadd168dd79a\") " pod="test-ns-feast/feast-credit-scoring-d4cdcd4c9-4ztck" Sep 17 07:41:32.274765 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:32.274591 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/cb6324c3-f17e-4c82-88f7-aadd168dd79a-offline-tls\") pod \"feast-credit-scoring-d4cdcd4c9-4ztck\" (UID: \"cb6324c3-f17e-4c82-88f7-aadd168dd79a\") " pod="test-ns-feast/feast-credit-scoring-d4cdcd4c9-4ztck" Sep 17 07:41:32.276932 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:32.276910 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/cb6324c3-f17e-4c82-88f7-aadd168dd79a-offline-tls\") pod \"feast-credit-scoring-d4cdcd4c9-4ztck\" (UID: \"cb6324c3-f17e-4c82-88f7-aadd168dd79a\") " pod="test-ns-feast/feast-credit-scoring-d4cdcd4c9-4ztck" Sep 17 07:41:32.277094 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:32.277075 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/cb6324c3-f17e-4c82-88f7-aadd168dd79a-registry-tls\") pod \"feast-credit-scoring-d4cdcd4c9-4ztck\" (UID: \"cb6324c3-f17e-4c82-88f7-aadd168dd79a\") " pod="test-ns-feast/feast-credit-scoring-d4cdcd4c9-4ztck" Sep 17 07:41:32.449015 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:32.448979 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-d4cdcd4c9-4ztck" Sep 17 07:41:32.731198 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:32.731169 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-credit-scoring-d4cdcd4c9-4ztck"] Sep 17 07:41:32.732966 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:41:32.732928 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb6324c3_f17e_4c82_88f7_aadd168dd79a.slice/crio-c7d181bb3ce86efca6bb70a806e3d014701413dd44920a3c85941af86a9df21e WatchSource:0}: Error finding container c7d181bb3ce86efca6bb70a806e3d014701413dd44920a3c85941af86a9df21e: Status 404 returned error can't find the container with id c7d181bb3ce86efca6bb70a806e3d014701413dd44920a3c85941af86a9df21e Sep 17 07:41:33.146701 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:33.146670 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-d4cdcd4c9-4ztck" event={"ID":"cb6324c3-f17e-4c82-88f7-aadd168dd79a","Type":"ContainerStarted","Data":"9d5a7d400e298dda36e8c60153bb0d0b8b41ad20986f5c7c67e41a88948b05ef"} Sep 17 07:41:33.146877 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:33.146722 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-d4cdcd4c9-4ztck" event={"ID":"cb6324c3-f17e-4c82-88f7-aadd168dd79a","Type":"ContainerStarted","Data":"c7d181bb3ce86efca6bb70a806e3d014701413dd44920a3c85941af86a9df21e"} Sep 17 07:41:35.159274 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:35.159236 2578 generic.go:356] "Generic (PLEG): container finished" podID="cb6324c3-f17e-4c82-88f7-aadd168dd79a" containerID="9d5a7d400e298dda36e8c60153bb0d0b8b41ad20986f5c7c67e41a88948b05ef" exitCode=0 Sep 17 07:41:35.159627 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:35.159306 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-d4cdcd4c9-4ztck" event={"ID":"cb6324c3-f17e-4c82-88f7-aadd168dd79a","Type":"ContainerDied","Data":"9d5a7d400e298dda36e8c60153bb0d0b8b41ad20986f5c7c67e41a88948b05ef"} Sep 17 07:41:36.165033 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:36.165000 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-d4cdcd4c9-4ztck" event={"ID":"cb6324c3-f17e-4c82-88f7-aadd168dd79a","Type":"ContainerStarted","Data":"310e13f16e168574479c1df3428eb77cf9629350ca4b13dc94354ea69f856b55"} Sep 17 07:41:42.191516 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:42.191483 2578 generic.go:356] "Generic (PLEG): container finished" podID="cb6324c3-f17e-4c82-88f7-aadd168dd79a" containerID="310e13f16e168574479c1df3428eb77cf9629350ca4b13dc94354ea69f856b55" exitCode=0 Sep 17 07:41:42.191956 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:42.191530 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-d4cdcd4c9-4ztck" event={"ID":"cb6324c3-f17e-4c82-88f7-aadd168dd79a","Type":"ContainerDied","Data":"310e13f16e168574479c1df3428eb77cf9629350ca4b13dc94354ea69f856b55"} Sep 17 07:41:42.201326 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:42.201306 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-intra-comm\"" Sep 17 07:41:43.201160 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:43.201112 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-d4cdcd4c9-4ztck" event={"ID":"cb6324c3-f17e-4c82-88f7-aadd168dd79a","Type":"ContainerStarted","Data":"8f8b3fa977096b20616fc144cd702b7cc2aca538fa96f708742fd7173c76a00b"} Sep 17 07:41:43.201631 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:43.201192 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-d4cdcd4c9-4ztck" event={"ID":"cb6324c3-f17e-4c82-88f7-aadd168dd79a","Type":"ContainerStarted","Data":"2eaff1f7dae756a563d5687527e9f6aa19977a47efe4681e839f3f10d938dd35"} Sep 17 07:41:43.201631 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:43.201223 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-d4cdcd4c9-4ztck" event={"ID":"cb6324c3-f17e-4c82-88f7-aadd168dd79a","Type":"ContainerStarted","Data":"d618624f0b9e196f5cead766658f65d25083afa93478532e2075a59b913f66dc"} Sep 17 07:41:43.233550 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:43.233484 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-credit-scoring-d4cdcd4c9-4ztck" podStartSLOduration=12.233461573 podStartE2EDuration="12.233461573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 07:41:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 07:41:43.230144893 +0000 UTC m=+847.111179283" watchObservedRunningTime="2026-09-17 07:41:43.233461573 +0000 UTC m=+847.114495933" Sep 17 07:41:44.449736 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:44.449684 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-credit-scoring-d4cdcd4c9-4ztck" Sep 17 07:41:44.450248 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:44.449801 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-credit-scoring-d4cdcd4c9-4ztck" Sep 17 07:41:44.450248 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:44.449843 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-credit-scoring-d4cdcd4c9-4ztck" Sep 17 07:41:44.451883 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:44.451834 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-d4cdcd4c9-4ztck" podUID="cb6324c3-f17e-4c82-88f7-aadd168dd79a" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.32:6571: connect: connection refused" Sep 17 07:41:44.452032 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:44.451912 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-d4cdcd4c9-4ztck" podUID="cb6324c3-f17e-4c82-88f7-aadd168dd79a" containerName="online" probeResult="failure" output="Get \"https://10.132.0.32:6567/health\": dial tcp 10.132.0.32:6567: connect: connection refused" Sep 17 07:41:44.452032 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:44.451834 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-d4cdcd4c9-4ztck" podUID="cb6324c3-f17e-4c82-88f7-aadd168dd79a" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.32:8816: connect: connection refused" Sep 17 07:41:47.450212 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:47.450164 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-d4cdcd4c9-4ztck" podUID="cb6324c3-f17e-4c82-88f7-aadd168dd79a" containerName="online" probeResult="failure" output="Get \"https://10.132.0.32:6567/health\": dial tcp 10.132.0.32:6567: connect: connection refused" Sep 17 07:41:47.450716 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:47.450162 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-d4cdcd4c9-4ztck" podUID="cb6324c3-f17e-4c82-88f7-aadd168dd79a" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.32:8816: connect: connection refused" Sep 17 07:41:47.450716 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:47.450161 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-d4cdcd4c9-4ztck" podUID="cb6324c3-f17e-4c82-88f7-aadd168dd79a" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.32:6571: connect: connection refused" Sep 17 07:41:50.450324 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:50.450270 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-d4cdcd4c9-4ztck" podUID="cb6324c3-f17e-4c82-88f7-aadd168dd79a" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.32:6571: connect: connection refused" Sep 17 07:41:50.450786 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:50.450279 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-d4cdcd4c9-4ztck" podUID="cb6324c3-f17e-4c82-88f7-aadd168dd79a" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.32:8816: connect: connection refused" Sep 17 07:41:50.450786 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:50.450295 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-d4cdcd4c9-4ztck" podUID="cb6324c3-f17e-4c82-88f7-aadd168dd79a" containerName="online" probeResult="failure" output="Get \"https://10.132.0.32:6567/health\": dial tcp 10.132.0.32:6567: connect: connection refused" Sep 17 07:41:52.449578 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:52.449539 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-d4cdcd4c9-4ztck" Sep 17 07:41:52.450093 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:52.449612 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-d4cdcd4c9-4ztck" Sep 17 07:41:52.450093 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:52.449636 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-d4cdcd4c9-4ztck" Sep 17 07:41:53.449459 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:53.449400 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-d4cdcd4c9-4ztck" podUID="cb6324c3-f17e-4c82-88f7-aadd168dd79a" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.32:6571: connect: connection refused" Sep 17 07:41:53.450073 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:53.450047 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-credit-scoring-d4cdcd4c9-4ztck" Sep 17 07:41:53.451038 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:53.451013 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-credit-scoring-d4cdcd4c9-4ztck" Sep 17 07:41:53.455540 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:53.455517 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-credit-scoring-d4cdcd4c9-4ztck" Sep 17 07:41:54.266233 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:54.266200 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-credit-scoring-d4cdcd4c9-4ztck" Sep 17 07:41:56.450122 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:56.450091 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-credit-scoring-d4cdcd4c9-4ztck" Sep 17 07:41:57.274747 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:41:57.274714 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-credit-scoring-d4cdcd4c9-4ztck" Sep 17 07:43:03.656118 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:43:03.656082 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-credit-scoring-d4cdcd4c9-4ztck"] Sep 17 07:43:03.658418 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:43:03.656334 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-credit-scoring-d4cdcd4c9-4ztck" podUID="cb6324c3-f17e-4c82-88f7-aadd168dd79a" containerName="registry" containerID="cri-o://d618624f0b9e196f5cead766658f65d25083afa93478532e2075a59b913f66dc" gracePeriod=30 Sep 17 07:43:03.658418 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:43:03.656343 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-credit-scoring-d4cdcd4c9-4ztck" podUID="cb6324c3-f17e-4c82-88f7-aadd168dd79a" containerName="offline" containerID="cri-o://8f8b3fa977096b20616fc144cd702b7cc2aca538fa96f708742fd7173c76a00b" gracePeriod=30 Sep 17 07:43:03.658418 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:43:03.656377 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-credit-scoring-d4cdcd4c9-4ztck" podUID="cb6324c3-f17e-4c82-88f7-aadd168dd79a" containerName="online" containerID="cri-o://2eaff1f7dae756a563d5687527e9f6aa19977a47efe4681e839f3f10d938dd35" gracePeriod=30 Sep 17 07:43:03.681932 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:43:03.681903 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/postgres-864846d4f5-phl98"] Sep 17 07:43:03.682094 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:43:03.682075 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/postgres-864846d4f5-phl98" podUID="9107f72a-8b2b-4616-b8ca-0e689bd449de" containerName="postgres" containerID="cri-o://d4d85dc90007febfd6bfb8fb6afbd3654e4820d4c272985f66f616a54a3c31c0" gracePeriod=30 Sep 17 07:43:03.691162 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:43:03.691131 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/redis-69bbd798b6-n2qsf"] Sep 17 07:43:03.691370 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:43:03.691324 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/redis-69bbd798b6-n2qsf" podUID="89d5c464-c239-487a-9bcf-6046b90900db" containerName="redis" containerID="cri-o://3f07a7a715e68502c8e31d4c39bdd2512ea98d8644bb7d4bd50c2cb46e08fff0" gracePeriod=30 Sep 17 07:43:03.947620 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:43:03.947594 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-864846d4f5-phl98" Sep 17 07:43:03.950496 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:43:03.950478 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/redis-69bbd798b6-n2qsf" Sep 17 07:43:04.138054 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:43:04.138009 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/9107f72a-8b2b-4616-b8ca-0e689bd449de-postgresdata\") pod \"9107f72a-8b2b-4616-b8ca-0e689bd449de\" (UID: \"9107f72a-8b2b-4616-b8ca-0e689bd449de\") " Sep 17 07:43:04.138252 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:43:04.138095 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4l4vs\" (UniqueName: \"kubernetes.io/projected/9107f72a-8b2b-4616-b8ca-0e689bd449de-kube-api-access-4l4vs\") pod \"9107f72a-8b2b-4616-b8ca-0e689bd449de\" (UID: \"9107f72a-8b2b-4616-b8ca-0e689bd449de\") " Sep 17 07:43:04.138252 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:43:04.138157 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-stf7s\" (UniqueName: \"kubernetes.io/projected/89d5c464-c239-487a-9bcf-6046b90900db-kube-api-access-stf7s\") pod \"89d5c464-c239-487a-9bcf-6046b90900db\" (UID: \"89d5c464-c239-487a-9bcf-6046b90900db\") " Sep 17 07:43:04.140234 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:43:04.140205 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89d5c464-c239-487a-9bcf-6046b90900db-kube-api-access-stf7s" (OuterVolumeSpecName: "kube-api-access-stf7s") pod "89d5c464-c239-487a-9bcf-6046b90900db" (UID: "89d5c464-c239-487a-9bcf-6046b90900db"). InnerVolumeSpecName "kube-api-access-stf7s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 07:43:04.140350 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:43:04.140270 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9107f72a-8b2b-4616-b8ca-0e689bd449de-kube-api-access-4l4vs" (OuterVolumeSpecName: "kube-api-access-4l4vs") pod "9107f72a-8b2b-4616-b8ca-0e689bd449de" (UID: "9107f72a-8b2b-4616-b8ca-0e689bd449de"). InnerVolumeSpecName "kube-api-access-4l4vs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 07:43:04.168264 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:43:04.168227 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9107f72a-8b2b-4616-b8ca-0e689bd449de-postgresdata" (OuterVolumeSpecName: "postgresdata") pod "9107f72a-8b2b-4616-b8ca-0e689bd449de" (UID: "9107f72a-8b2b-4616-b8ca-0e689bd449de"). InnerVolumeSpecName "postgresdata". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 07:43:04.238819 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:43:04.238736 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stf7s\" (UniqueName: \"kubernetes.io/projected/89d5c464-c239-487a-9bcf-6046b90900db-kube-api-access-stf7s\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Sep 17 07:43:04.238819 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:43:04.238768 2578 reconciler_common.go:299] "Volume detached for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/9107f72a-8b2b-4616-b8ca-0e689bd449de-postgresdata\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Sep 17 07:43:04.238819 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:43:04.238777 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4l4vs\" (UniqueName: \"kubernetes.io/projected/9107f72a-8b2b-4616-b8ca-0e689bd449de-kube-api-access-4l4vs\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Sep 17 07:43:04.262793 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:43:04.262755 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-d4cdcd4c9-4ztck" podUID="cb6324c3-f17e-4c82-88f7-aadd168dd79a" containerName="online" probeResult="failure" output="Get \"https://10.132.0.32:6567/health\": dial tcp 10.132.0.32:6567: connect: connection refused" Sep 17 07:43:04.551951 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:43:04.551843 2578 generic.go:356] "Generic (PLEG): container finished" podID="89d5c464-c239-487a-9bcf-6046b90900db" containerID="3f07a7a715e68502c8e31d4c39bdd2512ea98d8644bb7d4bd50c2cb46e08fff0" exitCode=0 Sep 17 07:43:04.551951 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:43:04.551914 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/redis-69bbd798b6-n2qsf" Sep 17 07:43:04.551951 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:43:04.551927 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/redis-69bbd798b6-n2qsf" event={"ID":"89d5c464-c239-487a-9bcf-6046b90900db","Type":"ContainerDied","Data":"3f07a7a715e68502c8e31d4c39bdd2512ea98d8644bb7d4bd50c2cb46e08fff0"} Sep 17 07:43:04.552201 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:43:04.551973 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/redis-69bbd798b6-n2qsf" event={"ID":"89d5c464-c239-487a-9bcf-6046b90900db","Type":"ContainerDied","Data":"6b643e8b0d276892f3860e96847b32bc4ab54b3d7bedae4b67f3de75a49bc6d1"} Sep 17 07:43:04.552201 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:43:04.551991 2578 scope.go:117] "RemoveContainer" containerID="3f07a7a715e68502c8e31d4c39bdd2512ea98d8644bb7d4bd50c2cb46e08fff0" Sep 17 07:43:04.553412 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:43:04.553387 2578 generic.go:356] "Generic (PLEG): container finished" podID="9107f72a-8b2b-4616-b8ca-0e689bd449de" containerID="d4d85dc90007febfd6bfb8fb6afbd3654e4820d4c272985f66f616a54a3c31c0" exitCode=0 Sep 17 07:43:04.553542 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:43:04.553420 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-864846d4f5-phl98" Sep 17 07:43:04.553542 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:43:04.553446 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-864846d4f5-phl98" event={"ID":"9107f72a-8b2b-4616-b8ca-0e689bd449de","Type":"ContainerDied","Data":"d4d85dc90007febfd6bfb8fb6afbd3654e4820d4c272985f66f616a54a3c31c0"} Sep 17 07:43:04.553542 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:43:04.553477 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-864846d4f5-phl98" event={"ID":"9107f72a-8b2b-4616-b8ca-0e689bd449de","Type":"ContainerDied","Data":"98442abfcecd0a4fb8f39a937959a5801fd193d418484ac944b7f8cc84474e97"} Sep 17 07:43:04.560949 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:43:04.560930 2578 scope.go:117] "RemoveContainer" containerID="3f07a7a715e68502c8e31d4c39bdd2512ea98d8644bb7d4bd50c2cb46e08fff0" Sep 17 07:43:04.561287 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:43:04.561269 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f07a7a715e68502c8e31d4c39bdd2512ea98d8644bb7d4bd50c2cb46e08fff0\": container with ID starting with 3f07a7a715e68502c8e31d4c39bdd2512ea98d8644bb7d4bd50c2cb46e08fff0 not found: ID does not exist" containerID="3f07a7a715e68502c8e31d4c39bdd2512ea98d8644bb7d4bd50c2cb46e08fff0" Sep 17 07:43:04.561366 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:43:04.561295 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f07a7a715e68502c8e31d4c39bdd2512ea98d8644bb7d4bd50c2cb46e08fff0"} err="failed to get container status \"3f07a7a715e68502c8e31d4c39bdd2512ea98d8644bb7d4bd50c2cb46e08fff0\": rpc error: code = NotFound desc = could not find container \"3f07a7a715e68502c8e31d4c39bdd2512ea98d8644bb7d4bd50c2cb46e08fff0\": container with ID starting with 3f07a7a715e68502c8e31d4c39bdd2512ea98d8644bb7d4bd50c2cb46e08fff0 not found: ID does not exist" Sep 17 07:43:04.561366 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:43:04.561312 2578 scope.go:117] "RemoveContainer" containerID="d4d85dc90007febfd6bfb8fb6afbd3654e4820d4c272985f66f616a54a3c31c0" Sep 17 07:43:04.570398 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:43:04.570376 2578 scope.go:117] "RemoveContainer" containerID="d4d85dc90007febfd6bfb8fb6afbd3654e4820d4c272985f66f616a54a3c31c0" Sep 17 07:43:04.570719 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:43:04.570699 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4d85dc90007febfd6bfb8fb6afbd3654e4820d4c272985f66f616a54a3c31c0\": container with ID starting with d4d85dc90007febfd6bfb8fb6afbd3654e4820d4c272985f66f616a54a3c31c0 not found: ID does not exist" containerID="d4d85dc90007febfd6bfb8fb6afbd3654e4820d4c272985f66f616a54a3c31c0" Sep 17 07:43:04.570763 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:43:04.570727 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4d85dc90007febfd6bfb8fb6afbd3654e4820d4c272985f66f616a54a3c31c0"} err="failed to get container status \"d4d85dc90007febfd6bfb8fb6afbd3654e4820d4c272985f66f616a54a3c31c0\": rpc error: code = NotFound desc = could not find container \"d4d85dc90007febfd6bfb8fb6afbd3654e4820d4c272985f66f616a54a3c31c0\": container with ID starting with d4d85dc90007febfd6bfb8fb6afbd3654e4820d4c272985f66f616a54a3c31c0 not found: ID does not exist" Sep 17 07:43:04.583107 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:43:04.583083 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/redis-69bbd798b6-n2qsf"] Sep 17 07:43:04.588458 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:43:04.588428 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/redis-69bbd798b6-n2qsf"] Sep 17 07:43:04.600282 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:43:04.600254 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/postgres-864846d4f5-phl98"] Sep 17 07:43:04.605411 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:43:04.605385 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/postgres-864846d4f5-phl98"] Sep 17 07:43:04.629546 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:43:04.629519 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="89d5c464-c239-487a-9bcf-6046b90900db" path="/var/lib/kubelet/pods/89d5c464-c239-487a-9bcf-6046b90900db/volumes" Sep 17 07:43:04.629895 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:43:04.629881 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9107f72a-8b2b-4616-b8ca-0e689bd449de" path="/var/lib/kubelet/pods/9107f72a-8b2b-4616-b8ca-0e689bd449de/volumes" Sep 17 07:43:05.559880 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:43:05.559828 2578 generic.go:356] "Generic (PLEG): container finished" podID="cb6324c3-f17e-4c82-88f7-aadd168dd79a" containerID="2eaff1f7dae756a563d5687527e9f6aa19977a47efe4681e839f3f10d938dd35" exitCode=0 Sep 17 07:43:05.560258 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:43:05.559901 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-d4cdcd4c9-4ztck" event={"ID":"cb6324c3-f17e-4c82-88f7-aadd168dd79a","Type":"ContainerDied","Data":"2eaff1f7dae756a563d5687527e9f6aa19977a47efe4681e839f3f10d938dd35"} Sep 17 07:43:14.262422 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:43:14.262380 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-d4cdcd4c9-4ztck" podUID="cb6324c3-f17e-4c82-88f7-aadd168dd79a" containerName="online" probeResult="failure" output="Get \"https://10.132.0.32:6567/health\": dial tcp 10.132.0.32:6567: connect: connection refused" Sep 17 07:43:24.262208 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:43:24.262165 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-d4cdcd4c9-4ztck" podUID="cb6324c3-f17e-4c82-88f7-aadd168dd79a" containerName="online" probeResult="failure" output="Get \"https://10.132.0.32:6567/health\": dial tcp 10.132.0.32:6567: connect: connection refused" Sep 17 07:43:24.262670 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:43:24.262286 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-d4cdcd4c9-4ztck" Sep 17 07:43:34.308743 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:43:34.308715 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-d4cdcd4c9-4ztck" Sep 17 07:43:34.341500 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:43:34.341465 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/cb6324c3-f17e-4c82-88f7-aadd168dd79a-offline-tls\") pod \"cb6324c3-f17e-4c82-88f7-aadd168dd79a\" (UID: \"cb6324c3-f17e-4c82-88f7-aadd168dd79a\") " Sep 17 07:43:34.341671 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:43:34.341546 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/cb6324c3-f17e-4c82-88f7-aadd168dd79a-registry-tls\") pod \"cb6324c3-f17e-4c82-88f7-aadd168dd79a\" (UID: \"cb6324c3-f17e-4c82-88f7-aadd168dd79a\") " Sep 17 07:43:34.341753 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:43:34.341706 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/cb6324c3-f17e-4c82-88f7-aadd168dd79a-feast-data\") pod \"cb6324c3-f17e-4c82-88f7-aadd168dd79a\" (UID: \"cb6324c3-f17e-4c82-88f7-aadd168dd79a\") " Sep 17 07:43:34.341810 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:43:34.341790 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/cb6324c3-f17e-4c82-88f7-aadd168dd79a-online-tls\") pod \"cb6324c3-f17e-4c82-88f7-aadd168dd79a\" (UID: \"cb6324c3-f17e-4c82-88f7-aadd168dd79a\") " Sep 17 07:43:34.341895 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:43:34.341821 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n2qbp\" (UniqueName: \"kubernetes.io/projected/cb6324c3-f17e-4c82-88f7-aadd168dd79a-kube-api-access-n2qbp\") pod \"cb6324c3-f17e-4c82-88f7-aadd168dd79a\" (UID: \"cb6324c3-f17e-4c82-88f7-aadd168dd79a\") " Sep 17 07:43:34.344307 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:43:34.344262 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb6324c3-f17e-4c82-88f7-aadd168dd79a-online-tls" (OuterVolumeSpecName: "online-tls") pod "cb6324c3-f17e-4c82-88f7-aadd168dd79a" (UID: "cb6324c3-f17e-4c82-88f7-aadd168dd79a"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 07:43:34.344432 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:43:34.344276 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb6324c3-f17e-4c82-88f7-aadd168dd79a-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "cb6324c3-f17e-4c82-88f7-aadd168dd79a" (UID: "cb6324c3-f17e-4c82-88f7-aadd168dd79a"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 07:43:34.344432 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:43:34.344389 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb6324c3-f17e-4c82-88f7-aadd168dd79a-kube-api-access-n2qbp" (OuterVolumeSpecName: "kube-api-access-n2qbp") pod "cb6324c3-f17e-4c82-88f7-aadd168dd79a" (UID: "cb6324c3-f17e-4c82-88f7-aadd168dd79a"). InnerVolumeSpecName "kube-api-access-n2qbp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 07:43:34.344781 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:43:34.344758 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb6324c3-f17e-4c82-88f7-aadd168dd79a-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "cb6324c3-f17e-4c82-88f7-aadd168dd79a" (UID: "cb6324c3-f17e-4c82-88f7-aadd168dd79a"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 07:43:34.350358 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:43:34.350326 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb6324c3-f17e-4c82-88f7-aadd168dd79a-feast-data" (OuterVolumeSpecName: "feast-data") pod "cb6324c3-f17e-4c82-88f7-aadd168dd79a" (UID: "cb6324c3-f17e-4c82-88f7-aadd168dd79a"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 07:43:34.443408 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:43:34.443324 2578 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/cb6324c3-f17e-4c82-88f7-aadd168dd79a-offline-tls\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Sep 17 07:43:34.443408 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:43:34.443354 2578 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/cb6324c3-f17e-4c82-88f7-aadd168dd79a-registry-tls\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Sep 17 07:43:34.443408 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:43:34.443365 2578 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/cb6324c3-f17e-4c82-88f7-aadd168dd79a-feast-data\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Sep 17 07:43:34.443408 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:43:34.443373 2578 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/cb6324c3-f17e-4c82-88f7-aadd168dd79a-online-tls\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Sep 17 07:43:34.443408 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:43:34.443382 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2qbp\" (UniqueName: \"kubernetes.io/projected/cb6324c3-f17e-4c82-88f7-aadd168dd79a-kube-api-access-n2qbp\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Sep 17 07:43:34.680135 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:43:34.680100 2578 generic.go:356] "Generic (PLEG): container finished" podID="cb6324c3-f17e-4c82-88f7-aadd168dd79a" containerID="8f8b3fa977096b20616fc144cd702b7cc2aca538fa96f708742fd7173c76a00b" exitCode=137 Sep 17 07:43:34.680135 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:43:34.680127 2578 generic.go:356] "Generic (PLEG): container finished" podID="cb6324c3-f17e-4c82-88f7-aadd168dd79a" containerID="d618624f0b9e196f5cead766658f65d25083afa93478532e2075a59b913f66dc" exitCode=137 Sep 17 07:43:34.680384 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:43:34.680160 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-d4cdcd4c9-4ztck" event={"ID":"cb6324c3-f17e-4c82-88f7-aadd168dd79a","Type":"ContainerDied","Data":"8f8b3fa977096b20616fc144cd702b7cc2aca538fa96f708742fd7173c76a00b"} Sep 17 07:43:34.680384 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:43:34.680167 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-d4cdcd4c9-4ztck" Sep 17 07:43:34.680384 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:43:34.680205 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-d4cdcd4c9-4ztck" event={"ID":"cb6324c3-f17e-4c82-88f7-aadd168dd79a","Type":"ContainerDied","Data":"d618624f0b9e196f5cead766658f65d25083afa93478532e2075a59b913f66dc"} Sep 17 07:43:34.680384 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:43:34.680223 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-d4cdcd4c9-4ztck" event={"ID":"cb6324c3-f17e-4c82-88f7-aadd168dd79a","Type":"ContainerDied","Data":"c7d181bb3ce86efca6bb70a806e3d014701413dd44920a3c85941af86a9df21e"} Sep 17 07:43:34.680384 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:43:34.680247 2578 scope.go:117] "RemoveContainer" containerID="8f8b3fa977096b20616fc144cd702b7cc2aca538fa96f708742fd7173c76a00b" Sep 17 07:43:34.688675 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:43:34.688654 2578 scope.go:117] "RemoveContainer" containerID="2eaff1f7dae756a563d5687527e9f6aa19977a47efe4681e839f3f10d938dd35" Sep 17 07:43:34.696226 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:43:34.696211 2578 scope.go:117] "RemoveContainer" containerID="d618624f0b9e196f5cead766658f65d25083afa93478532e2075a59b913f66dc" Sep 17 07:43:34.699547 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:43:34.699526 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-credit-scoring-d4cdcd4c9-4ztck"] Sep 17 07:43:34.704143 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:43:34.704118 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-credit-scoring-d4cdcd4c9-4ztck"] Sep 17 07:43:34.705276 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:43:34.705259 2578 scope.go:117] "RemoveContainer" containerID="310e13f16e168574479c1df3428eb77cf9629350ca4b13dc94354ea69f856b55" Sep 17 07:43:34.714010 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:43:34.713990 2578 scope.go:117] "RemoveContainer" containerID="9d5a7d400e298dda36e8c60153bb0d0b8b41ad20986f5c7c67e41a88948b05ef" Sep 17 07:43:34.721185 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:43:34.721169 2578 scope.go:117] "RemoveContainer" containerID="8f8b3fa977096b20616fc144cd702b7cc2aca538fa96f708742fd7173c76a00b" Sep 17 07:43:34.721442 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:43:34.721419 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f8b3fa977096b20616fc144cd702b7cc2aca538fa96f708742fd7173c76a00b\": container with ID starting with 8f8b3fa977096b20616fc144cd702b7cc2aca538fa96f708742fd7173c76a00b not found: ID does not exist" containerID="8f8b3fa977096b20616fc144cd702b7cc2aca538fa96f708742fd7173c76a00b" Sep 17 07:43:34.721494 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:43:34.721453 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f8b3fa977096b20616fc144cd702b7cc2aca538fa96f708742fd7173c76a00b"} err="failed to get container status \"8f8b3fa977096b20616fc144cd702b7cc2aca538fa96f708742fd7173c76a00b\": rpc error: code = NotFound desc = could not find container \"8f8b3fa977096b20616fc144cd702b7cc2aca538fa96f708742fd7173c76a00b\": container with ID starting with 8f8b3fa977096b20616fc144cd702b7cc2aca538fa96f708742fd7173c76a00b not found: ID does not exist" Sep 17 07:43:34.721494 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:43:34.721473 2578 scope.go:117] "RemoveContainer" containerID="2eaff1f7dae756a563d5687527e9f6aa19977a47efe4681e839f3f10d938dd35" Sep 17 07:43:34.721686 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:43:34.721669 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2eaff1f7dae756a563d5687527e9f6aa19977a47efe4681e839f3f10d938dd35\": container with ID starting with 2eaff1f7dae756a563d5687527e9f6aa19977a47efe4681e839f3f10d938dd35 not found: ID does not exist" containerID="2eaff1f7dae756a563d5687527e9f6aa19977a47efe4681e839f3f10d938dd35" Sep 17 07:43:34.721757 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:43:34.721697 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2eaff1f7dae756a563d5687527e9f6aa19977a47efe4681e839f3f10d938dd35"} err="failed to get container status \"2eaff1f7dae756a563d5687527e9f6aa19977a47efe4681e839f3f10d938dd35\": rpc error: code = NotFound desc = could not find container \"2eaff1f7dae756a563d5687527e9f6aa19977a47efe4681e839f3f10d938dd35\": container with ID starting with 2eaff1f7dae756a563d5687527e9f6aa19977a47efe4681e839f3f10d938dd35 not found: ID does not exist" Sep 17 07:43:34.721757 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:43:34.721718 2578 scope.go:117] "RemoveContainer" containerID="d618624f0b9e196f5cead766658f65d25083afa93478532e2075a59b913f66dc" Sep 17 07:43:34.722024 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:43:34.721956 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d618624f0b9e196f5cead766658f65d25083afa93478532e2075a59b913f66dc\": container with ID starting with d618624f0b9e196f5cead766658f65d25083afa93478532e2075a59b913f66dc not found: ID does not exist" containerID="d618624f0b9e196f5cead766658f65d25083afa93478532e2075a59b913f66dc" Sep 17 07:43:34.722024 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:43:34.721980 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d618624f0b9e196f5cead766658f65d25083afa93478532e2075a59b913f66dc"} err="failed to get container status \"d618624f0b9e196f5cead766658f65d25083afa93478532e2075a59b913f66dc\": rpc error: code = NotFound desc = could not find container \"d618624f0b9e196f5cead766658f65d25083afa93478532e2075a59b913f66dc\": container with ID starting with d618624f0b9e196f5cead766658f65d25083afa93478532e2075a59b913f66dc not found: ID does not exist" Sep 17 07:43:34.722024 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:43:34.721998 2578 scope.go:117] "RemoveContainer" containerID="310e13f16e168574479c1df3428eb77cf9629350ca4b13dc94354ea69f856b55" Sep 17 07:43:34.722260 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:43:34.722243 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"310e13f16e168574479c1df3428eb77cf9629350ca4b13dc94354ea69f856b55\": container with ID starting with 310e13f16e168574479c1df3428eb77cf9629350ca4b13dc94354ea69f856b55 not found: ID does not exist" containerID="310e13f16e168574479c1df3428eb77cf9629350ca4b13dc94354ea69f856b55" Sep 17 07:43:34.722304 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:43:34.722264 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"310e13f16e168574479c1df3428eb77cf9629350ca4b13dc94354ea69f856b55"} err="failed to get container status \"310e13f16e168574479c1df3428eb77cf9629350ca4b13dc94354ea69f856b55\": rpc error: code = NotFound desc = could not find container \"310e13f16e168574479c1df3428eb77cf9629350ca4b13dc94354ea69f856b55\": container with ID starting with 310e13f16e168574479c1df3428eb77cf9629350ca4b13dc94354ea69f856b55 not found: ID does not exist" Sep 17 07:43:34.722304 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:43:34.722277 2578 scope.go:117] "RemoveContainer" containerID="9d5a7d400e298dda36e8c60153bb0d0b8b41ad20986f5c7c67e41a88948b05ef" Sep 17 07:43:34.722482 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:43:34.722465 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d5a7d400e298dda36e8c60153bb0d0b8b41ad20986f5c7c67e41a88948b05ef\": container with ID starting with 9d5a7d400e298dda36e8c60153bb0d0b8b41ad20986f5c7c67e41a88948b05ef not found: ID does not exist" containerID="9d5a7d400e298dda36e8c60153bb0d0b8b41ad20986f5c7c67e41a88948b05ef" Sep 17 07:43:34.722524 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:43:34.722487 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d5a7d400e298dda36e8c60153bb0d0b8b41ad20986f5c7c67e41a88948b05ef"} err="failed to get container status \"9d5a7d400e298dda36e8c60153bb0d0b8b41ad20986f5c7c67e41a88948b05ef\": rpc error: code = NotFound desc = could not find container \"9d5a7d400e298dda36e8c60153bb0d0b8b41ad20986f5c7c67e41a88948b05ef\": container with ID starting with 9d5a7d400e298dda36e8c60153bb0d0b8b41ad20986f5c7c67e41a88948b05ef not found: ID does not exist" Sep 17 07:43:34.722524 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:43:34.722501 2578 scope.go:117] "RemoveContainer" containerID="8f8b3fa977096b20616fc144cd702b7cc2aca538fa96f708742fd7173c76a00b" Sep 17 07:43:34.722742 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:43:34.722726 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f8b3fa977096b20616fc144cd702b7cc2aca538fa96f708742fd7173c76a00b"} err="failed to get container status \"8f8b3fa977096b20616fc144cd702b7cc2aca538fa96f708742fd7173c76a00b\": rpc error: code = NotFound desc = could not find container \"8f8b3fa977096b20616fc144cd702b7cc2aca538fa96f708742fd7173c76a00b\": container with ID starting with 8f8b3fa977096b20616fc144cd702b7cc2aca538fa96f708742fd7173c76a00b not found: ID does not exist" Sep 17 07:43:34.722795 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:43:34.722743 2578 scope.go:117] "RemoveContainer" containerID="2eaff1f7dae756a563d5687527e9f6aa19977a47efe4681e839f3f10d938dd35" Sep 17 07:43:34.723023 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:43:34.722993 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2eaff1f7dae756a563d5687527e9f6aa19977a47efe4681e839f3f10d938dd35"} err="failed to get container status \"2eaff1f7dae756a563d5687527e9f6aa19977a47efe4681e839f3f10d938dd35\": rpc error: code = NotFound desc = could not find container \"2eaff1f7dae756a563d5687527e9f6aa19977a47efe4681e839f3f10d938dd35\": container with ID starting with 2eaff1f7dae756a563d5687527e9f6aa19977a47efe4681e839f3f10d938dd35 not found: ID does not exist" Sep 17 07:43:34.723101 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:43:34.723024 2578 scope.go:117] "RemoveContainer" containerID="d618624f0b9e196f5cead766658f65d25083afa93478532e2075a59b913f66dc" Sep 17 07:43:34.723253 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:43:34.723237 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d618624f0b9e196f5cead766658f65d25083afa93478532e2075a59b913f66dc"} err="failed to get container status \"d618624f0b9e196f5cead766658f65d25083afa93478532e2075a59b913f66dc\": rpc error: code = NotFound desc = could not find container \"d618624f0b9e196f5cead766658f65d25083afa93478532e2075a59b913f66dc\": container with ID starting with d618624f0b9e196f5cead766658f65d25083afa93478532e2075a59b913f66dc not found: ID does not exist" Sep 17 07:43:34.723297 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:43:34.723254 2578 scope.go:117] "RemoveContainer" containerID="310e13f16e168574479c1df3428eb77cf9629350ca4b13dc94354ea69f856b55" Sep 17 07:43:34.723462 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:43:34.723448 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"310e13f16e168574479c1df3428eb77cf9629350ca4b13dc94354ea69f856b55"} err="failed to get container status \"310e13f16e168574479c1df3428eb77cf9629350ca4b13dc94354ea69f856b55\": rpc error: code = NotFound desc = could not find container \"310e13f16e168574479c1df3428eb77cf9629350ca4b13dc94354ea69f856b55\": container with ID starting with 310e13f16e168574479c1df3428eb77cf9629350ca4b13dc94354ea69f856b55 not found: ID does not exist" Sep 17 07:43:34.723505 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:43:34.723461 2578 scope.go:117] "RemoveContainer" containerID="9d5a7d400e298dda36e8c60153bb0d0b8b41ad20986f5c7c67e41a88948b05ef" Sep 17 07:43:34.723638 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:43:34.723624 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d5a7d400e298dda36e8c60153bb0d0b8b41ad20986f5c7c67e41a88948b05ef"} err="failed to get container status \"9d5a7d400e298dda36e8c60153bb0d0b8b41ad20986f5c7c67e41a88948b05ef\": rpc error: code = NotFound desc = could not find container \"9d5a7d400e298dda36e8c60153bb0d0b8b41ad20986f5c7c67e41a88948b05ef\": container with ID starting with 9d5a7d400e298dda36e8c60153bb0d0b8b41ad20986f5c7c67e41a88948b05ef not found: ID does not exist" Sep 17 07:43:35.263511 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:43:35.263469 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-d4cdcd4c9-4ztck" podUID="cb6324c3-f17e-4c82-88f7-aadd168dd79a" containerName="online" probeResult="failure" output="Get \"https://10.132.0.32:6567/health\": context deadline exceeded" Sep 17 07:43:36.630054 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:43:36.630019 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb6324c3-f17e-4c82-88f7-aadd168dd79a" path="/var/lib/kubelet/pods/cb6324c3-f17e-4c82-88f7-aadd168dd79a/volumes" Sep 17 07:46:41.145151 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:46:41.145121 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-shdm5"] Sep 17 07:46:41.145606 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:46:41.145414 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb6324c3-f17e-4c82-88f7-aadd168dd79a" containerName="feast-apply" Sep 17 07:46:41.145606 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:46:41.145424 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb6324c3-f17e-4c82-88f7-aadd168dd79a" containerName="feast-apply" Sep 17 07:46:41.145606 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:46:41.145437 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89d5c464-c239-487a-9bcf-6046b90900db" containerName="redis" Sep 17 07:46:41.145606 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:46:41.145443 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="89d5c464-c239-487a-9bcf-6046b90900db" containerName="redis" Sep 17 07:46:41.145606 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:46:41.145451 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb6324c3-f17e-4c82-88f7-aadd168dd79a" containerName="feast-init" Sep 17 07:46:41.145606 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:46:41.145456 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb6324c3-f17e-4c82-88f7-aadd168dd79a" containerName="feast-init" Sep 17 07:46:41.145606 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:46:41.145462 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb6324c3-f17e-4c82-88f7-aadd168dd79a" containerName="registry" Sep 17 07:46:41.145606 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:46:41.145467 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb6324c3-f17e-4c82-88f7-aadd168dd79a" containerName="registry" Sep 17 07:46:41.145606 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:46:41.145478 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9107f72a-8b2b-4616-b8ca-0e689bd449de" containerName="postgres" Sep 17 07:46:41.145606 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:46:41.145483 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9107f72a-8b2b-4616-b8ca-0e689bd449de" containerName="postgres" Sep 17 07:46:41.145606 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:46:41.145488 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb6324c3-f17e-4c82-88f7-aadd168dd79a" containerName="online" Sep 17 07:46:41.145606 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:46:41.145495 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb6324c3-f17e-4c82-88f7-aadd168dd79a" containerName="online" Sep 17 07:46:41.145606 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:46:41.145502 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb6324c3-f17e-4c82-88f7-aadd168dd79a" containerName="offline" Sep 17 07:46:41.145606 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:46:41.145508 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb6324c3-f17e-4c82-88f7-aadd168dd79a" containerName="offline" Sep 17 07:46:41.145606 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:46:41.145562 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb6324c3-f17e-4c82-88f7-aadd168dd79a" containerName="online" Sep 17 07:46:41.145606 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:46:41.145570 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="89d5c464-c239-487a-9bcf-6046b90900db" containerName="redis" Sep 17 07:46:41.145606 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:46:41.145576 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb6324c3-f17e-4c82-88f7-aadd168dd79a" containerName="registry" Sep 17 07:46:41.145606 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:46:41.145581 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb6324c3-f17e-4c82-88f7-aadd168dd79a" containerName="offline" Sep 17 07:46:41.145606 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:46:41.145587 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="9107f72a-8b2b-4616-b8ca-0e689bd449de" containerName="postgres" Sep 17 07:46:41.148196 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:46:41.148177 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-69bbd798b6-shdm5" Sep 17 07:46:41.151194 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:46:41.151173 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"kube-root-ca.crt\"" Sep 17 07:46:41.151289 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:46:41.151180 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"default-dockercfg-vxcnk\"" Sep 17 07:46:41.153144 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:46:41.153121 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"openshift-service-ca.crt\"" Sep 17 07:46:41.156414 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:46:41.156390 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xml7\" (UniqueName: \"kubernetes.io/projected/1434c345-9bfd-4329-ad94-6ce063d1466c-kube-api-access-7xml7\") pod \"redis-69bbd798b6-shdm5\" (UID: \"1434c345-9bfd-4329-ad94-6ce063d1466c\") " pod="test-ns-feast-rest/redis-69bbd798b6-shdm5" Sep 17 07:46:41.196528 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:46:41.196504 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-shdm5"] Sep 17 07:46:41.257167 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:46:41.257136 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xml7\" (UniqueName: \"kubernetes.io/projected/1434c345-9bfd-4329-ad94-6ce063d1466c-kube-api-access-7xml7\") pod \"redis-69bbd798b6-shdm5\" (UID: \"1434c345-9bfd-4329-ad94-6ce063d1466c\") " pod="test-ns-feast-rest/redis-69bbd798b6-shdm5" Sep 17 07:46:41.269280 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:46:41.269257 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xml7\" (UniqueName: \"kubernetes.io/projected/1434c345-9bfd-4329-ad94-6ce063d1466c-kube-api-access-7xml7\") pod \"redis-69bbd798b6-shdm5\" (UID: \"1434c345-9bfd-4329-ad94-6ce063d1466c\") " pod="test-ns-feast-rest/redis-69bbd798b6-shdm5" Sep 17 07:46:41.474456 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:46:41.474367 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-69bbd798b6-shdm5" Sep 17 07:46:41.756393 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:46:41.756322 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-shdm5"] Sep 17 07:46:41.763983 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:46:41.763952 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1434c345_9bfd_4329_ad94_6ce063d1466c.slice/crio-084c42e8a4a5fad2b8f92def77f1a5733c2f60a3a5e0b06b6c9a2183a1e44a99 WatchSource:0}: Error finding container 084c42e8a4a5fad2b8f92def77f1a5733c2f60a3a5e0b06b6c9a2183a1e44a99: Status 404 returned error can't find the container with id 084c42e8a4a5fad2b8f92def77f1a5733c2f60a3a5e0b06b6c9a2183a1e44a99 Sep 17 07:46:42.400673 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:46:42.400632 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-69bbd798b6-shdm5" event={"ID":"1434c345-9bfd-4329-ad94-6ce063d1466c","Type":"ContainerStarted","Data":"c9b158a1085d6bc2f677ba5a59f1f6a9df0d9b36ef865a75b13c2949cea25f68"} Sep 17 07:46:42.400673 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:46:42.400674 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-69bbd798b6-shdm5" event={"ID":"1434c345-9bfd-4329-ad94-6ce063d1466c","Type":"ContainerStarted","Data":"084c42e8a4a5fad2b8f92def77f1a5733c2f60a3a5e0b06b6c9a2183a1e44a99"} Sep 17 07:46:42.422690 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:46:42.422640 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast-rest/redis-69bbd798b6-shdm5" podStartSLOduration=1.422625747 podStartE2EDuration="1.422625747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 07:46:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 07:46:42.420114198 +0000 UTC m=+1146.301148558" watchObservedRunningTime="2026-09-17 07:46:42.422625747 +0000 UTC m=+1146.303660105" Sep 17 07:46:46.652821 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:46:46.652779 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-nrbvg"] Sep 17 07:46:46.656142 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:46:46.656116 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-864846d4f5-nrbvg" Sep 17 07:46:46.661141 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:46:46.661115 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"postgres-secret\"" Sep 17 07:46:46.689958 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:46:46.689922 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/ce9fe81c-010a-425b-9c82-3d00fd3accd1-postgresdata\") pod \"postgres-864846d4f5-nrbvg\" (UID: \"ce9fe81c-010a-425b-9c82-3d00fd3accd1\") " pod="test-ns-feast-rest/postgres-864846d4f5-nrbvg" Sep 17 07:46:46.689958 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:46:46.689955 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75wbj\" (UniqueName: \"kubernetes.io/projected/ce9fe81c-010a-425b-9c82-3d00fd3accd1-kube-api-access-75wbj\") pod \"postgres-864846d4f5-nrbvg\" (UID: \"ce9fe81c-010a-425b-9c82-3d00fd3accd1\") " pod="test-ns-feast-rest/postgres-864846d4f5-nrbvg" Sep 17 07:46:46.691201 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:46:46.691169 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-nrbvg"] Sep 17 07:46:46.791005 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:46:46.790968 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/ce9fe81c-010a-425b-9c82-3d00fd3accd1-postgresdata\") pod \"postgres-864846d4f5-nrbvg\" (UID: \"ce9fe81c-010a-425b-9c82-3d00fd3accd1\") " pod="test-ns-feast-rest/postgres-864846d4f5-nrbvg" Sep 17 07:46:46.791005 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:46:46.791007 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75wbj\" (UniqueName: \"kubernetes.io/projected/ce9fe81c-010a-425b-9c82-3d00fd3accd1-kube-api-access-75wbj\") pod \"postgres-864846d4f5-nrbvg\" (UID: \"ce9fe81c-010a-425b-9c82-3d00fd3accd1\") " pod="test-ns-feast-rest/postgres-864846d4f5-nrbvg" Sep 17 07:46:46.791327 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:46:46.791305 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/ce9fe81c-010a-425b-9c82-3d00fd3accd1-postgresdata\") pod \"postgres-864846d4f5-nrbvg\" (UID: \"ce9fe81c-010a-425b-9c82-3d00fd3accd1\") " pod="test-ns-feast-rest/postgres-864846d4f5-nrbvg" Sep 17 07:46:46.808218 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:46:46.808185 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-75wbj\" (UniqueName: \"kubernetes.io/projected/ce9fe81c-010a-425b-9c82-3d00fd3accd1-kube-api-access-75wbj\") pod \"postgres-864846d4f5-nrbvg\" (UID: \"ce9fe81c-010a-425b-9c82-3d00fd3accd1\") " pod="test-ns-feast-rest/postgres-864846d4f5-nrbvg" Sep 17 07:46:46.966636 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:46:46.966545 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-864846d4f5-nrbvg" Sep 17 07:46:47.265464 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:46:47.265375 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-nrbvg"] Sep 17 07:46:47.268542 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:46:47.268517 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podce9fe81c_010a_425b_9c82_3d00fd3accd1.slice/crio-8f82251362232f048b49086c64a5cac966c3163c03b122ac7141ebb8b82675f4 WatchSource:0}: Error finding container 8f82251362232f048b49086c64a5cac966c3163c03b122ac7141ebb8b82675f4: Status 404 returned error can't find the container with id 8f82251362232f048b49086c64a5cac966c3163c03b122ac7141ebb8b82675f4 Sep 17 07:46:47.421727 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:46:47.421680 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-864846d4f5-nrbvg" event={"ID":"ce9fe81c-010a-425b-9c82-3d00fd3accd1","Type":"ContainerStarted","Data":"bc51100efd6e0bb2598d4e3d9a14f3ab1fc36b15c196a76d5017d074cc75f5fd"} Sep 17 07:46:47.421727 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:46:47.421723 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-864846d4f5-nrbvg" event={"ID":"ce9fe81c-010a-425b-9c82-3d00fd3accd1","Type":"ContainerStarted","Data":"8f82251362232f048b49086c64a5cac966c3163c03b122ac7141ebb8b82675f4"} Sep 17 07:46:47.453740 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:46:47.453681 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast-rest/postgres-864846d4f5-nrbvg" podStartSLOduration=1.453663943 podStartE2EDuration="1.453663943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 07:46:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 07:46:47.453155657 +0000 UTC m=+1151.334190016" watchObservedRunningTime="2026-09-17 07:46:47.453663943 +0000 UTC m=+1151.334698302" Sep 17 07:46:52.410529 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:46:52.410489 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-764cf876b9-cnxsn"] Sep 17 07:46:52.413970 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:46:52.413948 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-764cf876b9-cnxsn" Sep 17 07:46:52.417506 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:46:52.417471 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-registry-tls\"" Sep 17 07:46:52.417633 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:46:52.417548 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-intra-comm\"" Sep 17 07:46:52.417633 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:46:52.417564 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-dockercfg-fq6kf\"" Sep 17 07:46:52.417897 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:46:52.417879 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-online-tls\"" Sep 17 07:46:52.427998 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:46:52.427975 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/1c8fa530-cb80-465c-9583-8c69b0bdf80a-feast-data\") pod \"feast-credit-scoring-764cf876b9-cnxsn\" (UID: \"1c8fa530-cb80-465c-9583-8c69b0bdf80a\") " pod="test-ns-feast-rest/feast-credit-scoring-764cf876b9-cnxsn" Sep 17 07:46:52.428131 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:46:52.428114 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/1c8fa530-cb80-465c-9583-8c69b0bdf80a-online-tls\") pod \"feast-credit-scoring-764cf876b9-cnxsn\" (UID: \"1c8fa530-cb80-465c-9583-8c69b0bdf80a\") " pod="test-ns-feast-rest/feast-credit-scoring-764cf876b9-cnxsn" Sep 17 07:46:52.428192 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:46:52.428157 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g99xf\" (UniqueName: \"kubernetes.io/projected/1c8fa530-cb80-465c-9583-8c69b0bdf80a-kube-api-access-g99xf\") pod \"feast-credit-scoring-764cf876b9-cnxsn\" (UID: \"1c8fa530-cb80-465c-9583-8c69b0bdf80a\") " pod="test-ns-feast-rest/feast-credit-scoring-764cf876b9-cnxsn" Sep 17 07:46:52.428308 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:46:52.428199 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/1c8fa530-cb80-465c-9583-8c69b0bdf80a-registry-tls\") pod \"feast-credit-scoring-764cf876b9-cnxsn\" (UID: \"1c8fa530-cb80-465c-9583-8c69b0bdf80a\") " pod="test-ns-feast-rest/feast-credit-scoring-764cf876b9-cnxsn" Sep 17 07:46:52.429747 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:46:52.429726 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-764cf876b9-cnxsn"] Sep 17 07:46:52.529025 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:46:52.528994 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/1c8fa530-cb80-465c-9583-8c69b0bdf80a-online-tls\") pod \"feast-credit-scoring-764cf876b9-cnxsn\" (UID: \"1c8fa530-cb80-465c-9583-8c69b0bdf80a\") " pod="test-ns-feast-rest/feast-credit-scoring-764cf876b9-cnxsn" Sep 17 07:46:52.529025 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:46:52.529026 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g99xf\" (UniqueName: \"kubernetes.io/projected/1c8fa530-cb80-465c-9583-8c69b0bdf80a-kube-api-access-g99xf\") pod \"feast-credit-scoring-764cf876b9-cnxsn\" (UID: \"1c8fa530-cb80-465c-9583-8c69b0bdf80a\") " pod="test-ns-feast-rest/feast-credit-scoring-764cf876b9-cnxsn" Sep 17 07:46:52.529268 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:46:52.529059 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/1c8fa530-cb80-465c-9583-8c69b0bdf80a-registry-tls\") pod \"feast-credit-scoring-764cf876b9-cnxsn\" (UID: \"1c8fa530-cb80-465c-9583-8c69b0bdf80a\") " pod="test-ns-feast-rest/feast-credit-scoring-764cf876b9-cnxsn" Sep 17 07:46:52.529268 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:46:52.529094 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/1c8fa530-cb80-465c-9583-8c69b0bdf80a-feast-data\") pod \"feast-credit-scoring-764cf876b9-cnxsn\" (UID: \"1c8fa530-cb80-465c-9583-8c69b0bdf80a\") " pod="test-ns-feast-rest/feast-credit-scoring-764cf876b9-cnxsn" Sep 17 07:46:52.529625 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:46:52.529598 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/1c8fa530-cb80-465c-9583-8c69b0bdf80a-feast-data\") pod \"feast-credit-scoring-764cf876b9-cnxsn\" (UID: \"1c8fa530-cb80-465c-9583-8c69b0bdf80a\") " pod="test-ns-feast-rest/feast-credit-scoring-764cf876b9-cnxsn" Sep 17 07:46:52.531567 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:46:52.531546 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/1c8fa530-cb80-465c-9583-8c69b0bdf80a-online-tls\") pod \"feast-credit-scoring-764cf876b9-cnxsn\" (UID: \"1c8fa530-cb80-465c-9583-8c69b0bdf80a\") " pod="test-ns-feast-rest/feast-credit-scoring-764cf876b9-cnxsn" Sep 17 07:46:52.531671 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:46:52.531602 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/1c8fa530-cb80-465c-9583-8c69b0bdf80a-registry-tls\") pod \"feast-credit-scoring-764cf876b9-cnxsn\" (UID: \"1c8fa530-cb80-465c-9583-8c69b0bdf80a\") " pod="test-ns-feast-rest/feast-credit-scoring-764cf876b9-cnxsn" Sep 17 07:46:52.537762 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:46:52.537738 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g99xf\" (UniqueName: \"kubernetes.io/projected/1c8fa530-cb80-465c-9583-8c69b0bdf80a-kube-api-access-g99xf\") pod \"feast-credit-scoring-764cf876b9-cnxsn\" (UID: \"1c8fa530-cb80-465c-9583-8c69b0bdf80a\") " pod="test-ns-feast-rest/feast-credit-scoring-764cf876b9-cnxsn" Sep 17 07:46:52.724050 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:46:52.723965 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-764cf876b9-cnxsn" Sep 17 07:46:53.011307 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:46:53.011228 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-764cf876b9-cnxsn"] Sep 17 07:46:53.014306 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:46:53.014275 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c8fa530_cb80_465c_9583_8c69b0bdf80a.slice/crio-f954d429634cde989a61aa46094bfbbcb30c83a77b24a58680e7f43c7bfe918e WatchSource:0}: Error finding container f954d429634cde989a61aa46094bfbbcb30c83a77b24a58680e7f43c7bfe918e: Status 404 returned error can't find the container with id f954d429634cde989a61aa46094bfbbcb30c83a77b24a58680e7f43c7bfe918e Sep 17 07:46:53.450428 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:46:53.450392 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-764cf876b9-cnxsn" event={"ID":"1c8fa530-cb80-465c-9583-8c69b0bdf80a","Type":"ContainerStarted","Data":"135239f3c42da7440eba6fe469d0a28ae35b382b7409405c3ac51adcf1c7b9f9"} Sep 17 07:46:53.450807 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:46:53.450442 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-764cf876b9-cnxsn" event={"ID":"1c8fa530-cb80-465c-9583-8c69b0bdf80a","Type":"ContainerStarted","Data":"f954d429634cde989a61aa46094bfbbcb30c83a77b24a58680e7f43c7bfe918e"} Sep 17 07:46:55.459296 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:46:55.459262 2578 generic.go:356] "Generic (PLEG): container finished" podID="1c8fa530-cb80-465c-9583-8c69b0bdf80a" containerID="135239f3c42da7440eba6fe469d0a28ae35b382b7409405c3ac51adcf1c7b9f9" exitCode=0 Sep 17 07:46:55.459688 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:46:55.459317 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-764cf876b9-cnxsn" event={"ID":"1c8fa530-cb80-465c-9583-8c69b0bdf80a","Type":"ContainerDied","Data":"135239f3c42da7440eba6fe469d0a28ae35b382b7409405c3ac51adcf1c7b9f9"} Sep 17 07:46:56.465155 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:46:56.465114 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-764cf876b9-cnxsn" event={"ID":"1c8fa530-cb80-465c-9583-8c69b0bdf80a","Type":"ContainerStarted","Data":"f83f64e263a8d9c722406e9ab6e30e85324a0ce1d8cd542e609d3603231cdf28"} Sep 17 07:47:03.496199 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:47:03.496164 2578 generic.go:356] "Generic (PLEG): container finished" podID="1c8fa530-cb80-465c-9583-8c69b0bdf80a" containerID="f83f64e263a8d9c722406e9ab6e30e85324a0ce1d8cd542e609d3603231cdf28" exitCode=0 Sep 17 07:47:03.496199 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:47:03.496214 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-764cf876b9-cnxsn" event={"ID":"1c8fa530-cb80-465c-9583-8c69b0bdf80a","Type":"ContainerDied","Data":"f83f64e263a8d9c722406e9ab6e30e85324a0ce1d8cd542e609d3603231cdf28"} Sep 17 07:47:04.504379 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:47:04.504347 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-764cf876b9-cnxsn" event={"ID":"1c8fa530-cb80-465c-9583-8c69b0bdf80a","Type":"ContainerStarted","Data":"b86e8b47dcbb20cc4fff1171578a93a297fbb23a59968c457a6204ab242423d7"} Sep 17 07:47:04.504769 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:47:04.504399 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-764cf876b9-cnxsn" event={"ID":"1c8fa530-cb80-465c-9583-8c69b0bdf80a","Type":"ContainerStarted","Data":"d453662c58e640ef0bd007a91585ce96e28fba2ec4d7666dd3de52ffe1913a8e"} Sep 17 07:47:04.537300 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:47:04.537238 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast-rest/feast-credit-scoring-764cf876b9-cnxsn" podStartSLOduration=12.537217763 podStartE2EDuration="12.537217763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 07:46:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 07:47:04.534785976 +0000 UTC m=+1168.415820357" watchObservedRunningTime="2026-09-17 07:47:04.537217763 +0000 UTC m=+1168.418252125" Sep 17 07:47:04.725165 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:47:04.725124 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-credit-scoring-764cf876b9-cnxsn" Sep 17 07:47:04.725347 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:47:04.725206 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-credit-scoring-764cf876b9-cnxsn" Sep 17 07:47:04.726760 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:47:04.726721 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-764cf876b9-cnxsn" podUID="1c8fa530-cb80-465c-9583-8c69b0bdf80a" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.35:6571: connect: connection refused" Sep 17 07:47:04.726912 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:47:04.726770 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-764cf876b9-cnxsn" podUID="1c8fa530-cb80-465c-9583-8c69b0bdf80a" containerName="online" probeResult="failure" output="Get \"https://10.132.0.35:6567/health\": dial tcp 10.132.0.35:6567: connect: connection refused" Sep 17 07:47:07.724392 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:47:07.724340 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-764cf876b9-cnxsn" podUID="1c8fa530-cb80-465c-9583-8c69b0bdf80a" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.35:6571: connect: connection refused" Sep 17 07:47:07.724763 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:47:07.724349 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-764cf876b9-cnxsn" podUID="1c8fa530-cb80-465c-9583-8c69b0bdf80a" containerName="online" probeResult="failure" output="Get \"https://10.132.0.35:6567/health\": dial tcp 10.132.0.35:6567: connect: connection refused" Sep 17 07:47:10.724676 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:47:10.724625 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-764cf876b9-cnxsn" podUID="1c8fa530-cb80-465c-9583-8c69b0bdf80a" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.35:6571: connect: connection refused" Sep 17 07:47:10.730753 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:47:10.730727 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-credit-scoring-764cf876b9-cnxsn" Sep 17 07:47:10.731208 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:47:10.731174 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-credit-scoring-764cf876b9-cnxsn" Sep 17 07:47:10.734294 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:47:10.734267 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-credit-scoring-764cf876b9-cnxsn" Sep 17 07:47:12.724745 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:47:12.724686 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-credit-scoring-764cf876b9-cnxsn" Sep 17 07:47:13.725137 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:47:13.725088 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-764cf876b9-cnxsn" podUID="1c8fa530-cb80-465c-9583-8c69b0bdf80a" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.35:6571: connect: connection refused" Sep 17 07:47:16.724951 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:47:16.724920 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-credit-scoring-764cf876b9-cnxsn" Sep 17 07:47:16.725685 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:47:16.725662 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-credit-scoring-764cf876b9-cnxsn" Sep 17 07:47:18.249074 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:47:18.249041 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-8c464c654-wdvfb"] Sep 17 07:47:18.253468 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:47:18.253446 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-8c464c654-wdvfb" Sep 17 07:47:18.255896 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:47:18.255873 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-registry-tls\"" Sep 17 07:47:18.256014 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:47:18.255997 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-online-tls\"" Sep 17 07:47:18.256266 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:47:18.256250 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-dockercfg-fnvrl\"" Sep 17 07:47:18.256468 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:47:18.256451 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-intra-comm\"" Sep 17 07:47:18.267291 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:47:18.267266 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-8c464c654-wdvfb"] Sep 17 07:47:18.332725 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:47:18.332688 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/4ddca7c1-523d-4e79-a1b0-9fadf0a12530-online-tls\") pod \"feast-driver-ranking-8c464c654-wdvfb\" (UID: \"4ddca7c1-523d-4e79-a1b0-9fadf0a12530\") " pod="test-ns-feast-rest/feast-driver-ranking-8c464c654-wdvfb" Sep 17 07:47:18.332920 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:47:18.332753 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85bgg\" (UniqueName: \"kubernetes.io/projected/4ddca7c1-523d-4e79-a1b0-9fadf0a12530-kube-api-access-85bgg\") pod \"feast-driver-ranking-8c464c654-wdvfb\" (UID: \"4ddca7c1-523d-4e79-a1b0-9fadf0a12530\") " pod="test-ns-feast-rest/feast-driver-ranking-8c464c654-wdvfb" Sep 17 07:47:18.332920 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:47:18.332781 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/4ddca7c1-523d-4e79-a1b0-9fadf0a12530-feast-data\") pod \"feast-driver-ranking-8c464c654-wdvfb\" (UID: \"4ddca7c1-523d-4e79-a1b0-9fadf0a12530\") " pod="test-ns-feast-rest/feast-driver-ranking-8c464c654-wdvfb" Sep 17 07:47:18.332920 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:47:18.332803 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/4ddca7c1-523d-4e79-a1b0-9fadf0a12530-registry-tls\") pod \"feast-driver-ranking-8c464c654-wdvfb\" (UID: \"4ddca7c1-523d-4e79-a1b0-9fadf0a12530\") " pod="test-ns-feast-rest/feast-driver-ranking-8c464c654-wdvfb" Sep 17 07:47:18.434118 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:47:18.434077 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85bgg\" (UniqueName: \"kubernetes.io/projected/4ddca7c1-523d-4e79-a1b0-9fadf0a12530-kube-api-access-85bgg\") pod \"feast-driver-ranking-8c464c654-wdvfb\" (UID: \"4ddca7c1-523d-4e79-a1b0-9fadf0a12530\") " pod="test-ns-feast-rest/feast-driver-ranking-8c464c654-wdvfb" Sep 17 07:47:18.434293 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:47:18.434131 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/4ddca7c1-523d-4e79-a1b0-9fadf0a12530-feast-data\") pod \"feast-driver-ranking-8c464c654-wdvfb\" (UID: \"4ddca7c1-523d-4e79-a1b0-9fadf0a12530\") " pod="test-ns-feast-rest/feast-driver-ranking-8c464c654-wdvfb" Sep 17 07:47:18.434293 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:47:18.434157 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/4ddca7c1-523d-4e79-a1b0-9fadf0a12530-registry-tls\") pod \"feast-driver-ranking-8c464c654-wdvfb\" (UID: \"4ddca7c1-523d-4e79-a1b0-9fadf0a12530\") " pod="test-ns-feast-rest/feast-driver-ranking-8c464c654-wdvfb" Sep 17 07:47:18.434293 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:47:18.434185 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/4ddca7c1-523d-4e79-a1b0-9fadf0a12530-online-tls\") pod \"feast-driver-ranking-8c464c654-wdvfb\" (UID: \"4ddca7c1-523d-4e79-a1b0-9fadf0a12530\") " pod="test-ns-feast-rest/feast-driver-ranking-8c464c654-wdvfb" Sep 17 07:47:18.434472 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:47:18.434320 2578 secret.go:189] Couldn't get secret test-ns-feast-rest/feast-driver-ranking-registry-tls: secret "feast-driver-ranking-registry-tls" not found Sep 17 07:47:18.434472 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:47:18.434404 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ddca7c1-523d-4e79-a1b0-9fadf0a12530-registry-tls podName:4ddca7c1-523d-4e79-a1b0-9fadf0a12530 nodeName:}" failed. No retries permitted until 2026-09-17 07:47:18.934380811 +0000 UTC m=+1182.815415163 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/4ddca7c1-523d-4e79-a1b0-9fadf0a12530-registry-tls") pod "feast-driver-ranking-8c464c654-wdvfb" (UID: "4ddca7c1-523d-4e79-a1b0-9fadf0a12530") : secret "feast-driver-ranking-registry-tls" not found Sep 17 07:47:18.434558 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:47:18.434536 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/4ddca7c1-523d-4e79-a1b0-9fadf0a12530-feast-data\") pod \"feast-driver-ranking-8c464c654-wdvfb\" (UID: \"4ddca7c1-523d-4e79-a1b0-9fadf0a12530\") " pod="test-ns-feast-rest/feast-driver-ranking-8c464c654-wdvfb" Sep 17 07:47:18.436672 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:47:18.436654 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/4ddca7c1-523d-4e79-a1b0-9fadf0a12530-online-tls\") pod \"feast-driver-ranking-8c464c654-wdvfb\" (UID: \"4ddca7c1-523d-4e79-a1b0-9fadf0a12530\") " pod="test-ns-feast-rest/feast-driver-ranking-8c464c654-wdvfb" Sep 17 07:47:18.444252 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:47:18.444227 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-85bgg\" (UniqueName: \"kubernetes.io/projected/4ddca7c1-523d-4e79-a1b0-9fadf0a12530-kube-api-access-85bgg\") pod \"feast-driver-ranking-8c464c654-wdvfb\" (UID: \"4ddca7c1-523d-4e79-a1b0-9fadf0a12530\") " pod="test-ns-feast-rest/feast-driver-ranking-8c464c654-wdvfb" Sep 17 07:47:18.938839 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:47:18.938797 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/4ddca7c1-523d-4e79-a1b0-9fadf0a12530-registry-tls\") pod \"feast-driver-ranking-8c464c654-wdvfb\" (UID: \"4ddca7c1-523d-4e79-a1b0-9fadf0a12530\") " pod="test-ns-feast-rest/feast-driver-ranking-8c464c654-wdvfb" Sep 17 07:47:18.941216 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:47:18.941188 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/4ddca7c1-523d-4e79-a1b0-9fadf0a12530-registry-tls\") pod \"feast-driver-ranking-8c464c654-wdvfb\" (UID: \"4ddca7c1-523d-4e79-a1b0-9fadf0a12530\") " pod="test-ns-feast-rest/feast-driver-ranking-8c464c654-wdvfb" Sep 17 07:47:19.162381 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:47:19.162337 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-8c464c654-wdvfb" Sep 17 07:47:19.440654 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:47:19.440623 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-8c464c654-wdvfb"] Sep 17 07:47:19.574016 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:47:19.573981 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-8c464c654-wdvfb" event={"ID":"4ddca7c1-523d-4e79-a1b0-9fadf0a12530","Type":"ContainerStarted","Data":"2b3ff28606fcfab7ad457fcd02f8995df8eb1643a1bbed64f0d3ed0421bce700"} Sep 17 07:47:19.574187 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:47:19.574030 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-8c464c654-wdvfb" event={"ID":"4ddca7c1-523d-4e79-a1b0-9fadf0a12530","Type":"ContainerStarted","Data":"5d8542cd5c6db9668d7aace38e7bee9d7cecab6572d375a21df4e8eb69acc996"} Sep 17 07:47:24.594133 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:47:24.594096 2578 generic.go:356] "Generic (PLEG): container finished" podID="4ddca7c1-523d-4e79-a1b0-9fadf0a12530" containerID="2b3ff28606fcfab7ad457fcd02f8995df8eb1643a1bbed64f0d3ed0421bce700" exitCode=0 Sep 17 07:47:24.594537 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:47:24.594170 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-8c464c654-wdvfb" event={"ID":"4ddca7c1-523d-4e79-a1b0-9fadf0a12530","Type":"ContainerDied","Data":"2b3ff28606fcfab7ad457fcd02f8995df8eb1643a1bbed64f0d3ed0421bce700"} Sep 17 07:47:25.600353 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:47:25.600319 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-8c464c654-wdvfb" event={"ID":"4ddca7c1-523d-4e79-a1b0-9fadf0a12530","Type":"ContainerStarted","Data":"837fa4fb5690bb4816c4433ed32f59785c7696313c2431e8ef76fd676468f684"} Sep 17 07:47:32.631005 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:47:32.630968 2578 generic.go:356] "Generic (PLEG): container finished" podID="4ddca7c1-523d-4e79-a1b0-9fadf0a12530" containerID="837fa4fb5690bb4816c4433ed32f59785c7696313c2431e8ef76fd676468f684" exitCode=0 Sep 17 07:47:32.631356 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:47:32.631036 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-8c464c654-wdvfb" event={"ID":"4ddca7c1-523d-4e79-a1b0-9fadf0a12530","Type":"ContainerDied","Data":"837fa4fb5690bb4816c4433ed32f59785c7696313c2431e8ef76fd676468f684"} Sep 17 07:47:33.638222 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:47:33.638191 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-8c464c654-wdvfb" event={"ID":"4ddca7c1-523d-4e79-a1b0-9fadf0a12530","Type":"ContainerStarted","Data":"b8448bd67ec96a27a3edc34bdfbe31be566c2dbd5c3793cc0fec86a5ef3d43ac"} Sep 17 07:47:33.638604 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:47:33.638239 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-8c464c654-wdvfb" event={"ID":"4ddca7c1-523d-4e79-a1b0-9fadf0a12530","Type":"ContainerStarted","Data":"7af73c9130e9c890ff7d0f660331730cae21bb5c3342982e0a46b89c3e289423"} Sep 17 07:47:33.666875 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:47:33.666810 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast-rest/feast-driver-ranking-8c464c654-wdvfb" podStartSLOduration=15.666790286 podStartE2EDuration="15.666790286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 07:47:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 07:47:33.66546771 +0000 UTC m=+1197.546502080" watchObservedRunningTime="2026-09-17 07:47:33.666790286 +0000 UTC m=+1197.547824645" Sep 17 07:47:34.163065 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:47:34.163027 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-driver-ranking-8c464c654-wdvfb" Sep 17 07:47:34.163263 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:47:34.163105 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-driver-ranking-8c464c654-wdvfb" Sep 17 07:47:34.165026 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:47:34.164978 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-8c464c654-wdvfb" podUID="4ddca7c1-523d-4e79-a1b0-9fadf0a12530" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.36:6571: connect: connection refused" Sep 17 07:47:34.165026 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:47:34.165018 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-8c464c654-wdvfb" podUID="4ddca7c1-523d-4e79-a1b0-9fadf0a12530" containerName="online" probeResult="failure" output="Get \"https://10.132.0.36:6567/health\": dial tcp 10.132.0.36:6567: connect: connection refused" Sep 17 07:47:37.162960 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:47:37.162916 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-8c464c654-wdvfb" podUID="4ddca7c1-523d-4e79-a1b0-9fadf0a12530" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.36:6571: connect: connection refused" Sep 17 07:47:37.163340 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:47:37.162987 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-8c464c654-wdvfb" podUID="4ddca7c1-523d-4e79-a1b0-9fadf0a12530" containerName="online" probeResult="failure" output="Get \"https://10.132.0.36:6567/health\": dial tcp 10.132.0.36:6567: connect: connection refused" Sep 17 07:47:39.163081 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:47:39.163041 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-driver-ranking-8c464c654-wdvfb" Sep 17 07:47:39.163448 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:47:39.163115 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-driver-ranking-8c464c654-wdvfb" Sep 17 07:47:40.163013 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:47:40.162965 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-8c464c654-wdvfb" podUID="4ddca7c1-523d-4e79-a1b0-9fadf0a12530" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.36:6571: connect: connection refused" Sep 17 07:47:40.169045 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:47:40.169016 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-driver-ranking-8c464c654-wdvfb" Sep 17 07:47:40.172522 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:47:40.172497 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-driver-ranking-8c464c654-wdvfb" Sep 17 07:47:43.163637 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:47:43.163591 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-8c464c654-wdvfb" podUID="4ddca7c1-523d-4e79-a1b0-9fadf0a12530" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.36:6571: connect: connection refused" Sep 17 07:47:46.163722 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:47:46.163686 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-driver-ranking-8c464c654-wdvfb" Sep 17 07:47:46.164430 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:47:46.164411 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-driver-ranking-8c464c654-wdvfb" Sep 17 07:49:08.541982 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:08.541909 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-765754f648-2dc7h"] Sep 17 07:49:08.542440 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:08.542099 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="feast-operator-system/feast-operator-controller-manager-765754f648-2dc7h" podUID="8e776b6d-f611-4e71-b88f-130aa68a8300" containerName="manager" containerID="cri-o://00f366560fe52d3828fe77ff99ceb41902e9e0dfc4af9d261c106fd36593f2f5" gracePeriod=10 Sep 17 07:49:08.796066 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:08.796015 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-765754f648-2dc7h" Sep 17 07:49:08.962087 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:08.962058 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rl89s\" (UniqueName: \"kubernetes.io/projected/8e776b6d-f611-4e71-b88f-130aa68a8300-kube-api-access-rl89s\") pod \"8e776b6d-f611-4e71-b88f-130aa68a8300\" (UID: \"8e776b6d-f611-4e71-b88f-130aa68a8300\") " Sep 17 07:49:08.964315 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:08.964285 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e776b6d-f611-4e71-b88f-130aa68a8300-kube-api-access-rl89s" (OuterVolumeSpecName: "kube-api-access-rl89s") pod "8e776b6d-f611-4e71-b88f-130aa68a8300" (UID: "8e776b6d-f611-4e71-b88f-130aa68a8300"). InnerVolumeSpecName "kube-api-access-rl89s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 07:49:08.974609 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:08.974577 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-764cf876b9-cnxsn"] Sep 17 07:49:08.975046 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:08.974851 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast-rest/feast-credit-scoring-764cf876b9-cnxsn" podUID="1c8fa530-cb80-465c-9583-8c69b0bdf80a" containerName="registry" containerID="cri-o://d453662c58e640ef0bd007a91585ce96e28fba2ec4d7666dd3de52ffe1913a8e" gracePeriod=30 Sep 17 07:49:08.975046 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:08.974929 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast-rest/feast-credit-scoring-764cf876b9-cnxsn" podUID="1c8fa530-cb80-465c-9583-8c69b0bdf80a" containerName="online" containerID="cri-o://b86e8b47dcbb20cc4fff1171578a93a297fbb23a59968c457a6204ab242423d7" gracePeriod=30 Sep 17 07:49:08.999408 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:08.999381 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-8c464c654-wdvfb"] Sep 17 07:49:08.999602 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:08.999582 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast-rest/feast-driver-ranking-8c464c654-wdvfb" podUID="4ddca7c1-523d-4e79-a1b0-9fadf0a12530" containerName="registry" containerID="cri-o://7af73c9130e9c890ff7d0f660331730cae21bb5c3342982e0a46b89c3e289423" gracePeriod=30 Sep 17 07:49:08.999674 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:08.999621 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast-rest/feast-driver-ranking-8c464c654-wdvfb" podUID="4ddca7c1-523d-4e79-a1b0-9fadf0a12530" containerName="online" containerID="cri-o://b8448bd67ec96a27a3edc34bdfbe31be566c2dbd5c3793cc0fec86a5ef3d43ac" gracePeriod=30 Sep 17 07:49:09.060528 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:09.060457 2578 generic.go:356] "Generic (PLEG): container finished" podID="8e776b6d-f611-4e71-b88f-130aa68a8300" containerID="00f366560fe52d3828fe77ff99ceb41902e9e0dfc4af9d261c106fd36593f2f5" exitCode=0 Sep 17 07:49:09.060528 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:09.060510 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-765754f648-2dc7h" Sep 17 07:49:09.060528 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:09.060510 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-765754f648-2dc7h" event={"ID":"8e776b6d-f611-4e71-b88f-130aa68a8300","Type":"ContainerDied","Data":"00f366560fe52d3828fe77ff99ceb41902e9e0dfc4af9d261c106fd36593f2f5"} Sep 17 07:49:09.060760 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:09.060570 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-765754f648-2dc7h" event={"ID":"8e776b6d-f611-4e71-b88f-130aa68a8300","Type":"ContainerDied","Data":"ee1a106b0444df2937227d09d8cfa926089f842d5a842538451bd57b4b9f2227"} Sep 17 07:49:09.060760 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:09.060603 2578 scope.go:117] "RemoveContainer" containerID="00f366560fe52d3828fe77ff99ceb41902e9e0dfc4af9d261c106fd36593f2f5" Sep 17 07:49:09.062728 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:09.062706 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rl89s\" (UniqueName: \"kubernetes.io/projected/8e776b6d-f611-4e71-b88f-130aa68a8300-kube-api-access-rl89s\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Sep 17 07:49:09.069128 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:09.069110 2578 scope.go:117] "RemoveContainer" containerID="00f366560fe52d3828fe77ff99ceb41902e9e0dfc4af9d261c106fd36593f2f5" Sep 17 07:49:09.069432 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:49:09.069406 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00f366560fe52d3828fe77ff99ceb41902e9e0dfc4af9d261c106fd36593f2f5\": container with ID starting with 00f366560fe52d3828fe77ff99ceb41902e9e0dfc4af9d261c106fd36593f2f5 not found: ID does not exist" containerID="00f366560fe52d3828fe77ff99ceb41902e9e0dfc4af9d261c106fd36593f2f5" Sep 17 07:49:09.069525 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:09.069439 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00f366560fe52d3828fe77ff99ceb41902e9e0dfc4af9d261c106fd36593f2f5"} err="failed to get container status \"00f366560fe52d3828fe77ff99ceb41902e9e0dfc4af9d261c106fd36593f2f5\": rpc error: code = NotFound desc = could not find container \"00f366560fe52d3828fe77ff99ceb41902e9e0dfc4af9d261c106fd36593f2f5\": container with ID starting with 00f366560fe52d3828fe77ff99ceb41902e9e0dfc4af9d261c106fd36593f2f5 not found: ID does not exist" Sep 17 07:49:09.084777 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:09.084755 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-765754f648-2dc7h"] Sep 17 07:49:09.093036 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:09.093008 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-765754f648-2dc7h"] Sep 17 07:49:10.067673 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:10.067593 2578 generic.go:356] "Generic (PLEG): container finished" podID="1c8fa530-cb80-465c-9583-8c69b0bdf80a" containerID="b86e8b47dcbb20cc4fff1171578a93a297fbb23a59968c457a6204ab242423d7" exitCode=0 Sep 17 07:49:10.068064 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:10.067667 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-764cf876b9-cnxsn" event={"ID":"1c8fa530-cb80-465c-9583-8c69b0bdf80a","Type":"ContainerDied","Data":"b86e8b47dcbb20cc4fff1171578a93a297fbb23a59968c457a6204ab242423d7"} Sep 17 07:49:10.069788 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:10.069768 2578 generic.go:356] "Generic (PLEG): container finished" podID="4ddca7c1-523d-4e79-a1b0-9fadf0a12530" containerID="b8448bd67ec96a27a3edc34bdfbe31be566c2dbd5c3793cc0fec86a5ef3d43ac" exitCode=0 Sep 17 07:49:10.069931 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:10.069835 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-8c464c654-wdvfb" event={"ID":"4ddca7c1-523d-4e79-a1b0-9fadf0a12530","Type":"ContainerDied","Data":"b8448bd67ec96a27a3edc34bdfbe31be566c2dbd5c3793cc0fec86a5ef3d43ac"} Sep 17 07:49:10.170103 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:10.170069 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-driver-ranking-8c464c654-wdvfb" podUID="4ddca7c1-523d-4e79-a1b0-9fadf0a12530" containerName="online" probeResult="failure" output="Get \"https://10.132.0.36:6567/health\": dial tcp 10.132.0.36:6567: connect: connection refused" Sep 17 07:49:10.629090 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:10.629057 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e776b6d-f611-4e71-b88f-130aa68a8300" path="/var/lib/kubelet/pods/8e776b6d-f611-4e71-b88f-130aa68a8300/volumes" Sep 17 07:49:10.731794 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:10.731754 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-credit-scoring-764cf876b9-cnxsn" podUID="1c8fa530-cb80-465c-9583-8c69b0bdf80a" containerName="online" probeResult="failure" output="Get \"https://10.132.0.35:6567/health\": dial tcp 10.132.0.35:6567: connect: connection refused" Sep 17 07:49:10.772625 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:10.772593 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-nrbvg"] Sep 17 07:49:10.780446 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:10.780418 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-shdm5"] Sep 17 07:49:10.780582 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:10.780561 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast-rest/redis-69bbd798b6-shdm5" podUID="1434c345-9bfd-4329-ad94-6ce063d1466c" containerName="redis" containerID="cri-o://c9b158a1085d6bc2f677ba5a59f1f6a9df0d9b36ef865a75b13c2949cea25f68" gracePeriod=30 Sep 17 07:49:11.025818 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:11.025790 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-69bbd798b6-shdm5" Sep 17 07:49:11.075124 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:11.075085 2578 generic.go:356] "Generic (PLEG): container finished" podID="1434c345-9bfd-4329-ad94-6ce063d1466c" containerID="c9b158a1085d6bc2f677ba5a59f1f6a9df0d9b36ef865a75b13c2949cea25f68" exitCode=0 Sep 17 07:49:11.075484 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:11.075121 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-69bbd798b6-shdm5" Sep 17 07:49:11.075484 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:11.075163 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-69bbd798b6-shdm5" event={"ID":"1434c345-9bfd-4329-ad94-6ce063d1466c","Type":"ContainerDied","Data":"c9b158a1085d6bc2f677ba5a59f1f6a9df0d9b36ef865a75b13c2949cea25f68"} Sep 17 07:49:11.075484 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:11.075212 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-69bbd798b6-shdm5" event={"ID":"1434c345-9bfd-4329-ad94-6ce063d1466c","Type":"ContainerDied","Data":"084c42e8a4a5fad2b8f92def77f1a5733c2f60a3a5e0b06b6c9a2183a1e44a99"} Sep 17 07:49:11.075484 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:11.075235 2578 scope.go:117] "RemoveContainer" containerID="c9b158a1085d6bc2f677ba5a59f1f6a9df0d9b36ef865a75b13c2949cea25f68" Sep 17 07:49:11.075484 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:11.075311 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast-rest/postgres-864846d4f5-nrbvg" podUID="ce9fe81c-010a-425b-9c82-3d00fd3accd1" containerName="postgres" containerID="cri-o://bc51100efd6e0bb2598d4e3d9a14f3ab1fc36b15c196a76d5017d074cc75f5fd" gracePeriod=30 Sep 17 07:49:11.090731 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:11.090697 2578 scope.go:117] "RemoveContainer" containerID="c9b158a1085d6bc2f677ba5a59f1f6a9df0d9b36ef865a75b13c2949cea25f68" Sep 17 07:49:11.091123 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:49:11.091100 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9b158a1085d6bc2f677ba5a59f1f6a9df0d9b36ef865a75b13c2949cea25f68\": container with ID starting with c9b158a1085d6bc2f677ba5a59f1f6a9df0d9b36ef865a75b13c2949cea25f68 not found: ID does not exist" containerID="c9b158a1085d6bc2f677ba5a59f1f6a9df0d9b36ef865a75b13c2949cea25f68" Sep 17 07:49:11.091230 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:11.091134 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9b158a1085d6bc2f677ba5a59f1f6a9df0d9b36ef865a75b13c2949cea25f68"} err="failed to get container status \"c9b158a1085d6bc2f677ba5a59f1f6a9df0d9b36ef865a75b13c2949cea25f68\": rpc error: code = NotFound desc = could not find container \"c9b158a1085d6bc2f677ba5a59f1f6a9df0d9b36ef865a75b13c2949cea25f68\": container with ID starting with c9b158a1085d6bc2f677ba5a59f1f6a9df0d9b36ef865a75b13c2949cea25f68 not found: ID does not exist" Sep 17 07:49:11.181111 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:11.181041 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7xml7\" (UniqueName: \"kubernetes.io/projected/1434c345-9bfd-4329-ad94-6ce063d1466c-kube-api-access-7xml7\") pod \"1434c345-9bfd-4329-ad94-6ce063d1466c\" (UID: \"1434c345-9bfd-4329-ad94-6ce063d1466c\") " Sep 17 07:49:11.183107 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:11.183082 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1434c345-9bfd-4329-ad94-6ce063d1466c-kube-api-access-7xml7" (OuterVolumeSpecName: "kube-api-access-7xml7") pod "1434c345-9bfd-4329-ad94-6ce063d1466c" (UID: "1434c345-9bfd-4329-ad94-6ce063d1466c"). InnerVolumeSpecName "kube-api-access-7xml7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 07:49:11.282203 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:11.282176 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xml7\" (UniqueName: \"kubernetes.io/projected/1434c345-9bfd-4329-ad94-6ce063d1466c-kube-api-access-7xml7\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Sep 17 07:49:11.303724 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:11.303703 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-864846d4f5-nrbvg" Sep 17 07:49:11.409450 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:11.409425 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-shdm5"] Sep 17 07:49:11.413018 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:11.412996 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-shdm5"] Sep 17 07:49:11.484151 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:11.484095 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-75wbj\" (UniqueName: \"kubernetes.io/projected/ce9fe81c-010a-425b-9c82-3d00fd3accd1-kube-api-access-75wbj\") pod \"ce9fe81c-010a-425b-9c82-3d00fd3accd1\" (UID: \"ce9fe81c-010a-425b-9c82-3d00fd3accd1\") " Sep 17 07:49:11.484225 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:11.484178 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/ce9fe81c-010a-425b-9c82-3d00fd3accd1-postgresdata\") pod \"ce9fe81c-010a-425b-9c82-3d00fd3accd1\" (UID: \"ce9fe81c-010a-425b-9c82-3d00fd3accd1\") " Sep 17 07:49:11.486058 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:11.486036 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce9fe81c-010a-425b-9c82-3d00fd3accd1-kube-api-access-75wbj" (OuterVolumeSpecName: "kube-api-access-75wbj") pod "ce9fe81c-010a-425b-9c82-3d00fd3accd1" (UID: "ce9fe81c-010a-425b-9c82-3d00fd3accd1"). InnerVolumeSpecName "kube-api-access-75wbj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 07:49:11.513883 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:11.513837 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce9fe81c-010a-425b-9c82-3d00fd3accd1-postgresdata" (OuterVolumeSpecName: "postgresdata") pod "ce9fe81c-010a-425b-9c82-3d00fd3accd1" (UID: "ce9fe81c-010a-425b-9c82-3d00fd3accd1"). InnerVolumeSpecName "postgresdata". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 07:49:11.585099 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:11.585077 2578 reconciler_common.go:299] "Volume detached for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/ce9fe81c-010a-425b-9c82-3d00fd3accd1-postgresdata\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Sep 17 07:49:11.585099 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:11.585099 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75wbj\" (UniqueName: \"kubernetes.io/projected/ce9fe81c-010a-425b-9c82-3d00fd3accd1-kube-api-access-75wbj\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Sep 17 07:49:12.081563 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:12.081528 2578 generic.go:356] "Generic (PLEG): container finished" podID="ce9fe81c-010a-425b-9c82-3d00fd3accd1" containerID="bc51100efd6e0bb2598d4e3d9a14f3ab1fc36b15c196a76d5017d074cc75f5fd" exitCode=0 Sep 17 07:49:12.082020 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:12.081575 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-864846d4f5-nrbvg" Sep 17 07:49:12.082020 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:12.081571 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-864846d4f5-nrbvg" event={"ID":"ce9fe81c-010a-425b-9c82-3d00fd3accd1","Type":"ContainerDied","Data":"bc51100efd6e0bb2598d4e3d9a14f3ab1fc36b15c196a76d5017d074cc75f5fd"} Sep 17 07:49:12.082020 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:12.081683 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-864846d4f5-nrbvg" event={"ID":"ce9fe81c-010a-425b-9c82-3d00fd3accd1","Type":"ContainerDied","Data":"8f82251362232f048b49086c64a5cac966c3163c03b122ac7141ebb8b82675f4"} Sep 17 07:49:12.082020 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:12.081702 2578 scope.go:117] "RemoveContainer" containerID="bc51100efd6e0bb2598d4e3d9a14f3ab1fc36b15c196a76d5017d074cc75f5fd" Sep 17 07:49:12.090380 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:12.090363 2578 scope.go:117] "RemoveContainer" containerID="bc51100efd6e0bb2598d4e3d9a14f3ab1fc36b15c196a76d5017d074cc75f5fd" Sep 17 07:49:12.090650 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:49:12.090632 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc51100efd6e0bb2598d4e3d9a14f3ab1fc36b15c196a76d5017d074cc75f5fd\": container with ID starting with bc51100efd6e0bb2598d4e3d9a14f3ab1fc36b15c196a76d5017d074cc75f5fd not found: ID does not exist" containerID="bc51100efd6e0bb2598d4e3d9a14f3ab1fc36b15c196a76d5017d074cc75f5fd" Sep 17 07:49:12.090701 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:12.090658 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc51100efd6e0bb2598d4e3d9a14f3ab1fc36b15c196a76d5017d074cc75f5fd"} err="failed to get container status \"bc51100efd6e0bb2598d4e3d9a14f3ab1fc36b15c196a76d5017d074cc75f5fd\": rpc error: code = NotFound desc = could not find container \"bc51100efd6e0bb2598d4e3d9a14f3ab1fc36b15c196a76d5017d074cc75f5fd\": container with ID starting with bc51100efd6e0bb2598d4e3d9a14f3ab1fc36b15c196a76d5017d074cc75f5fd not found: ID does not exist" Sep 17 07:49:12.103619 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:12.103600 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-nrbvg"] Sep 17 07:49:12.107805 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:12.107785 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-nrbvg"] Sep 17 07:49:12.628656 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:12.628620 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1434c345-9bfd-4329-ad94-6ce063d1466c" path="/var/lib/kubelet/pods/1434c345-9bfd-4329-ad94-6ce063d1466c/volumes" Sep 17 07:49:12.628985 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:12.628971 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce9fe81c-010a-425b-9c82-3d00fd3accd1" path="/var/lib/kubelet/pods/ce9fe81c-010a-425b-9c82-3d00fd3accd1/volumes" Sep 17 07:49:20.169957 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:20.169912 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-driver-ranking-8c464c654-wdvfb" podUID="4ddca7c1-523d-4e79-a1b0-9fadf0a12530" containerName="online" probeResult="failure" output="Get \"https://10.132.0.36:6567/health\": dial tcp 10.132.0.36:6567: connect: connection refused" Sep 17 07:49:20.731966 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:20.731927 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-credit-scoring-764cf876b9-cnxsn" podUID="1c8fa530-cb80-465c-9583-8c69b0bdf80a" containerName="online" probeResult="failure" output="Get \"https://10.132.0.35:6567/health\": dial tcp 10.132.0.35:6567: connect: connection refused" Sep 17 07:49:30.170356 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:30.170305 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-driver-ranking-8c464c654-wdvfb" podUID="4ddca7c1-523d-4e79-a1b0-9fadf0a12530" containerName="online" probeResult="failure" output="Get \"https://10.132.0.36:6567/health\": dial tcp 10.132.0.36:6567: connect: connection refused" Sep 17 07:49:30.170848 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:30.170435 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-driver-ranking-8c464c654-wdvfb" Sep 17 07:49:30.731729 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:30.731676 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-credit-scoring-764cf876b9-cnxsn" podUID="1c8fa530-cb80-465c-9583-8c69b0bdf80a" containerName="online" probeResult="failure" output="Get \"https://10.132.0.35:6567/health\": dial tcp 10.132.0.35:6567: connect: connection refused" Sep 17 07:49:30.731939 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:30.731822 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-credit-scoring-764cf876b9-cnxsn" Sep 17 07:49:39.194318 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:39.194283 2578 generic.go:356] "Generic (PLEG): container finished" podID="1c8fa530-cb80-465c-9583-8c69b0bdf80a" containerID="d453662c58e640ef0bd007a91585ce96e28fba2ec4d7666dd3de52ffe1913a8e" exitCode=137 Sep 17 07:49:39.194711 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:39.194357 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-764cf876b9-cnxsn" event={"ID":"1c8fa530-cb80-465c-9583-8c69b0bdf80a","Type":"ContainerDied","Data":"d453662c58e640ef0bd007a91585ce96e28fba2ec4d7666dd3de52ffe1913a8e"} Sep 17 07:49:39.196732 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:39.196705 2578 generic.go:356] "Generic (PLEG): container finished" podID="4ddca7c1-523d-4e79-a1b0-9fadf0a12530" containerID="7af73c9130e9c890ff7d0f660331730cae21bb5c3342982e0a46b89c3e289423" exitCode=137 Sep 17 07:49:39.196855 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:39.196778 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-8c464c654-wdvfb" event={"ID":"4ddca7c1-523d-4e79-a1b0-9fadf0a12530","Type":"ContainerDied","Data":"7af73c9130e9c890ff7d0f660331730cae21bb5c3342982e0a46b89c3e289423"} Sep 17 07:49:39.633111 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:39.633085 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-764cf876b9-cnxsn" Sep 17 07:49:39.661089 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:39.661059 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-8c464c654-wdvfb" Sep 17 07:49:39.692300 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:39.692268 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/4ddca7c1-523d-4e79-a1b0-9fadf0a12530-feast-data\") pod \"4ddca7c1-523d-4e79-a1b0-9fadf0a12530\" (UID: \"4ddca7c1-523d-4e79-a1b0-9fadf0a12530\") " Sep 17 07:49:39.692485 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:39.692340 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-85bgg\" (UniqueName: \"kubernetes.io/projected/4ddca7c1-523d-4e79-a1b0-9fadf0a12530-kube-api-access-85bgg\") pod \"4ddca7c1-523d-4e79-a1b0-9fadf0a12530\" (UID: \"4ddca7c1-523d-4e79-a1b0-9fadf0a12530\") " Sep 17 07:49:39.692485 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:39.692387 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/1c8fa530-cb80-465c-9583-8c69b0bdf80a-online-tls\") pod \"1c8fa530-cb80-465c-9583-8c69b0bdf80a\" (UID: \"1c8fa530-cb80-465c-9583-8c69b0bdf80a\") " Sep 17 07:49:39.692485 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:39.692420 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/1c8fa530-cb80-465c-9583-8c69b0bdf80a-registry-tls\") pod \"1c8fa530-cb80-465c-9583-8c69b0bdf80a\" (UID: \"1c8fa530-cb80-465c-9583-8c69b0bdf80a\") " Sep 17 07:49:39.692485 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:39.692444 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/4ddca7c1-523d-4e79-a1b0-9fadf0a12530-registry-tls\") pod \"4ddca7c1-523d-4e79-a1b0-9fadf0a12530\" (UID: \"4ddca7c1-523d-4e79-a1b0-9fadf0a12530\") " Sep 17 07:49:39.692712 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:39.692540 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g99xf\" (UniqueName: \"kubernetes.io/projected/1c8fa530-cb80-465c-9583-8c69b0bdf80a-kube-api-access-g99xf\") pod \"1c8fa530-cb80-465c-9583-8c69b0bdf80a\" (UID: \"1c8fa530-cb80-465c-9583-8c69b0bdf80a\") " Sep 17 07:49:39.692712 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:39.692600 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/1c8fa530-cb80-465c-9583-8c69b0bdf80a-feast-data\") pod \"1c8fa530-cb80-465c-9583-8c69b0bdf80a\" (UID: \"1c8fa530-cb80-465c-9583-8c69b0bdf80a\") " Sep 17 07:49:39.692712 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:39.692648 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/4ddca7c1-523d-4e79-a1b0-9fadf0a12530-online-tls\") pod \"4ddca7c1-523d-4e79-a1b0-9fadf0a12530\" (UID: \"4ddca7c1-523d-4e79-a1b0-9fadf0a12530\") " Sep 17 07:49:39.692894 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:39.692812 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ddca7c1-523d-4e79-a1b0-9fadf0a12530-feast-data" (OuterVolumeSpecName: "feast-data") pod "4ddca7c1-523d-4e79-a1b0-9fadf0a12530" (UID: "4ddca7c1-523d-4e79-a1b0-9fadf0a12530"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 07:49:39.693209 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:39.693170 2578 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/4ddca7c1-523d-4e79-a1b0-9fadf0a12530-feast-data\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Sep 17 07:49:39.695157 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:39.695130 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c8fa530-cb80-465c-9583-8c69b0bdf80a-online-tls" (OuterVolumeSpecName: "online-tls") pod "1c8fa530-cb80-465c-9583-8c69b0bdf80a" (UID: "1c8fa530-cb80-465c-9583-8c69b0bdf80a"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 07:49:39.695322 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:39.695279 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ddca7c1-523d-4e79-a1b0-9fadf0a12530-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "4ddca7c1-523d-4e79-a1b0-9fadf0a12530" (UID: "4ddca7c1-523d-4e79-a1b0-9fadf0a12530"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 07:49:39.695402 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:39.695364 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c8fa530-cb80-465c-9583-8c69b0bdf80a-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "1c8fa530-cb80-465c-9583-8c69b0bdf80a" (UID: "1c8fa530-cb80-465c-9583-8c69b0bdf80a"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 07:49:39.695614 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:39.695582 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ddca7c1-523d-4e79-a1b0-9fadf0a12530-online-tls" (OuterVolumeSpecName: "online-tls") pod "4ddca7c1-523d-4e79-a1b0-9fadf0a12530" (UID: "4ddca7c1-523d-4e79-a1b0-9fadf0a12530"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 07:49:39.695706 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:39.695638 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c8fa530-cb80-465c-9583-8c69b0bdf80a-kube-api-access-g99xf" (OuterVolumeSpecName: "kube-api-access-g99xf") pod "1c8fa530-cb80-465c-9583-8c69b0bdf80a" (UID: "1c8fa530-cb80-465c-9583-8c69b0bdf80a"). InnerVolumeSpecName "kube-api-access-g99xf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 07:49:39.695762 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:39.695737 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ddca7c1-523d-4e79-a1b0-9fadf0a12530-kube-api-access-85bgg" (OuterVolumeSpecName: "kube-api-access-85bgg") pod "4ddca7c1-523d-4e79-a1b0-9fadf0a12530" (UID: "4ddca7c1-523d-4e79-a1b0-9fadf0a12530"). InnerVolumeSpecName "kube-api-access-85bgg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 07:49:39.702934 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:39.702875 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c8fa530-cb80-465c-9583-8c69b0bdf80a-feast-data" (OuterVolumeSpecName: "feast-data") pod "1c8fa530-cb80-465c-9583-8c69b0bdf80a" (UID: "1c8fa530-cb80-465c-9583-8c69b0bdf80a"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 07:49:39.794815 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:39.794688 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85bgg\" (UniqueName: \"kubernetes.io/projected/4ddca7c1-523d-4e79-a1b0-9fadf0a12530-kube-api-access-85bgg\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Sep 17 07:49:39.794815 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:39.794731 2578 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/1c8fa530-cb80-465c-9583-8c69b0bdf80a-online-tls\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Sep 17 07:49:39.794815 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:39.794742 2578 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/1c8fa530-cb80-465c-9583-8c69b0bdf80a-registry-tls\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Sep 17 07:49:39.794815 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:39.794751 2578 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/4ddca7c1-523d-4e79-a1b0-9fadf0a12530-registry-tls\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Sep 17 07:49:39.794815 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:39.794760 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g99xf\" (UniqueName: \"kubernetes.io/projected/1c8fa530-cb80-465c-9583-8c69b0bdf80a-kube-api-access-g99xf\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Sep 17 07:49:39.794815 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:39.794769 2578 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/1c8fa530-cb80-465c-9583-8c69b0bdf80a-feast-data\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Sep 17 07:49:39.794815 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:39.794778 2578 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/4ddca7c1-523d-4e79-a1b0-9fadf0a12530-online-tls\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Sep 17 07:49:40.202741 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:40.202707 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-764cf876b9-cnxsn" Sep 17 07:49:40.203254 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:40.202707 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-764cf876b9-cnxsn" event={"ID":"1c8fa530-cb80-465c-9583-8c69b0bdf80a","Type":"ContainerDied","Data":"f954d429634cde989a61aa46094bfbbcb30c83a77b24a58680e7f43c7bfe918e"} Sep 17 07:49:40.203254 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:40.202909 2578 scope.go:117] "RemoveContainer" containerID="b86e8b47dcbb20cc4fff1171578a93a297fbb23a59968c457a6204ab242423d7" Sep 17 07:49:40.205437 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:40.205406 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-8c464c654-wdvfb" event={"ID":"4ddca7c1-523d-4e79-a1b0-9fadf0a12530","Type":"ContainerDied","Data":"5d8542cd5c6db9668d7aace38e7bee9d7cecab6572d375a21df4e8eb69acc996"} Sep 17 07:49:40.205540 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:40.205416 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-8c464c654-wdvfb" Sep 17 07:49:40.213080 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:40.213063 2578 scope.go:117] "RemoveContainer" containerID="d453662c58e640ef0bd007a91585ce96e28fba2ec4d7666dd3de52ffe1913a8e" Sep 17 07:49:40.221731 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:40.221713 2578 scope.go:117] "RemoveContainer" containerID="f83f64e263a8d9c722406e9ab6e30e85324a0ce1d8cd542e609d3603231cdf28" Sep 17 07:49:40.229502 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:40.229485 2578 scope.go:117] "RemoveContainer" containerID="135239f3c42da7440eba6fe469d0a28ae35b382b7409405c3ac51adcf1c7b9f9" Sep 17 07:49:40.236949 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:40.236926 2578 scope.go:117] "RemoveContainer" containerID="b8448bd67ec96a27a3edc34bdfbe31be566c2dbd5c3793cc0fec86a5ef3d43ac" Sep 17 07:49:40.243699 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:40.243671 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-764cf876b9-cnxsn"] Sep 17 07:49:40.245190 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:40.245171 2578 scope.go:117] "RemoveContainer" containerID="7af73c9130e9c890ff7d0f660331730cae21bb5c3342982e0a46b89c3e289423" Sep 17 07:49:40.249635 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:40.249611 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-764cf876b9-cnxsn"] Sep 17 07:49:40.253732 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:40.253714 2578 scope.go:117] "RemoveContainer" containerID="837fa4fb5690bb4816c4433ed32f59785c7696313c2431e8ef76fd676468f684" Sep 17 07:49:40.261264 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:40.261243 2578 scope.go:117] "RemoveContainer" containerID="2b3ff28606fcfab7ad457fcd02f8995df8eb1643a1bbed64f0d3ed0421bce700" Sep 17 07:49:40.266434 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:40.266411 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-8c464c654-wdvfb"] Sep 17 07:49:40.270423 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:40.270398 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-8c464c654-wdvfb"] Sep 17 07:49:40.629002 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:40.628917 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c8fa530-cb80-465c-9583-8c69b0bdf80a" path="/var/lib/kubelet/pods/1c8fa530-cb80-465c-9583-8c69b0bdf80a/volumes" Sep 17 07:49:40.629438 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:40.629424 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ddca7c1-523d-4e79-a1b0-9fadf0a12530" path="/var/lib/kubelet/pods/4ddca7c1-523d-4e79-a1b0-9fadf0a12530/volumes" Sep 17 07:49:48.428761 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:48.428724 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-lb6bc"] Sep 17 07:49:48.429151 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:48.429061 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c8fa530-cb80-465c-9583-8c69b0bdf80a" containerName="feast-init" Sep 17 07:49:48.429151 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:48.429073 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c8fa530-cb80-465c-9583-8c69b0bdf80a" containerName="feast-init" Sep 17 07:49:48.429151 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:48.429084 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ddca7c1-523d-4e79-a1b0-9fadf0a12530" containerName="registry" Sep 17 07:49:48.429151 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:48.429093 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ddca7c1-523d-4e79-a1b0-9fadf0a12530" containerName="registry" Sep 17 07:49:48.429151 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:48.429105 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ddca7c1-523d-4e79-a1b0-9fadf0a12530" containerName="feast-apply" Sep 17 07:49:48.429151 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:48.429110 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ddca7c1-523d-4e79-a1b0-9fadf0a12530" containerName="feast-apply" Sep 17 07:49:48.429151 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:48.429121 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c8fa530-cb80-465c-9583-8c69b0bdf80a" containerName="registry" Sep 17 07:49:48.429151 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:48.429127 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c8fa530-cb80-465c-9583-8c69b0bdf80a" containerName="registry" Sep 17 07:49:48.429151 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:48.429136 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c8fa530-cb80-465c-9583-8c69b0bdf80a" containerName="online" Sep 17 07:49:48.429151 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:48.429141 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c8fa530-cb80-465c-9583-8c69b0bdf80a" containerName="online" Sep 17 07:49:48.429151 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:48.429148 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ddca7c1-523d-4e79-a1b0-9fadf0a12530" containerName="online" Sep 17 07:49:48.429151 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:48.429153 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ddca7c1-523d-4e79-a1b0-9fadf0a12530" containerName="online" Sep 17 07:49:48.429512 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:48.429162 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce9fe81c-010a-425b-9c82-3d00fd3accd1" containerName="postgres" Sep 17 07:49:48.429512 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:48.429170 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce9fe81c-010a-425b-9c82-3d00fd3accd1" containerName="postgres" Sep 17 07:49:48.429512 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:48.429180 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ddca7c1-523d-4e79-a1b0-9fadf0a12530" containerName="feast-init" Sep 17 07:49:48.429512 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:48.429185 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ddca7c1-523d-4e79-a1b0-9fadf0a12530" containerName="feast-init" Sep 17 07:49:48.429512 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:48.429193 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c8fa530-cb80-465c-9583-8c69b0bdf80a" containerName="feast-apply" Sep 17 07:49:48.429512 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:48.429199 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c8fa530-cb80-465c-9583-8c69b0bdf80a" containerName="feast-apply" Sep 17 07:49:48.429512 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:48.429208 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e776b6d-f611-4e71-b88f-130aa68a8300" containerName="manager" Sep 17 07:49:48.429512 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:48.429213 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e776b6d-f611-4e71-b88f-130aa68a8300" containerName="manager" Sep 17 07:49:48.429512 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:48.429220 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1434c345-9bfd-4329-ad94-6ce063d1466c" containerName="redis" Sep 17 07:49:48.429512 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:48.429225 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1434c345-9bfd-4329-ad94-6ce063d1466c" containerName="redis" Sep 17 07:49:48.429512 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:48.429280 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="1434c345-9bfd-4329-ad94-6ce063d1466c" containerName="redis" Sep 17 07:49:48.429512 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:48.429287 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ddca7c1-523d-4e79-a1b0-9fadf0a12530" containerName="registry" Sep 17 07:49:48.429512 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:48.429293 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ddca7c1-523d-4e79-a1b0-9fadf0a12530" containerName="online" Sep 17 07:49:48.429512 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:48.429299 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c8fa530-cb80-465c-9583-8c69b0bdf80a" containerName="online" Sep 17 07:49:48.429512 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:48.429305 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce9fe81c-010a-425b-9c82-3d00fd3accd1" containerName="postgres" Sep 17 07:49:48.429512 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:48.429310 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e776b6d-f611-4e71-b88f-130aa68a8300" containerName="manager" Sep 17 07:49:48.429512 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:48.429318 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c8fa530-cb80-465c-9583-8c69b0bdf80a" containerName="registry" Sep 17 07:49:48.435047 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:48.435024 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-lb6bc" Sep 17 07:49:48.438027 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:48.438001 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-offline-tls\"" Sep 17 07:49:48.438371 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:48.438347 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-bf66d\"" Sep 17 07:49:48.438495 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:48.438377 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-registry-tls\"" Sep 17 07:49:48.438495 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:48.438386 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\"" Sep 17 07:49:48.438495 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:48.438478 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-online-tls\"" Sep 17 07:49:48.438495 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:48.438386 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\"" Sep 17 07:49:48.438733 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:48.438545 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-ui-tls\"" Sep 17 07:49:48.449169 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:48.449144 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-lb6bc"] Sep 17 07:49:48.565276 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:48.565238 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/04a0f3a7-6363-49ea-ace8-44be29f72561-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-lb6bc\" (UID: \"04a0f3a7-6363-49ea-ace8-44be29f72561\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-lb6bc" Sep 17 07:49:48.565276 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:48.565280 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/04a0f3a7-6363-49ea-ace8-44be29f72561-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-lb6bc\" (UID: \"04a0f3a7-6363-49ea-ace8-44be29f72561\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-lb6bc" Sep 17 07:49:48.565503 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:48.565308 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/04a0f3a7-6363-49ea-ace8-44be29f72561-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-lb6bc\" (UID: \"04a0f3a7-6363-49ea-ace8-44be29f72561\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-lb6bc" Sep 17 07:49:48.565503 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:48.565332 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djx7x\" (UniqueName: \"kubernetes.io/projected/04a0f3a7-6363-49ea-ace8-44be29f72561-kube-api-access-djx7x\") pod \"feast-simple-feast-setup-57576b8fcf-lb6bc\" (UID: \"04a0f3a7-6363-49ea-ace8-44be29f72561\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-lb6bc" Sep 17 07:49:48.565503 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:48.565355 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/04a0f3a7-6363-49ea-ace8-44be29f72561-feast-data\") pod \"feast-simple-feast-setup-57576b8fcf-lb6bc\" (UID: \"04a0f3a7-6363-49ea-ace8-44be29f72561\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-lb6bc" Sep 17 07:49:48.565503 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:48.565420 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/04a0f3a7-6363-49ea-ace8-44be29f72561-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-lb6bc\" (UID: \"04a0f3a7-6363-49ea-ace8-44be29f72561\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-lb6bc" Sep 17 07:49:48.666595 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:48.666554 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/04a0f3a7-6363-49ea-ace8-44be29f72561-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-lb6bc\" (UID: \"04a0f3a7-6363-49ea-ace8-44be29f72561\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-lb6bc" Sep 17 07:49:48.666595 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:48.666592 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/04a0f3a7-6363-49ea-ace8-44be29f72561-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-lb6bc\" (UID: \"04a0f3a7-6363-49ea-ace8-44be29f72561\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-lb6bc" Sep 17 07:49:48.666882 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:48.666615 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/04a0f3a7-6363-49ea-ace8-44be29f72561-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-lb6bc\" (UID: \"04a0f3a7-6363-49ea-ace8-44be29f72561\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-lb6bc" Sep 17 07:49:48.666882 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:48.666638 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djx7x\" (UniqueName: \"kubernetes.io/projected/04a0f3a7-6363-49ea-ace8-44be29f72561-kube-api-access-djx7x\") pod \"feast-simple-feast-setup-57576b8fcf-lb6bc\" (UID: \"04a0f3a7-6363-49ea-ace8-44be29f72561\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-lb6bc" Sep 17 07:49:48.666882 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:48.666658 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/04a0f3a7-6363-49ea-ace8-44be29f72561-feast-data\") pod \"feast-simple-feast-setup-57576b8fcf-lb6bc\" (UID: \"04a0f3a7-6363-49ea-ace8-44be29f72561\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-lb6bc" Sep 17 07:49:48.666882 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:48.666684 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/04a0f3a7-6363-49ea-ace8-44be29f72561-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-lb6bc\" (UID: \"04a0f3a7-6363-49ea-ace8-44be29f72561\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-lb6bc" Sep 17 07:49:48.666882 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:49:48.666715 2578 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-offline-tls: secret "feast-simple-feast-setup-offline-tls" not found Sep 17 07:49:48.666882 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:49:48.666807 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04a0f3a7-6363-49ea-ace8-44be29f72561-offline-tls podName:04a0f3a7-6363-49ea-ace8-44be29f72561 nodeName:}" failed. No retries permitted until 2026-09-17 07:49:49.166783758 +0000 UTC m=+1333.047818103 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/04a0f3a7-6363-49ea-ace8-44be29f72561-offline-tls") pod "feast-simple-feast-setup-57576b8fcf-lb6bc" (UID: "04a0f3a7-6363-49ea-ace8-44be29f72561") : secret "feast-simple-feast-setup-offline-tls" not found Sep 17 07:49:48.667205 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:48.667055 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/04a0f3a7-6363-49ea-ace8-44be29f72561-feast-data\") pod \"feast-simple-feast-setup-57576b8fcf-lb6bc\" (UID: \"04a0f3a7-6363-49ea-ace8-44be29f72561\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-lb6bc" Sep 17 07:49:48.669271 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:48.669247 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/04a0f3a7-6363-49ea-ace8-44be29f72561-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-lb6bc\" (UID: \"04a0f3a7-6363-49ea-ace8-44be29f72561\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-lb6bc" Sep 17 07:49:48.669421 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:48.669400 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/04a0f3a7-6363-49ea-ace8-44be29f72561-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-lb6bc\" (UID: \"04a0f3a7-6363-49ea-ace8-44be29f72561\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-lb6bc" Sep 17 07:49:48.669462 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:48.669400 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/04a0f3a7-6363-49ea-ace8-44be29f72561-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-lb6bc\" (UID: \"04a0f3a7-6363-49ea-ace8-44be29f72561\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-lb6bc" Sep 17 07:49:48.681379 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:48.681319 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-djx7x\" (UniqueName: \"kubernetes.io/projected/04a0f3a7-6363-49ea-ace8-44be29f72561-kube-api-access-djx7x\") pod \"feast-simple-feast-setup-57576b8fcf-lb6bc\" (UID: \"04a0f3a7-6363-49ea-ace8-44be29f72561\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-lb6bc" Sep 17 07:49:49.177198 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:49.171877 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/04a0f3a7-6363-49ea-ace8-44be29f72561-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-lb6bc\" (UID: \"04a0f3a7-6363-49ea-ace8-44be29f72561\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-lb6bc" Sep 17 07:49:49.177198 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:49.175321 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/04a0f3a7-6363-49ea-ace8-44be29f72561-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-lb6bc\" (UID: \"04a0f3a7-6363-49ea-ace8-44be29f72561\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-lb6bc" Sep 17 07:49:49.356937 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:49.356893 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-lb6bc" Sep 17 07:49:49.646513 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:49.646458 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-lb6bc"] Sep 17 07:49:49.650413 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:49:49.650371 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod04a0f3a7_6363_49ea_ace8_44be29f72561.slice/crio-93e4dc353898146dd46f72232cf715ca458040fc2fe9eebd891340fb1097c2d8 WatchSource:0}: Error finding container 93e4dc353898146dd46f72232cf715ca458040fc2fe9eebd891340fb1097c2d8: Status 404 returned error can't find the container with id 93e4dc353898146dd46f72232cf715ca458040fc2fe9eebd891340fb1097c2d8 Sep 17 07:49:49.652616 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:49.652581 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 17 07:49:50.251195 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:49:50.251120 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-lb6bc" event={"ID":"04a0f3a7-6363-49ea-ace8-44be29f72561","Type":"ContainerStarted","Data":"93e4dc353898146dd46f72232cf715ca458040fc2fe9eebd891340fb1097c2d8"} Sep 17 07:50:03.320118 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:50:03.320078 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-lb6bc" event={"ID":"04a0f3a7-6363-49ea-ace8-44be29f72561","Type":"ContainerStarted","Data":"baea809e8c1f8b98b3f59031e5f9cd06c1763433cc46735abd3ff00b1b695e01"} Sep 17 07:50:08.346531 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:50:08.346443 2578 generic.go:356] "Generic (PLEG): container finished" podID="04a0f3a7-6363-49ea-ace8-44be29f72561" containerID="baea809e8c1f8b98b3f59031e5f9cd06c1763433cc46735abd3ff00b1b695e01" exitCode=0 Sep 17 07:50:08.346531 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:50:08.346515 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-lb6bc" event={"ID":"04a0f3a7-6363-49ea-ace8-44be29f72561","Type":"ContainerDied","Data":"baea809e8c1f8b98b3f59031e5f9cd06c1763433cc46735abd3ff00b1b695e01"} Sep 17 07:50:09.353697 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:50:09.353662 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-lb6bc" event={"ID":"04a0f3a7-6363-49ea-ace8-44be29f72561","Type":"ContainerStarted","Data":"fbd9d1ea53875bf48084501fc2730ee2c555e8b52e79267fd186b21cf8d2aefa"} Sep 17 07:50:15.383000 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:50:15.382963 2578 generic.go:356] "Generic (PLEG): container finished" podID="04a0f3a7-6363-49ea-ace8-44be29f72561" containerID="fbd9d1ea53875bf48084501fc2730ee2c555e8b52e79267fd186b21cf8d2aefa" exitCode=0 Sep 17 07:50:15.383394 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:50:15.383035 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-lb6bc" event={"ID":"04a0f3a7-6363-49ea-ace8-44be29f72561","Type":"ContainerDied","Data":"fbd9d1ea53875bf48084501fc2730ee2c555e8b52e79267fd186b21cf8d2aefa"} Sep 17 07:50:16.394952 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:50:16.394900 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-lb6bc" event={"ID":"04a0f3a7-6363-49ea-ace8-44be29f72561","Type":"ContainerStarted","Data":"982f9127e843c84ccade1a238a833f9f166e08b5180120dd49a158f939b9e51f"} Sep 17 07:50:16.395413 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:50:16.394981 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-lb6bc" event={"ID":"04a0f3a7-6363-49ea-ace8-44be29f72561","Type":"ContainerStarted","Data":"9aec3790a4429f04fb84191247a711d96bc7b55bda9a2824e2ae586d24d7cad7"} Sep 17 07:50:16.395413 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:50:16.395003 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-lb6bc" event={"ID":"04a0f3a7-6363-49ea-ace8-44be29f72561","Type":"ContainerStarted","Data":"207d92c01e578b4f0aed4a4d0d070526c4ca81538dbe151ceb3467f9f370c705"} Sep 17 07:50:16.395413 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:50:16.395020 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-lb6bc" event={"ID":"04a0f3a7-6363-49ea-ace8-44be29f72561","Type":"ContainerStarted","Data":"f068b7c26ff449fadbc6d71258aaaaf87671aff2cd22ce316c2334fb6f593ee6"} Sep 17 07:50:16.431046 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:50:16.430981 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-lb6bc" podStartSLOduration=15.28972071 podStartE2EDuration="28.430958068s" totalImagesPullingTime="13.141237358s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 07:49:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 07:50:16.428541728 +0000 UTC m=+1360.309576090" watchObservedRunningTime="2026-09-17 07:50:16.430958068 +0000 UTC m=+1360.311992427" Sep 17 07:50:19.357913 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:50:19.357835 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-lb6bc" Sep 17 07:50:19.358471 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:50:19.358438 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-lb6bc" Sep 17 07:50:19.358471 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:50:19.358473 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-lb6bc" Sep 17 07:50:19.358684 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:50:19.358494 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-lb6bc" Sep 17 07:50:19.358684 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:50:19.358512 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-lb6bc" Sep 17 07:50:19.358684 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:50:19.358529 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-lb6bc" Sep 17 07:50:19.358684 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:50:19.358548 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-lb6bc" Sep 17 07:50:19.358684 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:50:19.358564 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-lb6bc" Sep 17 07:50:19.360395 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:50:19.360356 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-lb6bc" podUID="04a0f3a7-6363-49ea-ace8-44be29f72561" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.37:6571: connect: connection refused" Sep 17 07:50:19.360395 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:50:19.360374 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-lb6bc" podUID="04a0f3a7-6363-49ea-ace8-44be29f72561" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.37:8816: connect: connection refused" Sep 17 07:50:19.360578 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:50:19.360400 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-lb6bc" podUID="04a0f3a7-6363-49ea-ace8-44be29f72561" containerName="online" probeResult="failure" output="Get \"https://10.132.0.37:6567/health\": dial tcp 10.132.0.37:6567: connect: connection refused" Sep 17 07:50:19.360578 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:50:19.360470 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-lb6bc" podUID="04a0f3a7-6363-49ea-ace8-44be29f72561" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.37:8443: connect: connection refused" Sep 17 07:50:22.357623 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:50:22.357573 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-lb6bc" podUID="04a0f3a7-6363-49ea-ace8-44be29f72561" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.37:8816: connect: connection refused" Sep 17 07:50:22.358234 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:50:22.357619 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-lb6bc" podUID="04a0f3a7-6363-49ea-ace8-44be29f72561" containerName="online" probeResult="failure" output="Get \"https://10.132.0.37:6567/health\": dial tcp 10.132.0.37:6567: connect: connection refused" Sep 17 07:50:22.358234 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:50:22.357593 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-lb6bc" podUID="04a0f3a7-6363-49ea-ace8-44be29f72561" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.37:8443: connect: connection refused" Sep 17 07:50:22.358234 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:50:22.357940 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-lb6bc" podUID="04a0f3a7-6363-49ea-ace8-44be29f72561" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.37:6571: connect: connection refused" Sep 17 07:50:25.358708 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:50:25.358677 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-lb6bc" Sep 17 07:50:25.359176 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:50:25.358801 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-lb6bc" Sep 17 07:50:25.359176 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:50:25.359067 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-lb6bc" Sep 17 07:50:25.359851 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:50:25.359830 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-lb6bc" Sep 17 07:50:25.359969 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:50:25.359919 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-lb6bc" Sep 17 07:50:25.359969 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:50:25.359967 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-lb6bc" Sep 17 07:50:25.363732 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:50:25.363716 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-lb6bc" Sep 17 07:50:25.445695 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:50:25.445667 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-lb6bc" Sep 17 07:51:27.472662 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:51:27.472625 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-lb6bc"] Sep 17 07:51:27.473196 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:51:27.472925 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-lb6bc" podUID="04a0f3a7-6363-49ea-ace8-44be29f72561" containerName="registry" containerID="cri-o://f068b7c26ff449fadbc6d71258aaaaf87671aff2cd22ce316c2334fb6f593ee6" gracePeriod=30 Sep 17 07:51:27.473196 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:51:27.472948 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-lb6bc" podUID="04a0f3a7-6363-49ea-ace8-44be29f72561" containerName="ui" containerID="cri-o://982f9127e843c84ccade1a238a833f9f166e08b5180120dd49a158f939b9e51f" gracePeriod=30 Sep 17 07:51:27.473196 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:51:27.472948 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-lb6bc" podUID="04a0f3a7-6363-49ea-ace8-44be29f72561" containerName="offline" containerID="cri-o://9aec3790a4429f04fb84191247a711d96bc7b55bda9a2824e2ae586d24d7cad7" gracePeriod=30 Sep 17 07:51:27.473196 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:51:27.472948 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-lb6bc" podUID="04a0f3a7-6363-49ea-ace8-44be29f72561" containerName="online" containerID="cri-o://207d92c01e578b4f0aed4a4d0d070526c4ca81538dbe151ceb3467f9f370c705" gracePeriod=30 Sep 17 07:51:28.018290 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:51:28.018251 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-2589m"] Sep 17 07:51:28.021958 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:51:28.021938 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-2589m" Sep 17 07:51:28.025920 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:51:28.025892 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-d922j\"" Sep 17 07:51:28.056797 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:51:28.056762 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-2589m"] Sep 17 07:51:28.095105 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:51:28.095064 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/f2592b75-852d-4865-8157-9a3a9283a0e0-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-2589m\" (UID: \"f2592b75-852d-4865-8157-9a3a9283a0e0\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-2589m" Sep 17 07:51:28.095289 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:51:28.095175 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/f2592b75-852d-4865-8157-9a3a9283a0e0-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-2589m\" (UID: \"f2592b75-852d-4865-8157-9a3a9283a0e0\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-2589m" Sep 17 07:51:28.095289 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:51:28.095205 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdf92\" (UniqueName: \"kubernetes.io/projected/f2592b75-852d-4865-8157-9a3a9283a0e0-kube-api-access-cdf92\") pod \"feast-simple-feast-setup-57576b8fcf-2589m\" (UID: \"f2592b75-852d-4865-8157-9a3a9283a0e0\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-2589m" Sep 17 07:51:28.095289 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:51:28.095230 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/f2592b75-852d-4865-8157-9a3a9283a0e0-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-2589m\" (UID: \"f2592b75-852d-4865-8157-9a3a9283a0e0\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-2589m" Sep 17 07:51:28.095289 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:51:28.095251 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/f2592b75-852d-4865-8157-9a3a9283a0e0-feast-data\") pod \"feast-simple-feast-setup-57576b8fcf-2589m\" (UID: \"f2592b75-852d-4865-8157-9a3a9283a0e0\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-2589m" Sep 17 07:51:28.095486 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:51:28.095336 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/f2592b75-852d-4865-8157-9a3a9283a0e0-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-2589m\" (UID: \"f2592b75-852d-4865-8157-9a3a9283a0e0\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-2589m" Sep 17 07:51:28.196254 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:51:28.196226 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/f2592b75-852d-4865-8157-9a3a9283a0e0-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-2589m\" (UID: \"f2592b75-852d-4865-8157-9a3a9283a0e0\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-2589m" Sep 17 07:51:28.196382 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:51:28.196262 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdf92\" (UniqueName: \"kubernetes.io/projected/f2592b75-852d-4865-8157-9a3a9283a0e0-kube-api-access-cdf92\") pod \"feast-simple-feast-setup-57576b8fcf-2589m\" (UID: \"f2592b75-852d-4865-8157-9a3a9283a0e0\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-2589m" Sep 17 07:51:28.196382 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:51:28.196280 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/f2592b75-852d-4865-8157-9a3a9283a0e0-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-2589m\" (UID: \"f2592b75-852d-4865-8157-9a3a9283a0e0\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-2589m" Sep 17 07:51:28.196500 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:51:28.196421 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/f2592b75-852d-4865-8157-9a3a9283a0e0-feast-data\") pod \"feast-simple-feast-setup-57576b8fcf-2589m\" (UID: \"f2592b75-852d-4865-8157-9a3a9283a0e0\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-2589m" Sep 17 07:51:28.196500 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:51:28.196474 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/f2592b75-852d-4865-8157-9a3a9283a0e0-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-2589m\" (UID: \"f2592b75-852d-4865-8157-9a3a9283a0e0\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-2589m" Sep 17 07:51:28.196602 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:51:28.196505 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/f2592b75-852d-4865-8157-9a3a9283a0e0-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-2589m\" (UID: \"f2592b75-852d-4865-8157-9a3a9283a0e0\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-2589m" Sep 17 07:51:28.196602 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:51:28.196531 2578 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-registry-tls: secret "feast-simple-feast-setup-registry-tls" not found Sep 17 07:51:28.196602 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:51:28.196600 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2592b75-852d-4865-8157-9a3a9283a0e0-registry-tls podName:f2592b75-852d-4865-8157-9a3a9283a0e0 nodeName:}" failed. No retries permitted until 2026-09-17 07:51:28.696578027 +0000 UTC m=+1432.577612385 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/f2592b75-852d-4865-8157-9a3a9283a0e0-registry-tls") pod "feast-simple-feast-setup-57576b8fcf-2589m" (UID: "f2592b75-852d-4865-8157-9a3a9283a0e0") : secret "feast-simple-feast-setup-registry-tls" not found Sep 17 07:51:28.196755 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:51:28.196643 2578 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-offline-tls: secret "feast-simple-feast-setup-offline-tls" not found Sep 17 07:51:28.196755 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:51:28.196695 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2592b75-852d-4865-8157-9a3a9283a0e0-offline-tls podName:f2592b75-852d-4865-8157-9a3a9283a0e0 nodeName:}" failed. No retries permitted until 2026-09-17 07:51:28.696678209 +0000 UTC m=+1432.577712570 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/f2592b75-852d-4865-8157-9a3a9283a0e0-offline-tls") pod "feast-simple-feast-setup-57576b8fcf-2589m" (UID: "f2592b75-852d-4865-8157-9a3a9283a0e0") : secret "feast-simple-feast-setup-offline-tls" not found Sep 17 07:51:28.196970 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:51:28.196944 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/f2592b75-852d-4865-8157-9a3a9283a0e0-feast-data\") pod \"feast-simple-feast-setup-57576b8fcf-2589m\" (UID: \"f2592b75-852d-4865-8157-9a3a9283a0e0\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-2589m" Sep 17 07:51:28.198921 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:51:28.198893 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/f2592b75-852d-4865-8157-9a3a9283a0e0-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-2589m\" (UID: \"f2592b75-852d-4865-8157-9a3a9283a0e0\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-2589m" Sep 17 07:51:28.199029 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:51:28.198956 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/f2592b75-852d-4865-8157-9a3a9283a0e0-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-2589m\" (UID: \"f2592b75-852d-4865-8157-9a3a9283a0e0\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-2589m" Sep 17 07:51:28.208018 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:51:28.207997 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdf92\" (UniqueName: \"kubernetes.io/projected/f2592b75-852d-4865-8157-9a3a9283a0e0-kube-api-access-cdf92\") pod \"feast-simple-feast-setup-57576b8fcf-2589m\" (UID: \"f2592b75-852d-4865-8157-9a3a9283a0e0\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-2589m" Sep 17 07:51:28.686491 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:51:28.686460 2578 generic.go:356] "Generic (PLEG): container finished" podID="04a0f3a7-6363-49ea-ace8-44be29f72561" containerID="982f9127e843c84ccade1a238a833f9f166e08b5180120dd49a158f939b9e51f" exitCode=0 Sep 17 07:51:28.686491 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:51:28.686484 2578 generic.go:356] "Generic (PLEG): container finished" podID="04a0f3a7-6363-49ea-ace8-44be29f72561" containerID="207d92c01e578b4f0aed4a4d0d070526c4ca81538dbe151ceb3467f9f370c705" exitCode=0 Sep 17 07:51:28.686882 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:51:28.686518 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-lb6bc" event={"ID":"04a0f3a7-6363-49ea-ace8-44be29f72561","Type":"ContainerDied","Data":"982f9127e843c84ccade1a238a833f9f166e08b5180120dd49a158f939b9e51f"} Sep 17 07:51:28.686882 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:51:28.686548 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-lb6bc" event={"ID":"04a0f3a7-6363-49ea-ace8-44be29f72561","Type":"ContainerDied","Data":"207d92c01e578b4f0aed4a4d0d070526c4ca81538dbe151ceb3467f9f370c705"} Sep 17 07:51:28.701819 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:51:28.701793 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/f2592b75-852d-4865-8157-9a3a9283a0e0-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-2589m\" (UID: \"f2592b75-852d-4865-8157-9a3a9283a0e0\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-2589m" Sep 17 07:51:28.702065 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:51:28.702044 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/f2592b75-852d-4865-8157-9a3a9283a0e0-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-2589m\" (UID: \"f2592b75-852d-4865-8157-9a3a9283a0e0\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-2589m" Sep 17 07:51:28.704161 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:51:28.704139 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/f2592b75-852d-4865-8157-9a3a9283a0e0-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-2589m\" (UID: \"f2592b75-852d-4865-8157-9a3a9283a0e0\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-2589m" Sep 17 07:51:28.704278 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:51:28.704261 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/f2592b75-852d-4865-8157-9a3a9283a0e0-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-2589m\" (UID: \"f2592b75-852d-4865-8157-9a3a9283a0e0\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-2589m" Sep 17 07:51:28.933977 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:51:28.933935 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-2589m" Sep 17 07:51:29.209120 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:51:29.209092 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-2589m"] Sep 17 07:51:29.211495 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:51:29.211461 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf2592b75_852d_4865_8157_9a3a9283a0e0.slice/crio-4897ad9fb713734ac47e1cedc753038880b45a363365a5b248fcb7984b2e5421 WatchSource:0}: Error finding container 4897ad9fb713734ac47e1cedc753038880b45a363365a5b248fcb7984b2e5421: Status 404 returned error can't find the container with id 4897ad9fb713734ac47e1cedc753038880b45a363365a5b248fcb7984b2e5421 Sep 17 07:51:29.692122 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:51:29.692090 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-2589m" event={"ID":"f2592b75-852d-4865-8157-9a3a9283a0e0","Type":"ContainerStarted","Data":"504309b255fbaffd3ecad8ed5ac7adff267ac747855536d9cc5bfaa6d320a7f4"} Sep 17 07:51:29.692550 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:51:29.692139 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-2589m" event={"ID":"f2592b75-852d-4865-8157-9a3a9283a0e0","Type":"ContainerStarted","Data":"4897ad9fb713734ac47e1cedc753038880b45a363365a5b248fcb7984b2e5421"} Sep 17 07:51:34.713365 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:51:34.713331 2578 generic.go:356] "Generic (PLEG): container finished" podID="f2592b75-852d-4865-8157-9a3a9283a0e0" containerID="504309b255fbaffd3ecad8ed5ac7adff267ac747855536d9cc5bfaa6d320a7f4" exitCode=0 Sep 17 07:51:34.713825 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:51:34.713407 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-2589m" event={"ID":"f2592b75-852d-4865-8157-9a3a9283a0e0","Type":"ContainerDied","Data":"504309b255fbaffd3ecad8ed5ac7adff267ac747855536d9cc5bfaa6d320a7f4"} Sep 17 07:51:35.360082 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:51:35.360042 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-lb6bc" podUID="04a0f3a7-6363-49ea-ace8-44be29f72561" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.37:8443: connect: connection refused" Sep 17 07:51:35.443435 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:51:35.443389 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-lb6bc" podUID="04a0f3a7-6363-49ea-ace8-44be29f72561" containerName="online" probeResult="failure" output="Get \"https://10.132.0.37:6567/health\": dial tcp 10.132.0.37:6567: connect: connection refused" Sep 17 07:51:35.720092 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:51:35.720058 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-2589m" event={"ID":"f2592b75-852d-4865-8157-9a3a9283a0e0","Type":"ContainerStarted","Data":"999bdf5c067843fe57992a840f69a9e85357ab1b1e39a062c74a8bcc03cb7910"} Sep 17 07:51:40.747422 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:51:40.747337 2578 generic.go:356] "Generic (PLEG): container finished" podID="f2592b75-852d-4865-8157-9a3a9283a0e0" containerID="999bdf5c067843fe57992a840f69a9e85357ab1b1e39a062c74a8bcc03cb7910" exitCode=0 Sep 17 07:51:40.747422 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:51:40.747389 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-2589m" event={"ID":"f2592b75-852d-4865-8157-9a3a9283a0e0","Type":"ContainerDied","Data":"999bdf5c067843fe57992a840f69a9e85357ab1b1e39a062c74a8bcc03cb7910"} Sep 17 07:51:41.758647 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:51:41.758604 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-2589m" event={"ID":"f2592b75-852d-4865-8157-9a3a9283a0e0","Type":"ContainerStarted","Data":"d5d92cb2cec7a3810e0af5fa912145bd2b91a63f5bc2b09210ae6e608ef9654d"} Sep 17 07:51:41.759162 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:51:41.758673 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-2589m" event={"ID":"f2592b75-852d-4865-8157-9a3a9283a0e0","Type":"ContainerStarted","Data":"3e68739d56bfe8cb8df5599a36ced93c16a36a1b9b4c4841de68f2fbf9f178dc"} Sep 17 07:51:41.759162 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:51:41.758695 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-2589m" event={"ID":"f2592b75-852d-4865-8157-9a3a9283a0e0","Type":"ContainerStarted","Data":"a9d496273f6547d3b779f5223ce327fd8d793409386d6681800052c8af5b6e3a"} Sep 17 07:51:41.759162 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:51:41.758714 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-2589m" event={"ID":"f2592b75-852d-4865-8157-9a3a9283a0e0","Type":"ContainerStarted","Data":"5217e235d05ebbe5b084942ae5f28daaeb95dacace973f097d2be858db1f549d"} Sep 17 07:51:41.789656 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:51:41.789590 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-2589m" podStartSLOduration=14.789568742 podStartE2EDuration="14.789568742s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 07:51:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 07:51:41.786455354 +0000 UTC m=+1445.667489738" watchObservedRunningTime="2026-09-17 07:51:41.789568742 +0000 UTC m=+1445.670603105" Sep 17 07:51:43.934407 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:51:43.934364 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-2589m" Sep 17 07:51:43.934989 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:51:43.934965 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-2589m" Sep 17 07:51:43.935145 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:51:43.934995 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-2589m" Sep 17 07:51:43.935145 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:51:43.935013 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-2589m" Sep 17 07:51:43.936873 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:51:43.936821 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-2589m" podUID="f2592b75-852d-4865-8157-9a3a9283a0e0" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.38:6571: connect: connection refused" Sep 17 07:51:43.937055 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:51:43.936850 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-2589m" podUID="f2592b75-852d-4865-8157-9a3a9283a0e0" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.38:8443: connect: connection refused" Sep 17 07:51:43.937163 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:51:43.936904 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-2589m" podUID="f2592b75-852d-4865-8157-9a3a9283a0e0" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.38:8816: connect: connection refused" Sep 17 07:51:43.937163 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:51:43.936900 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-2589m" podUID="f2592b75-852d-4865-8157-9a3a9283a0e0" containerName="online" probeResult="failure" output="Get \"https://10.132.0.38:6567/health\": dial tcp 10.132.0.38:6567: connect: connection refused" Sep 17 07:51:45.359747 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:51:45.359679 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-lb6bc" podUID="04a0f3a7-6363-49ea-ace8-44be29f72561" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.37:8443: connect: connection refused" Sep 17 07:51:45.443059 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:51:45.443006 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-lb6bc" podUID="04a0f3a7-6363-49ea-ace8-44be29f72561" containerName="online" probeResult="failure" output="Get \"https://10.132.0.37:6567/health\": dial tcp 10.132.0.37:6567: connect: connection refused" Sep 17 07:51:46.935323 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:51:46.935274 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-2589m" podUID="f2592b75-852d-4865-8157-9a3a9283a0e0" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.38:8816: connect: connection refused" Sep 17 07:51:46.935820 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:51:46.935279 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-2589m" podUID="f2592b75-852d-4865-8157-9a3a9283a0e0" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.38:6571: connect: connection refused" Sep 17 07:51:46.935820 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:51:46.935352 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-2589m" podUID="f2592b75-852d-4865-8157-9a3a9283a0e0" containerName="online" probeResult="failure" output="Get \"https://10.132.0.38:6567/health\": dial tcp 10.132.0.38:6567: connect: connection refused" Sep 17 07:51:46.935820 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:51:46.935292 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-2589m" podUID="f2592b75-852d-4865-8157-9a3a9283a0e0" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.38:8443: connect: connection refused" Sep 17 07:51:48.934107 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:51:48.934068 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-2589m" Sep 17 07:51:48.934603 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:51:48.934131 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-2589m" Sep 17 07:51:48.934603 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:51:48.934150 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-2589m" Sep 17 07:51:48.934603 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:51:48.934165 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-2589m" Sep 17 07:51:49.935176 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:51:49.935146 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-2589m" Sep 17 07:51:49.935608 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:51:49.935263 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-2589m" Sep 17 07:51:49.935608 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:51:49.935455 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-2589m" Sep 17 07:51:49.936034 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:51:49.936013 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-2589m" Sep 17 07:51:49.936213 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:51:49.936196 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-2589m" Sep 17 07:51:49.936279 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:51:49.936268 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-2589m" Sep 17 07:51:49.940150 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:51:49.940133 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-2589m" Sep 17 07:51:50.825689 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:51:50.825657 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-2589m" Sep 17 07:51:55.359571 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:51:55.359529 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-lb6bc" podUID="04a0f3a7-6363-49ea-ace8-44be29f72561" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.37:8443: connect: connection refused" Sep 17 07:51:55.360053 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:51:55.359683 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-lb6bc" Sep 17 07:51:55.442944 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:51:55.442905 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-lb6bc" podUID="04a0f3a7-6363-49ea-ace8-44be29f72561" containerName="online" probeResult="failure" output="Get \"https://10.132.0.37:6567/health\": dial tcp 10.132.0.37:6567: connect: connection refused" Sep 17 07:51:55.443107 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:51:55.443006 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-lb6bc" Sep 17 07:51:57.855093 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:51:57.855012 2578 generic.go:356] "Generic (PLEG): container finished" podID="04a0f3a7-6363-49ea-ace8-44be29f72561" containerID="9aec3790a4429f04fb84191247a711d96bc7b55bda9a2824e2ae586d24d7cad7" exitCode=137 Sep 17 07:51:57.855093 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:51:57.855039 2578 generic.go:356] "Generic (PLEG): container finished" podID="04a0f3a7-6363-49ea-ace8-44be29f72561" containerID="f068b7c26ff449fadbc6d71258aaaaf87671aff2cd22ce316c2334fb6f593ee6" exitCode=137 Sep 17 07:51:57.855093 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:51:57.855085 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-lb6bc" event={"ID":"04a0f3a7-6363-49ea-ace8-44be29f72561","Type":"ContainerDied","Data":"9aec3790a4429f04fb84191247a711d96bc7b55bda9a2824e2ae586d24d7cad7"} Sep 17 07:51:57.855478 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:51:57.855125 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-lb6bc" event={"ID":"04a0f3a7-6363-49ea-ace8-44be29f72561","Type":"ContainerDied","Data":"f068b7c26ff449fadbc6d71258aaaaf87671aff2cd22ce316c2334fb6f593ee6"} Sep 17 07:51:58.125009 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:51:58.124985 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-lb6bc" Sep 17 07:51:58.250525 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:51:58.250494 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/04a0f3a7-6363-49ea-ace8-44be29f72561-feast-data\") pod \"04a0f3a7-6363-49ea-ace8-44be29f72561\" (UID: \"04a0f3a7-6363-49ea-ace8-44be29f72561\") " Sep 17 07:51:58.250705 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:51:58.250536 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/04a0f3a7-6363-49ea-ace8-44be29f72561-ui-tls\") pod \"04a0f3a7-6363-49ea-ace8-44be29f72561\" (UID: \"04a0f3a7-6363-49ea-ace8-44be29f72561\") " Sep 17 07:51:58.250705 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:51:58.250589 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/04a0f3a7-6363-49ea-ace8-44be29f72561-online-tls\") pod \"04a0f3a7-6363-49ea-ace8-44be29f72561\" (UID: \"04a0f3a7-6363-49ea-ace8-44be29f72561\") " Sep 17 07:51:58.250705 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:51:58.250612 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/04a0f3a7-6363-49ea-ace8-44be29f72561-registry-tls\") pod \"04a0f3a7-6363-49ea-ace8-44be29f72561\" (UID: \"04a0f3a7-6363-49ea-ace8-44be29f72561\") " Sep 17 07:51:58.250705 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:51:58.250638 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/04a0f3a7-6363-49ea-ace8-44be29f72561-offline-tls\") pod \"04a0f3a7-6363-49ea-ace8-44be29f72561\" (UID: \"04a0f3a7-6363-49ea-ace8-44be29f72561\") " Sep 17 07:51:58.250705 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:51:58.250659 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-djx7x\" (UniqueName: \"kubernetes.io/projected/04a0f3a7-6363-49ea-ace8-44be29f72561-kube-api-access-djx7x\") pod \"04a0f3a7-6363-49ea-ace8-44be29f72561\" (UID: \"04a0f3a7-6363-49ea-ace8-44be29f72561\") " Sep 17 07:51:58.251133 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:51:58.251106 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04a0f3a7-6363-49ea-ace8-44be29f72561-feast-data" (OuterVolumeSpecName: "feast-data") pod "04a0f3a7-6363-49ea-ace8-44be29f72561" (UID: "04a0f3a7-6363-49ea-ace8-44be29f72561"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 07:51:58.252903 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:51:58.252849 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04a0f3a7-6363-49ea-ace8-44be29f72561-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "04a0f3a7-6363-49ea-ace8-44be29f72561" (UID: "04a0f3a7-6363-49ea-ace8-44be29f72561"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 07:51:58.253258 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:51:58.253227 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04a0f3a7-6363-49ea-ace8-44be29f72561-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "04a0f3a7-6363-49ea-ace8-44be29f72561" (UID: "04a0f3a7-6363-49ea-ace8-44be29f72561"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 07:51:58.253347 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:51:58.253259 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04a0f3a7-6363-49ea-ace8-44be29f72561-online-tls" (OuterVolumeSpecName: "online-tls") pod "04a0f3a7-6363-49ea-ace8-44be29f72561" (UID: "04a0f3a7-6363-49ea-ace8-44be29f72561"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 07:51:58.253347 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:51:58.253235 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04a0f3a7-6363-49ea-ace8-44be29f72561-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "04a0f3a7-6363-49ea-ace8-44be29f72561" (UID: "04a0f3a7-6363-49ea-ace8-44be29f72561"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 07:51:58.253347 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:51:58.253301 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04a0f3a7-6363-49ea-ace8-44be29f72561-kube-api-access-djx7x" (OuterVolumeSpecName: "kube-api-access-djx7x") pod "04a0f3a7-6363-49ea-ace8-44be29f72561" (UID: "04a0f3a7-6363-49ea-ace8-44be29f72561"). InnerVolumeSpecName "kube-api-access-djx7x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 07:51:58.351544 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:51:58.351517 2578 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/04a0f3a7-6363-49ea-ace8-44be29f72561-online-tls\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Sep 17 07:51:58.351544 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:51:58.351541 2578 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/04a0f3a7-6363-49ea-ace8-44be29f72561-registry-tls\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Sep 17 07:51:58.351544 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:51:58.351549 2578 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/04a0f3a7-6363-49ea-ace8-44be29f72561-offline-tls\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Sep 17 07:51:58.351754 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:51:58.351560 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djx7x\" (UniqueName: \"kubernetes.io/projected/04a0f3a7-6363-49ea-ace8-44be29f72561-kube-api-access-djx7x\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Sep 17 07:51:58.351754 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:51:58.351569 2578 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/04a0f3a7-6363-49ea-ace8-44be29f72561-feast-data\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Sep 17 07:51:58.351754 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:51:58.351579 2578 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/04a0f3a7-6363-49ea-ace8-44be29f72561-ui-tls\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Sep 17 07:51:58.862468 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:51:58.862439 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-lb6bc" Sep 17 07:51:58.862838 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:51:58.862438 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-lb6bc" event={"ID":"04a0f3a7-6363-49ea-ace8-44be29f72561","Type":"ContainerDied","Data":"93e4dc353898146dd46f72232cf715ca458040fc2fe9eebd891340fb1097c2d8"} Sep 17 07:51:58.862838 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:51:58.862560 2578 scope.go:117] "RemoveContainer" containerID="982f9127e843c84ccade1a238a833f9f166e08b5180120dd49a158f939b9e51f" Sep 17 07:51:58.870499 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:51:58.870481 2578 scope.go:117] "RemoveContainer" containerID="9aec3790a4429f04fb84191247a711d96bc7b55bda9a2824e2ae586d24d7cad7" Sep 17 07:51:58.877712 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:51:58.877694 2578 scope.go:117] "RemoveContainer" containerID="207d92c01e578b4f0aed4a4d0d070526c4ca81538dbe151ceb3467f9f370c705" Sep 17 07:51:58.885110 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:51:58.885087 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-lb6bc"] Sep 17 07:51:58.885387 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:51:58.885372 2578 scope.go:117] "RemoveContainer" containerID="f068b7c26ff449fadbc6d71258aaaaf87671aff2cd22ce316c2334fb6f593ee6" Sep 17 07:51:58.890393 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:51:58.890368 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-lb6bc"] Sep 17 07:51:58.892933 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:51:58.892908 2578 scope.go:117] "RemoveContainer" containerID="fbd9d1ea53875bf48084501fc2730ee2c555e8b52e79267fd186b21cf8d2aefa" Sep 17 07:51:58.899623 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:51:58.899608 2578 scope.go:117] "RemoveContainer" containerID="baea809e8c1f8b98b3f59031e5f9cd06c1763433cc46735abd3ff00b1b695e01" Sep 17 07:52:00.628617 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:52:00.628581 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="04a0f3a7-6363-49ea-ace8-44be29f72561" path="/var/lib/kubelet/pods/04a0f3a7-6363-49ea-ace8-44be29f72561/volumes" Sep 17 07:52:53.149167 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:52:53.149133 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dgt7q"] Sep 17 07:52:53.149594 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:52:53.149425 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04a0f3a7-6363-49ea-ace8-44be29f72561" containerName="feast-apply" Sep 17 07:52:53.149594 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:52:53.149435 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="04a0f3a7-6363-49ea-ace8-44be29f72561" containerName="feast-apply" Sep 17 07:52:53.149594 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:52:53.149449 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04a0f3a7-6363-49ea-ace8-44be29f72561" containerName="online" Sep 17 07:52:53.149594 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:52:53.149454 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="04a0f3a7-6363-49ea-ace8-44be29f72561" containerName="online" Sep 17 07:52:53.149594 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:52:53.149461 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04a0f3a7-6363-49ea-ace8-44be29f72561" containerName="ui" Sep 17 07:52:53.149594 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:52:53.149467 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="04a0f3a7-6363-49ea-ace8-44be29f72561" containerName="ui" Sep 17 07:52:53.149594 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:52:53.149478 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04a0f3a7-6363-49ea-ace8-44be29f72561" containerName="registry" Sep 17 07:52:53.149594 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:52:53.149483 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="04a0f3a7-6363-49ea-ace8-44be29f72561" containerName="registry" Sep 17 07:52:53.149594 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:52:53.149489 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04a0f3a7-6363-49ea-ace8-44be29f72561" containerName="feast-init" Sep 17 07:52:53.149594 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:52:53.149494 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="04a0f3a7-6363-49ea-ace8-44be29f72561" containerName="feast-init" Sep 17 07:52:53.149594 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:52:53.149500 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04a0f3a7-6363-49ea-ace8-44be29f72561" containerName="offline" Sep 17 07:52:53.149594 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:52:53.149505 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="04a0f3a7-6363-49ea-ace8-44be29f72561" containerName="offline" Sep 17 07:52:53.149594 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:52:53.149553 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="04a0f3a7-6363-49ea-ace8-44be29f72561" containerName="registry" Sep 17 07:52:53.149594 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:52:53.149563 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="04a0f3a7-6363-49ea-ace8-44be29f72561" containerName="ui" Sep 17 07:52:53.149594 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:52:53.149569 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="04a0f3a7-6363-49ea-ace8-44be29f72561" containerName="offline" Sep 17 07:52:53.149594 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:52:53.149575 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="04a0f3a7-6363-49ea-ace8-44be29f72561" containerName="online" Sep 17 07:52:53.153056 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:52:53.153038 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dgt7q" Sep 17 07:52:53.156690 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:52:53.156671 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-client-ca\"" Sep 17 07:52:53.156788 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:52:53.156761 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-ui-tls\"" Sep 17 07:52:53.156788 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:52:53.156767 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-offline-tls\"" Sep 17 07:52:53.156892 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:52:53.156764 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-online-tls\"" Sep 17 07:52:53.157167 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:52:53.157153 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"kube-root-ca.crt\"" Sep 17 07:52:53.157254 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:52:53.157238 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-dockercfg-2bsrt\"" Sep 17 07:52:53.157303 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:52:53.157272 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"openshift-service-ca.crt\"" Sep 17 07:52:53.167402 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:52:53.167375 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dgt7q"] Sep 17 07:52:53.269097 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:52:53.269065 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/51756a99-098d-4f45-8685-3484422723fe-ui-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-dgt7q\" (UID: \"51756a99-098d-4f45-8685-3484422723fe\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dgt7q" Sep 17 07:52:53.269097 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:52:53.269101 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/51756a99-098d-4f45-8685-3484422723fe-online-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-dgt7q\" (UID: \"51756a99-098d-4f45-8685-3484422723fe\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dgt7q" Sep 17 07:52:53.269300 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:52:53.269201 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/51756a99-098d-4f45-8685-3484422723fe-offline-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-dgt7q\" (UID: \"51756a99-098d-4f45-8685-3484422723fe\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dgt7q" Sep 17 07:52:53.269300 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:52:53.269240 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/51756a99-098d-4f45-8685-3484422723fe-feast-data\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-dgt7q\" (UID: \"51756a99-098d-4f45-8685-3484422723fe\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dgt7q" Sep 17 07:52:53.269300 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:52:53.269258 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xwn6\" (UniqueName: \"kubernetes.io/projected/51756a99-098d-4f45-8685-3484422723fe-kube-api-access-6xwn6\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-dgt7q\" (UID: \"51756a99-098d-4f45-8685-3484422723fe\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dgt7q" Sep 17 07:52:53.269300 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:52:53.269279 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/51756a99-098d-4f45-8685-3484422723fe-registry-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-dgt7q\" (UID: \"51756a99-098d-4f45-8685-3484422723fe\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dgt7q" Sep 17 07:52:53.369729 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:52:53.369693 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/51756a99-098d-4f45-8685-3484422723fe-online-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-dgt7q\" (UID: \"51756a99-098d-4f45-8685-3484422723fe\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dgt7q" Sep 17 07:52:53.369914 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:52:53.369738 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/51756a99-098d-4f45-8685-3484422723fe-offline-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-dgt7q\" (UID: \"51756a99-098d-4f45-8685-3484422723fe\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dgt7q" Sep 17 07:52:53.369914 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:52:53.369821 2578 secret.go:189] Couldn't get secret test-ns-remote-registry/feast-simple-feast-remote-setup-offline-tls: secret "feast-simple-feast-remote-setup-offline-tls" not found Sep 17 07:52:53.369914 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:52:53.369830 2578 secret.go:189] Couldn't get secret test-ns-remote-registry/feast-simple-feast-remote-setup-online-tls: secret "feast-simple-feast-remote-setup-online-tls" not found Sep 17 07:52:53.369914 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:52:53.369850 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/51756a99-098d-4f45-8685-3484422723fe-feast-data\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-dgt7q\" (UID: \"51756a99-098d-4f45-8685-3484422723fe\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dgt7q" Sep 17 07:52:53.369914 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:52:53.369914 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51756a99-098d-4f45-8685-3484422723fe-offline-tls podName:51756a99-098d-4f45-8685-3484422723fe nodeName:}" failed. No retries permitted until 2026-09-17 07:52:53.869891408 +0000 UTC m=+1517.750925744 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/51756a99-098d-4f45-8685-3484422723fe-offline-tls") pod "feast-simple-feast-remote-setup-c5d49d8c6-dgt7q" (UID: "51756a99-098d-4f45-8685-3484422723fe") : secret "feast-simple-feast-remote-setup-offline-tls" not found Sep 17 07:52:53.370197 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:52:53.369933 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51756a99-098d-4f45-8685-3484422723fe-online-tls podName:51756a99-098d-4f45-8685-3484422723fe nodeName:}" failed. No retries permitted until 2026-09-17 07:52:53.869925639 +0000 UTC m=+1517.750959976 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/51756a99-098d-4f45-8685-3484422723fe-online-tls") pod "feast-simple-feast-remote-setup-c5d49d8c6-dgt7q" (UID: "51756a99-098d-4f45-8685-3484422723fe") : secret "feast-simple-feast-remote-setup-online-tls" not found Sep 17 07:52:53.370197 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:52:53.369959 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xwn6\" (UniqueName: \"kubernetes.io/projected/51756a99-098d-4f45-8685-3484422723fe-kube-api-access-6xwn6\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-dgt7q\" (UID: \"51756a99-098d-4f45-8685-3484422723fe\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dgt7q" Sep 17 07:52:53.370197 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:52:53.369979 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/51756a99-098d-4f45-8685-3484422723fe-registry-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-dgt7q\" (UID: \"51756a99-098d-4f45-8685-3484422723fe\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dgt7q" Sep 17 07:52:53.370197 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:52:53.370015 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/51756a99-098d-4f45-8685-3484422723fe-ui-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-dgt7q\" (UID: \"51756a99-098d-4f45-8685-3484422723fe\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dgt7q" Sep 17 07:52:53.370197 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:52:53.370096 2578 secret.go:189] Couldn't get secret test-ns-remote-registry/feast-simple-feast-remote-setup-ui-tls: secret "feast-simple-feast-remote-setup-ui-tls" not found Sep 17 07:52:53.370197 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:52:53.370129 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51756a99-098d-4f45-8685-3484422723fe-ui-tls podName:51756a99-098d-4f45-8685-3484422723fe nodeName:}" failed. No retries permitted until 2026-09-17 07:52:53.870118837 +0000 UTC m=+1517.751153173 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "ui-tls" (UniqueName: "kubernetes.io/secret/51756a99-098d-4f45-8685-3484422723fe-ui-tls") pod "feast-simple-feast-remote-setup-c5d49d8c6-dgt7q" (UID: "51756a99-098d-4f45-8685-3484422723fe") : secret "feast-simple-feast-remote-setup-ui-tls" not found Sep 17 07:52:53.370429 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:52:53.370270 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/51756a99-098d-4f45-8685-3484422723fe-feast-data\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-dgt7q\" (UID: \"51756a99-098d-4f45-8685-3484422723fe\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dgt7q" Sep 17 07:52:53.370532 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:52:53.370514 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/51756a99-098d-4f45-8685-3484422723fe-registry-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-dgt7q\" (UID: \"51756a99-098d-4f45-8685-3484422723fe\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dgt7q" Sep 17 07:52:53.381171 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:52:53.381153 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xwn6\" (UniqueName: \"kubernetes.io/projected/51756a99-098d-4f45-8685-3484422723fe-kube-api-access-6xwn6\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-dgt7q\" (UID: \"51756a99-098d-4f45-8685-3484422723fe\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dgt7q" Sep 17 07:52:53.875065 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:52:53.875029 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/51756a99-098d-4f45-8685-3484422723fe-ui-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-dgt7q\" (UID: \"51756a99-098d-4f45-8685-3484422723fe\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dgt7q" Sep 17 07:52:53.875065 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:52:53.875068 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/51756a99-098d-4f45-8685-3484422723fe-online-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-dgt7q\" (UID: \"51756a99-098d-4f45-8685-3484422723fe\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dgt7q" Sep 17 07:52:53.875327 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:52:53.875102 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/51756a99-098d-4f45-8685-3484422723fe-offline-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-dgt7q\" (UID: \"51756a99-098d-4f45-8685-3484422723fe\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dgt7q" Sep 17 07:52:53.877522 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:52:53.877492 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/51756a99-098d-4f45-8685-3484422723fe-ui-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-dgt7q\" (UID: \"51756a99-098d-4f45-8685-3484422723fe\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dgt7q" Sep 17 07:52:53.877658 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:52:53.877546 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/51756a99-098d-4f45-8685-3484422723fe-online-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-dgt7q\" (UID: \"51756a99-098d-4f45-8685-3484422723fe\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dgt7q" Sep 17 07:52:53.877658 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:52:53.877600 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/51756a99-098d-4f45-8685-3484422723fe-offline-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-dgt7q\" (UID: \"51756a99-098d-4f45-8685-3484422723fe\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dgt7q" Sep 17 07:52:54.061947 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:52:54.061916 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dgt7q" Sep 17 07:52:54.339784 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:52:54.339756 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dgt7q"] Sep 17 07:52:54.341576 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:52:54.341555 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod51756a99_098d_4f45_8685_3484422723fe.slice/crio-9aa2bddfc3476227789b00a4a277ed636708457d729a2a5986f1bb5552c4ca94 WatchSource:0}: Error finding container 9aa2bddfc3476227789b00a4a277ed636708457d729a2a5986f1bb5552c4ca94: Status 404 returned error can't find the container with id 9aa2bddfc3476227789b00a4a277ed636708457d729a2a5986f1bb5552c4ca94 Sep 17 07:52:55.082631 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:52:55.082593 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dgt7q" event={"ID":"51756a99-098d-4f45-8685-3484422723fe","Type":"ContainerStarted","Data":"ce383584b55b38e81363dc3c02c19ce05c6281af7b7435caa832cdedf4ad4ee4"} Sep 17 07:52:55.082829 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:52:55.082640 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dgt7q" event={"ID":"51756a99-098d-4f45-8685-3484422723fe","Type":"ContainerStarted","Data":"9aa2bddfc3476227789b00a4a277ed636708457d729a2a5986f1bb5552c4ca94"} Sep 17 07:52:59.099932 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:52:59.099895 2578 generic.go:356] "Generic (PLEG): container finished" podID="51756a99-098d-4f45-8685-3484422723fe" containerID="ce383584b55b38e81363dc3c02c19ce05c6281af7b7435caa832cdedf4ad4ee4" exitCode=0 Sep 17 07:52:59.100258 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:52:59.099972 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dgt7q" event={"ID":"51756a99-098d-4f45-8685-3484422723fe","Type":"ContainerDied","Data":"ce383584b55b38e81363dc3c02c19ce05c6281af7b7435caa832cdedf4ad4ee4"} Sep 17 07:53:00.106784 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:53:00.106750 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dgt7q" event={"ID":"51756a99-098d-4f45-8685-3484422723fe","Type":"ContainerStarted","Data":"b21d7ea7f2f7f9830290fe64c080cd03f65e61297a8cd4cf9c267eae36d14b01"} Sep 17 07:53:26.215828 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:53:26.215789 2578 generic.go:356] "Generic (PLEG): container finished" podID="51756a99-098d-4f45-8685-3484422723fe" containerID="b21d7ea7f2f7f9830290fe64c080cd03f65e61297a8cd4cf9c267eae36d14b01" exitCode=0 Sep 17 07:53:26.216249 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:53:26.215844 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dgt7q" event={"ID":"51756a99-098d-4f45-8685-3484422723fe","Type":"ContainerDied","Data":"b21d7ea7f2f7f9830290fe64c080cd03f65e61297a8cd4cf9c267eae36d14b01"} Sep 17 07:53:27.224118 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:53:27.224078 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dgt7q" event={"ID":"51756a99-098d-4f45-8685-3484422723fe","Type":"ContainerStarted","Data":"fed7abb5db452a6b1d16fb52b2650b2fb51c61a922dce1aed13c2b6871cca4b7"} Sep 17 07:53:27.224589 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:53:27.224138 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dgt7q" event={"ID":"51756a99-098d-4f45-8685-3484422723fe","Type":"ContainerStarted","Data":"a5fab8c2661716eb48e2b3ec947feb50cf736e77c952151429eb9d4f2ac5d3e4"} Sep 17 07:53:27.224589 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:53:27.224156 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dgt7q" event={"ID":"51756a99-098d-4f45-8685-3484422723fe","Type":"ContainerStarted","Data":"d23e9db3dff66498899b74e7efeb47908ff0babf2c5011d67c2e60ce8a7a6e5b"} Sep 17 07:53:27.254659 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:53:27.254599 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dgt7q" podStartSLOduration=34.254582407 podStartE2EDuration="34.254582407s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 07:52:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 07:53:27.251577716 +0000 UTC m=+1551.132612090" watchObservedRunningTime="2026-09-17 07:53:27.254582407 +0000 UTC m=+1551.135616765" Sep 17 07:53:30.062559 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:53:30.062517 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dgt7q" Sep 17 07:53:30.063075 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:53:30.062578 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dgt7q" Sep 17 07:53:30.063075 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:53:30.062598 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dgt7q" Sep 17 07:53:30.064358 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:53:30.064318 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dgt7q" podUID="51756a99-098d-4f45-8685-3484422723fe" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.39:8443: connect: connection refused" Sep 17 07:53:30.064358 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:53:30.064338 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dgt7q" podUID="51756a99-098d-4f45-8685-3484422723fe" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.39:8816: connect: connection refused" Sep 17 07:53:30.064565 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:53:30.064381 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dgt7q" podUID="51756a99-098d-4f45-8685-3484422723fe" containerName="online" probeResult="failure" output="Get \"https://10.132.0.39:6567/health\": dial tcp 10.132.0.39:6567: connect: connection refused" Sep 17 07:53:33.062248 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:53:33.062204 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dgt7q" podUID="51756a99-098d-4f45-8685-3484422723fe" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.39:8816: connect: connection refused" Sep 17 07:53:33.062801 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:53:33.062257 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dgt7q" podUID="51756a99-098d-4f45-8685-3484422723fe" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.39:8443: connect: connection refused" Sep 17 07:53:33.062801 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:53:33.062295 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dgt7q" podUID="51756a99-098d-4f45-8685-3484422723fe" containerName="online" probeResult="failure" output="Get \"https://10.132.0.39:6567/health\": dial tcp 10.132.0.39:6567: connect: connection refused" Sep 17 07:53:34.062853 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:53:34.062813 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dgt7q" Sep 17 07:53:34.063233 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:53:34.062915 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dgt7q" Sep 17 07:53:34.063233 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:53:34.062939 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dgt7q" Sep 17 07:53:36.063636 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:53:36.063604 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dgt7q" Sep 17 07:53:36.064094 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:53:36.063855 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dgt7q" Sep 17 07:53:36.064430 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:53:36.064405 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dgt7q" Sep 17 07:53:36.268570 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:53:36.268540 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dgt7q" Sep 17 07:53:37.067999 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:53:37.067956 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dgt7q" podUID="51756a99-098d-4f45-8685-3484422723fe" containerName="online" probeResult="failure" output="Get \"https://10.132.0.39:6567/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 17 07:53:40.066016 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:53:40.065975 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dgt7q" podUID="51756a99-098d-4f45-8685-3484422723fe" containerName="online" probeResult="failure" output="Get \"https://10.132.0.39:6567/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 17 07:53:43.028602 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:53:43.028569 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dgt7q" Sep 17 07:53:43.986925 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:53:43.986891 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dgt7q" Sep 17 07:54:24.033987 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:54:24.033946 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dgt7q" podUID="51756a99-098d-4f45-8685-3484422723fe" containerName="online" probeResult="failure" output="Get \"https://10.132.0.39:6567/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Sep 17 07:54:44.035033 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:54:44.034955 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dgt7q" podUID="51756a99-098d-4f45-8685-3484422723fe" containerName="online" probeResult="failure" output="Get \"https://10.132.0.39:6567/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 17 07:54:45.583711 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:54:45.583679 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dgt7q"] Sep 17 07:54:45.584208 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:54:45.583925 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dgt7q" podUID="51756a99-098d-4f45-8685-3484422723fe" containerName="online" containerID="cri-o://d23e9db3dff66498899b74e7efeb47908ff0babf2c5011d67c2e60ce8a7a6e5b" gracePeriod=30 Sep 17 07:54:45.584208 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:54:45.583958 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dgt7q" podUID="51756a99-098d-4f45-8685-3484422723fe" containerName="ui" containerID="cri-o://fed7abb5db452a6b1d16fb52b2650b2fb51c61a922dce1aed13c2b6871cca4b7" gracePeriod=30 Sep 17 07:54:45.584208 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:54:45.584009 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dgt7q" podUID="51756a99-098d-4f45-8685-3484422723fe" containerName="offline" containerID="cri-o://a5fab8c2661716eb48e2b3ec947feb50cf736e77c952151429eb9d4f2ac5d3e4" gracePeriod=30 Sep 17 07:54:45.760345 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:54:45.760313 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-2589m"] Sep 17 07:54:45.760761 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:54:45.760558 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-2589m" podUID="f2592b75-852d-4865-8157-9a3a9283a0e0" containerName="registry" containerID="cri-o://5217e235d05ebbe5b084942ae5f28daaeb95dacace973f097d2be858db1f549d" gracePeriod=30 Sep 17 07:54:45.760761 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:54:45.760597 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-2589m" podUID="f2592b75-852d-4865-8157-9a3a9283a0e0" containerName="ui" containerID="cri-o://d5d92cb2cec7a3810e0af5fa912145bd2b91a63f5bc2b09210ae6e608ef9654d" gracePeriod=30 Sep 17 07:54:45.760761 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:54:45.760612 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-2589m" podUID="f2592b75-852d-4865-8157-9a3a9283a0e0" containerName="online" containerID="cri-o://a9d496273f6547d3b779f5223ce327fd8d793409386d6681800052c8af5b6e3a" gracePeriod=30 Sep 17 07:54:45.760761 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:54:45.760647 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-2589m" podUID="f2592b75-852d-4865-8157-9a3a9283a0e0" containerName="offline" containerID="cri-o://3e68739d56bfe8cb8df5599a36ced93c16a36a1b9b4c4841de68f2fbf9f178dc" gracePeriod=30 Sep 17 07:54:46.065019 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:54:46.064967 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dgt7q" podUID="51756a99-098d-4f45-8685-3484422723fe" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.39:8443: connect: connection refused" Sep 17 07:54:46.545268 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:54:46.545238 2578 generic.go:356] "Generic (PLEG): container finished" podID="f2592b75-852d-4865-8157-9a3a9283a0e0" containerID="d5d92cb2cec7a3810e0af5fa912145bd2b91a63f5bc2b09210ae6e608ef9654d" exitCode=0 Sep 17 07:54:46.545437 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:54:46.545308 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-2589m" event={"ID":"f2592b75-852d-4865-8157-9a3a9283a0e0","Type":"ContainerDied","Data":"d5d92cb2cec7a3810e0af5fa912145bd2b91a63f5bc2b09210ae6e608ef9654d"} Sep 17 07:54:46.548218 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:54:46.548197 2578 generic.go:356] "Generic (PLEG): container finished" podID="51756a99-098d-4f45-8685-3484422723fe" containerID="fed7abb5db452a6b1d16fb52b2650b2fb51c61a922dce1aed13c2b6871cca4b7" exitCode=0 Sep 17 07:54:46.548218 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:54:46.548216 2578 generic.go:356] "Generic (PLEG): container finished" podID="51756a99-098d-4f45-8685-3484422723fe" containerID="d23e9db3dff66498899b74e7efeb47908ff0babf2c5011d67c2e60ce8a7a6e5b" exitCode=0 Sep 17 07:54:46.548369 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:54:46.548267 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dgt7q" event={"ID":"51756a99-098d-4f45-8685-3484422723fe","Type":"ContainerDied","Data":"fed7abb5db452a6b1d16fb52b2650b2fb51c61a922dce1aed13c2b6871cca4b7"} Sep 17 07:54:46.548369 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:54:46.548296 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dgt7q" event={"ID":"51756a99-098d-4f45-8685-3484422723fe","Type":"ContainerDied","Data":"d23e9db3dff66498899b74e7efeb47908ff0babf2c5011d67c2e60ce8a7a6e5b"} Sep 17 07:54:47.555217 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:54:47.555189 2578 generic.go:356] "Generic (PLEG): container finished" podID="f2592b75-852d-4865-8157-9a3a9283a0e0" containerID="a9d496273f6547d3b779f5223ce327fd8d793409386d6681800052c8af5b6e3a" exitCode=0 Sep 17 07:54:47.555589 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:54:47.555238 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-2589m" event={"ID":"f2592b75-852d-4865-8157-9a3a9283a0e0","Type":"ContainerDied","Data":"a9d496273f6547d3b779f5223ce327fd8d793409386d6681800052c8af5b6e3a"} Sep 17 07:54:49.936568 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:54:49.936518 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-2589m" podUID="f2592b75-852d-4865-8157-9a3a9283a0e0" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.38:8443: connect: connection refused" Sep 17 07:54:50.823028 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:54:50.822982 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-2589m" podUID="f2592b75-852d-4865-8157-9a3a9283a0e0" containerName="online" probeResult="failure" output="Get \"https://10.132.0.38:6567/health\": dial tcp 10.132.0.38:6567: connect: connection refused" Sep 17 07:54:53.029770 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:54:53.029729 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dgt7q" podUID="51756a99-098d-4f45-8685-3484422723fe" containerName="online" probeResult="failure" output="Get \"https://10.132.0.39:6567/health\": dial tcp 10.132.0.39:6567: connect: connection refused" Sep 17 07:54:56.064664 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:54:56.064621 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dgt7q" podUID="51756a99-098d-4f45-8685-3484422723fe" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.39:8443: connect: connection refused" Sep 17 07:54:59.936716 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:54:59.936673 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-2589m" podUID="f2592b75-852d-4865-8157-9a3a9283a0e0" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.38:8443: connect: connection refused" Sep 17 07:55:00.823241 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:00.823202 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-2589m" podUID="f2592b75-852d-4865-8157-9a3a9283a0e0" containerName="online" probeResult="failure" output="Get \"https://10.132.0.38:6567/health\": dial tcp 10.132.0.38:6567: connect: connection refused" Sep 17 07:55:03.029535 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:03.029490 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dgt7q" podUID="51756a99-098d-4f45-8685-3484422723fe" containerName="online" probeResult="failure" output="Get \"https://10.132.0.39:6567/health\": dial tcp 10.132.0.39:6567: connect: connection refused" Sep 17 07:55:03.029943 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:03.029610 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dgt7q" Sep 17 07:55:06.064448 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:06.064402 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dgt7q" podUID="51756a99-098d-4f45-8685-3484422723fe" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.39:8443: connect: connection refused" Sep 17 07:55:06.064932 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:06.064534 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dgt7q" Sep 17 07:55:09.936255 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:09.936211 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-2589m" podUID="f2592b75-852d-4865-8157-9a3a9283a0e0" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.38:8443: connect: connection refused" Sep 17 07:55:09.936697 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:09.936341 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-2589m" Sep 17 07:55:10.823232 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:10.823198 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-2589m" podUID="f2592b75-852d-4865-8157-9a3a9283a0e0" containerName="online" probeResult="failure" output="Get \"https://10.132.0.38:6567/health\": dial tcp 10.132.0.38:6567: connect: connection refused" Sep 17 07:55:10.823411 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:10.823319 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-2589m" Sep 17 07:55:13.029660 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:13.029613 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dgt7q" podUID="51756a99-098d-4f45-8685-3484422723fe" containerName="online" probeResult="failure" output="Get \"https://10.132.0.39:6567/health\": dial tcp 10.132.0.39:6567: connect: connection refused" Sep 17 07:55:15.666631 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:15.666601 2578 generic.go:356] "Generic (PLEG): container finished" podID="51756a99-098d-4f45-8685-3484422723fe" containerID="a5fab8c2661716eb48e2b3ec947feb50cf736e77c952151429eb9d4f2ac5d3e4" exitCode=137 Sep 17 07:55:15.666955 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:15.666673 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dgt7q" event={"ID":"51756a99-098d-4f45-8685-3484422723fe","Type":"ContainerDied","Data":"a5fab8c2661716eb48e2b3ec947feb50cf736e77c952151429eb9d4f2ac5d3e4"} Sep 17 07:55:16.064506 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:16.064468 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dgt7q" podUID="51756a99-098d-4f45-8685-3484422723fe" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.39:8443: connect: connection refused" Sep 17 07:55:16.226096 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:16.226074 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dgt7q" Sep 17 07:55:16.347425 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:16.347400 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/51756a99-098d-4f45-8685-3484422723fe-feast-data\") pod \"51756a99-098d-4f45-8685-3484422723fe\" (UID: \"51756a99-098d-4f45-8685-3484422723fe\") " Sep 17 07:55:16.347558 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:16.347444 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/51756a99-098d-4f45-8685-3484422723fe-offline-tls\") pod \"51756a99-098d-4f45-8685-3484422723fe\" (UID: \"51756a99-098d-4f45-8685-3484422723fe\") " Sep 17 07:55:16.347558 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:16.347466 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/51756a99-098d-4f45-8685-3484422723fe-registry-tls\") pod \"51756a99-098d-4f45-8685-3484422723fe\" (UID: \"51756a99-098d-4f45-8685-3484422723fe\") " Sep 17 07:55:16.347558 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:16.347491 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6xwn6\" (UniqueName: \"kubernetes.io/projected/51756a99-098d-4f45-8685-3484422723fe-kube-api-access-6xwn6\") pod \"51756a99-098d-4f45-8685-3484422723fe\" (UID: \"51756a99-098d-4f45-8685-3484422723fe\") " Sep 17 07:55:16.347558 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:16.347515 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/51756a99-098d-4f45-8685-3484422723fe-online-tls\") pod \"51756a99-098d-4f45-8685-3484422723fe\" (UID: \"51756a99-098d-4f45-8685-3484422723fe\") " Sep 17 07:55:16.347558 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:16.347543 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/51756a99-098d-4f45-8685-3484422723fe-ui-tls\") pod \"51756a99-098d-4f45-8685-3484422723fe\" (UID: \"51756a99-098d-4f45-8685-3484422723fe\") " Sep 17 07:55:16.347910 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:16.347887 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51756a99-098d-4f45-8685-3484422723fe-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "51756a99-098d-4f45-8685-3484422723fe" (UID: "51756a99-098d-4f45-8685-3484422723fe"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 07:55:16.348019 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:16.347988 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51756a99-098d-4f45-8685-3484422723fe-feast-data" (OuterVolumeSpecName: "feast-data") pod "51756a99-098d-4f45-8685-3484422723fe" (UID: "51756a99-098d-4f45-8685-3484422723fe"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 07:55:16.349918 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:16.349891 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51756a99-098d-4f45-8685-3484422723fe-online-tls" (OuterVolumeSpecName: "online-tls") pod "51756a99-098d-4f45-8685-3484422723fe" (UID: "51756a99-098d-4f45-8685-3484422723fe"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 07:55:16.350041 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:16.350013 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51756a99-098d-4f45-8685-3484422723fe-kube-api-access-6xwn6" (OuterVolumeSpecName: "kube-api-access-6xwn6") pod "51756a99-098d-4f45-8685-3484422723fe" (UID: "51756a99-098d-4f45-8685-3484422723fe"). InnerVolumeSpecName "kube-api-access-6xwn6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 07:55:16.350110 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:16.350093 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51756a99-098d-4f45-8685-3484422723fe-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "51756a99-098d-4f45-8685-3484422723fe" (UID: "51756a99-098d-4f45-8685-3484422723fe"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 07:55:16.350155 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:16.350109 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51756a99-098d-4f45-8685-3484422723fe-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "51756a99-098d-4f45-8685-3484422723fe" (UID: "51756a99-098d-4f45-8685-3484422723fe"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 07:55:16.396643 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:16.396623 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-2589m" Sep 17 07:55:16.448209 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:16.448183 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/f2592b75-852d-4865-8157-9a3a9283a0e0-feast-data\") pod \"f2592b75-852d-4865-8157-9a3a9283a0e0\" (UID: \"f2592b75-852d-4865-8157-9a3a9283a0e0\") " Sep 17 07:55:16.448325 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:16.448220 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/f2592b75-852d-4865-8157-9a3a9283a0e0-registry-tls\") pod \"f2592b75-852d-4865-8157-9a3a9283a0e0\" (UID: \"f2592b75-852d-4865-8157-9a3a9283a0e0\") " Sep 17 07:55:16.448325 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:16.448236 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/f2592b75-852d-4865-8157-9a3a9283a0e0-online-tls\") pod \"f2592b75-852d-4865-8157-9a3a9283a0e0\" (UID: \"f2592b75-852d-4865-8157-9a3a9283a0e0\") " Sep 17 07:55:16.448325 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:16.448255 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/f2592b75-852d-4865-8157-9a3a9283a0e0-offline-tls\") pod \"f2592b75-852d-4865-8157-9a3a9283a0e0\" (UID: \"f2592b75-852d-4865-8157-9a3a9283a0e0\") " Sep 17 07:55:16.448325 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:16.448270 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/f2592b75-852d-4865-8157-9a3a9283a0e0-ui-tls\") pod \"f2592b75-852d-4865-8157-9a3a9283a0e0\" (UID: \"f2592b75-852d-4865-8157-9a3a9283a0e0\") " Sep 17 07:55:16.448325 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:16.448308 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cdf92\" (UniqueName: \"kubernetes.io/projected/f2592b75-852d-4865-8157-9a3a9283a0e0-kube-api-access-cdf92\") pod \"f2592b75-852d-4865-8157-9a3a9283a0e0\" (UID: \"f2592b75-852d-4865-8157-9a3a9283a0e0\") " Sep 17 07:55:16.448562 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:16.448502 2578 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/51756a99-098d-4f45-8685-3484422723fe-feast-data\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Sep 17 07:55:16.448562 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:16.448521 2578 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/51756a99-098d-4f45-8685-3484422723fe-offline-tls\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Sep 17 07:55:16.448562 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:16.448535 2578 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/51756a99-098d-4f45-8685-3484422723fe-registry-tls\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Sep 17 07:55:16.448562 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:16.448549 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xwn6\" (UniqueName: \"kubernetes.io/projected/51756a99-098d-4f45-8685-3484422723fe-kube-api-access-6xwn6\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Sep 17 07:55:16.448770 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:16.448563 2578 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/51756a99-098d-4f45-8685-3484422723fe-online-tls\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Sep 17 07:55:16.448770 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:16.448580 2578 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/51756a99-098d-4f45-8685-3484422723fe-ui-tls\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Sep 17 07:55:16.448850 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:16.448778 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2592b75-852d-4865-8157-9a3a9283a0e0-feast-data" (OuterVolumeSpecName: "feast-data") pod "f2592b75-852d-4865-8157-9a3a9283a0e0" (UID: "f2592b75-852d-4865-8157-9a3a9283a0e0"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 07:55:16.450432 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:16.450408 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2592b75-852d-4865-8157-9a3a9283a0e0-online-tls" (OuterVolumeSpecName: "online-tls") pod "f2592b75-852d-4865-8157-9a3a9283a0e0" (UID: "f2592b75-852d-4865-8157-9a3a9283a0e0"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 07:55:16.450904 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:16.450879 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2592b75-852d-4865-8157-9a3a9283a0e0-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "f2592b75-852d-4865-8157-9a3a9283a0e0" (UID: "f2592b75-852d-4865-8157-9a3a9283a0e0"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 07:55:16.450977 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:16.450879 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2592b75-852d-4865-8157-9a3a9283a0e0-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "f2592b75-852d-4865-8157-9a3a9283a0e0" (UID: "f2592b75-852d-4865-8157-9a3a9283a0e0"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 07:55:16.450977 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:16.450920 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2592b75-852d-4865-8157-9a3a9283a0e0-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "f2592b75-852d-4865-8157-9a3a9283a0e0" (UID: "f2592b75-852d-4865-8157-9a3a9283a0e0"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 07:55:16.450977 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:16.450961 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2592b75-852d-4865-8157-9a3a9283a0e0-kube-api-access-cdf92" (OuterVolumeSpecName: "kube-api-access-cdf92") pod "f2592b75-852d-4865-8157-9a3a9283a0e0" (UID: "f2592b75-852d-4865-8157-9a3a9283a0e0"). InnerVolumeSpecName "kube-api-access-cdf92". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 07:55:16.549400 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:16.549373 2578 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/f2592b75-852d-4865-8157-9a3a9283a0e0-feast-data\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Sep 17 07:55:16.549400 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:16.549396 2578 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/f2592b75-852d-4865-8157-9a3a9283a0e0-registry-tls\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Sep 17 07:55:16.549400 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:16.549406 2578 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/f2592b75-852d-4865-8157-9a3a9283a0e0-online-tls\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Sep 17 07:55:16.549599 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:16.549414 2578 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/f2592b75-852d-4865-8157-9a3a9283a0e0-offline-tls\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Sep 17 07:55:16.549599 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:16.549421 2578 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/f2592b75-852d-4865-8157-9a3a9283a0e0-ui-tls\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Sep 17 07:55:16.549599 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:16.549429 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdf92\" (UniqueName: \"kubernetes.io/projected/f2592b75-852d-4865-8157-9a3a9283a0e0-kube-api-access-cdf92\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Sep 17 07:55:16.673498 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:16.673473 2578 generic.go:356] "Generic (PLEG): container finished" podID="f2592b75-852d-4865-8157-9a3a9283a0e0" containerID="3e68739d56bfe8cb8df5599a36ced93c16a36a1b9b4c4841de68f2fbf9f178dc" exitCode=137 Sep 17 07:55:16.673498 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:16.673496 2578 generic.go:356] "Generic (PLEG): container finished" podID="f2592b75-852d-4865-8157-9a3a9283a0e0" containerID="5217e235d05ebbe5b084942ae5f28daaeb95dacace973f097d2be858db1f549d" exitCode=137 Sep 17 07:55:16.673957 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:16.673529 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-2589m" Sep 17 07:55:16.673957 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:16.673545 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-2589m" event={"ID":"f2592b75-852d-4865-8157-9a3a9283a0e0","Type":"ContainerDied","Data":"3e68739d56bfe8cb8df5599a36ced93c16a36a1b9b4c4841de68f2fbf9f178dc"} Sep 17 07:55:16.673957 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:16.673588 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-2589m" event={"ID":"f2592b75-852d-4865-8157-9a3a9283a0e0","Type":"ContainerDied","Data":"5217e235d05ebbe5b084942ae5f28daaeb95dacace973f097d2be858db1f549d"} Sep 17 07:55:16.673957 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:16.673605 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-2589m" event={"ID":"f2592b75-852d-4865-8157-9a3a9283a0e0","Type":"ContainerDied","Data":"4897ad9fb713734ac47e1cedc753038880b45a363365a5b248fcb7984b2e5421"} Sep 17 07:55:16.673957 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:16.673622 2578 scope.go:117] "RemoveContainer" containerID="d5d92cb2cec7a3810e0af5fa912145bd2b91a63f5bc2b09210ae6e608ef9654d" Sep 17 07:55:16.676577 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:16.676557 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dgt7q" event={"ID":"51756a99-098d-4f45-8685-3484422723fe","Type":"ContainerDied","Data":"9aa2bddfc3476227789b00a4a277ed636708457d729a2a5986f1bb5552c4ca94"} Sep 17 07:55:16.676664 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:16.676569 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dgt7q" Sep 17 07:55:16.682084 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:16.682072 2578 scope.go:117] "RemoveContainer" containerID="3e68739d56bfe8cb8df5599a36ced93c16a36a1b9b4c4841de68f2fbf9f178dc" Sep 17 07:55:16.689115 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:16.689099 2578 scope.go:117] "RemoveContainer" containerID="a9d496273f6547d3b779f5223ce327fd8d793409386d6681800052c8af5b6e3a" Sep 17 07:55:16.696103 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:16.696088 2578 scope.go:117] "RemoveContainer" containerID="5217e235d05ebbe5b084942ae5f28daaeb95dacace973f097d2be858db1f549d" Sep 17 07:55:16.702836 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:16.702821 2578 scope.go:117] "RemoveContainer" containerID="999bdf5c067843fe57992a840f69a9e85357ab1b1e39a062c74a8bcc03cb7910" Sep 17 07:55:16.705287 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:16.705268 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-2589m"] Sep 17 07:55:16.708473 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:16.708454 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-2589m"] Sep 17 07:55:16.710488 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:16.710467 2578 scope.go:117] "RemoveContainer" containerID="504309b255fbaffd3ecad8ed5ac7adff267ac747855536d9cc5bfaa6d320a7f4" Sep 17 07:55:16.717262 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:16.717245 2578 scope.go:117] "RemoveContainer" containerID="d5d92cb2cec7a3810e0af5fa912145bd2b91a63f5bc2b09210ae6e608ef9654d" Sep 17 07:55:16.717504 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:55:16.717486 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5d92cb2cec7a3810e0af5fa912145bd2b91a63f5bc2b09210ae6e608ef9654d\": container with ID starting with d5d92cb2cec7a3810e0af5fa912145bd2b91a63f5bc2b09210ae6e608ef9654d not found: ID does not exist" containerID="d5d92cb2cec7a3810e0af5fa912145bd2b91a63f5bc2b09210ae6e608ef9654d" Sep 17 07:55:16.717554 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:16.717511 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5d92cb2cec7a3810e0af5fa912145bd2b91a63f5bc2b09210ae6e608ef9654d"} err="failed to get container status \"d5d92cb2cec7a3810e0af5fa912145bd2b91a63f5bc2b09210ae6e608ef9654d\": rpc error: code = NotFound desc = could not find container \"d5d92cb2cec7a3810e0af5fa912145bd2b91a63f5bc2b09210ae6e608ef9654d\": container with ID starting with d5d92cb2cec7a3810e0af5fa912145bd2b91a63f5bc2b09210ae6e608ef9654d not found: ID does not exist" Sep 17 07:55:16.717554 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:16.717526 2578 scope.go:117] "RemoveContainer" containerID="3e68739d56bfe8cb8df5599a36ced93c16a36a1b9b4c4841de68f2fbf9f178dc" Sep 17 07:55:16.717764 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:55:16.717749 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e68739d56bfe8cb8df5599a36ced93c16a36a1b9b4c4841de68f2fbf9f178dc\": container with ID starting with 3e68739d56bfe8cb8df5599a36ced93c16a36a1b9b4c4841de68f2fbf9f178dc not found: ID does not exist" containerID="3e68739d56bfe8cb8df5599a36ced93c16a36a1b9b4c4841de68f2fbf9f178dc" Sep 17 07:55:16.717813 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:16.717767 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e68739d56bfe8cb8df5599a36ced93c16a36a1b9b4c4841de68f2fbf9f178dc"} err="failed to get container status \"3e68739d56bfe8cb8df5599a36ced93c16a36a1b9b4c4841de68f2fbf9f178dc\": rpc error: code = NotFound desc = could not find container \"3e68739d56bfe8cb8df5599a36ced93c16a36a1b9b4c4841de68f2fbf9f178dc\": container with ID starting with 3e68739d56bfe8cb8df5599a36ced93c16a36a1b9b4c4841de68f2fbf9f178dc not found: ID does not exist" Sep 17 07:55:16.717813 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:16.717779 2578 scope.go:117] "RemoveContainer" containerID="a9d496273f6547d3b779f5223ce327fd8d793409386d6681800052c8af5b6e3a" Sep 17 07:55:16.718088 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:55:16.718069 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9d496273f6547d3b779f5223ce327fd8d793409386d6681800052c8af5b6e3a\": container with ID starting with a9d496273f6547d3b779f5223ce327fd8d793409386d6681800052c8af5b6e3a not found: ID does not exist" containerID="a9d496273f6547d3b779f5223ce327fd8d793409386d6681800052c8af5b6e3a" Sep 17 07:55:16.718127 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:16.718093 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9d496273f6547d3b779f5223ce327fd8d793409386d6681800052c8af5b6e3a"} err="failed to get container status \"a9d496273f6547d3b779f5223ce327fd8d793409386d6681800052c8af5b6e3a\": rpc error: code = NotFound desc = could not find container \"a9d496273f6547d3b779f5223ce327fd8d793409386d6681800052c8af5b6e3a\": container with ID starting with a9d496273f6547d3b779f5223ce327fd8d793409386d6681800052c8af5b6e3a not found: ID does not exist" Sep 17 07:55:16.718127 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:16.718107 2578 scope.go:117] "RemoveContainer" containerID="5217e235d05ebbe5b084942ae5f28daaeb95dacace973f097d2be858db1f549d" Sep 17 07:55:16.718350 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:55:16.718333 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5217e235d05ebbe5b084942ae5f28daaeb95dacace973f097d2be858db1f549d\": container with ID starting with 5217e235d05ebbe5b084942ae5f28daaeb95dacace973f097d2be858db1f549d not found: ID does not exist" containerID="5217e235d05ebbe5b084942ae5f28daaeb95dacace973f097d2be858db1f549d" Sep 17 07:55:16.718402 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:16.718352 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5217e235d05ebbe5b084942ae5f28daaeb95dacace973f097d2be858db1f549d"} err="failed to get container status \"5217e235d05ebbe5b084942ae5f28daaeb95dacace973f097d2be858db1f549d\": rpc error: code = NotFound desc = could not find container \"5217e235d05ebbe5b084942ae5f28daaeb95dacace973f097d2be858db1f549d\": container with ID starting with 5217e235d05ebbe5b084942ae5f28daaeb95dacace973f097d2be858db1f549d not found: ID does not exist" Sep 17 07:55:16.718402 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:16.718366 2578 scope.go:117] "RemoveContainer" containerID="999bdf5c067843fe57992a840f69a9e85357ab1b1e39a062c74a8bcc03cb7910" Sep 17 07:55:16.718560 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:55:16.718546 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"999bdf5c067843fe57992a840f69a9e85357ab1b1e39a062c74a8bcc03cb7910\": container with ID starting with 999bdf5c067843fe57992a840f69a9e85357ab1b1e39a062c74a8bcc03cb7910 not found: ID does not exist" containerID="999bdf5c067843fe57992a840f69a9e85357ab1b1e39a062c74a8bcc03cb7910" Sep 17 07:55:16.718596 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:16.718562 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"999bdf5c067843fe57992a840f69a9e85357ab1b1e39a062c74a8bcc03cb7910"} err="failed to get container status \"999bdf5c067843fe57992a840f69a9e85357ab1b1e39a062c74a8bcc03cb7910\": rpc error: code = NotFound desc = could not find container \"999bdf5c067843fe57992a840f69a9e85357ab1b1e39a062c74a8bcc03cb7910\": container with ID starting with 999bdf5c067843fe57992a840f69a9e85357ab1b1e39a062c74a8bcc03cb7910 not found: ID does not exist" Sep 17 07:55:16.718596 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:16.718574 2578 scope.go:117] "RemoveContainer" containerID="504309b255fbaffd3ecad8ed5ac7adff267ac747855536d9cc5bfaa6d320a7f4" Sep 17 07:55:16.718791 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:55:16.718777 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"504309b255fbaffd3ecad8ed5ac7adff267ac747855536d9cc5bfaa6d320a7f4\": container with ID starting with 504309b255fbaffd3ecad8ed5ac7adff267ac747855536d9cc5bfaa6d320a7f4 not found: ID does not exist" containerID="504309b255fbaffd3ecad8ed5ac7adff267ac747855536d9cc5bfaa6d320a7f4" Sep 17 07:55:16.718840 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:16.718794 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"504309b255fbaffd3ecad8ed5ac7adff267ac747855536d9cc5bfaa6d320a7f4"} err="failed to get container status \"504309b255fbaffd3ecad8ed5ac7adff267ac747855536d9cc5bfaa6d320a7f4\": rpc error: code = NotFound desc = could not find container \"504309b255fbaffd3ecad8ed5ac7adff267ac747855536d9cc5bfaa6d320a7f4\": container with ID starting with 504309b255fbaffd3ecad8ed5ac7adff267ac747855536d9cc5bfaa6d320a7f4 not found: ID does not exist" Sep 17 07:55:16.718840 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:16.718806 2578 scope.go:117] "RemoveContainer" containerID="d5d92cb2cec7a3810e0af5fa912145bd2b91a63f5bc2b09210ae6e608ef9654d" Sep 17 07:55:16.719038 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:16.719022 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5d92cb2cec7a3810e0af5fa912145bd2b91a63f5bc2b09210ae6e608ef9654d"} err="failed to get container status \"d5d92cb2cec7a3810e0af5fa912145bd2b91a63f5bc2b09210ae6e608ef9654d\": rpc error: code = NotFound desc = could not find container \"d5d92cb2cec7a3810e0af5fa912145bd2b91a63f5bc2b09210ae6e608ef9654d\": container with ID starting with d5d92cb2cec7a3810e0af5fa912145bd2b91a63f5bc2b09210ae6e608ef9654d not found: ID does not exist" Sep 17 07:55:16.719038 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:16.719038 2578 scope.go:117] "RemoveContainer" containerID="3e68739d56bfe8cb8df5599a36ced93c16a36a1b9b4c4841de68f2fbf9f178dc" Sep 17 07:55:16.719269 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:16.719253 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e68739d56bfe8cb8df5599a36ced93c16a36a1b9b4c4841de68f2fbf9f178dc"} err="failed to get container status \"3e68739d56bfe8cb8df5599a36ced93c16a36a1b9b4c4841de68f2fbf9f178dc\": rpc error: code = NotFound desc = could not find container \"3e68739d56bfe8cb8df5599a36ced93c16a36a1b9b4c4841de68f2fbf9f178dc\": container with ID starting with 3e68739d56bfe8cb8df5599a36ced93c16a36a1b9b4c4841de68f2fbf9f178dc not found: ID does not exist" Sep 17 07:55:16.719269 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:16.719267 2578 scope.go:117] "RemoveContainer" containerID="a9d496273f6547d3b779f5223ce327fd8d793409386d6681800052c8af5b6e3a" Sep 17 07:55:16.719505 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:16.719487 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9d496273f6547d3b779f5223ce327fd8d793409386d6681800052c8af5b6e3a"} err="failed to get container status \"a9d496273f6547d3b779f5223ce327fd8d793409386d6681800052c8af5b6e3a\": rpc error: code = NotFound desc = could not find container \"a9d496273f6547d3b779f5223ce327fd8d793409386d6681800052c8af5b6e3a\": container with ID starting with a9d496273f6547d3b779f5223ce327fd8d793409386d6681800052c8af5b6e3a not found: ID does not exist" Sep 17 07:55:16.719565 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:16.719506 2578 scope.go:117] "RemoveContainer" containerID="5217e235d05ebbe5b084942ae5f28daaeb95dacace973f097d2be858db1f549d" Sep 17 07:55:16.719741 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:16.719724 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5217e235d05ebbe5b084942ae5f28daaeb95dacace973f097d2be858db1f549d"} err="failed to get container status \"5217e235d05ebbe5b084942ae5f28daaeb95dacace973f097d2be858db1f549d\": rpc error: code = NotFound desc = could not find container \"5217e235d05ebbe5b084942ae5f28daaeb95dacace973f097d2be858db1f549d\": container with ID starting with 5217e235d05ebbe5b084942ae5f28daaeb95dacace973f097d2be858db1f549d not found: ID does not exist" Sep 17 07:55:16.719741 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:16.719740 2578 scope.go:117] "RemoveContainer" containerID="999bdf5c067843fe57992a840f69a9e85357ab1b1e39a062c74a8bcc03cb7910" Sep 17 07:55:16.719974 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:16.719957 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"999bdf5c067843fe57992a840f69a9e85357ab1b1e39a062c74a8bcc03cb7910"} err="failed to get container status \"999bdf5c067843fe57992a840f69a9e85357ab1b1e39a062c74a8bcc03cb7910\": rpc error: code = NotFound desc = could not find container \"999bdf5c067843fe57992a840f69a9e85357ab1b1e39a062c74a8bcc03cb7910\": container with ID starting with 999bdf5c067843fe57992a840f69a9e85357ab1b1e39a062c74a8bcc03cb7910 not found: ID does not exist" Sep 17 07:55:16.720026 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:16.719974 2578 scope.go:117] "RemoveContainer" containerID="504309b255fbaffd3ecad8ed5ac7adff267ac747855536d9cc5bfaa6d320a7f4" Sep 17 07:55:16.720183 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:16.720166 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"504309b255fbaffd3ecad8ed5ac7adff267ac747855536d9cc5bfaa6d320a7f4"} err="failed to get container status \"504309b255fbaffd3ecad8ed5ac7adff267ac747855536d9cc5bfaa6d320a7f4\": rpc error: code = NotFound desc = could not find container \"504309b255fbaffd3ecad8ed5ac7adff267ac747855536d9cc5bfaa6d320a7f4\": container with ID starting with 504309b255fbaffd3ecad8ed5ac7adff267ac747855536d9cc5bfaa6d320a7f4 not found: ID does not exist" Sep 17 07:55:16.720225 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:16.720183 2578 scope.go:117] "RemoveContainer" containerID="fed7abb5db452a6b1d16fb52b2650b2fb51c61a922dce1aed13c2b6871cca4b7" Sep 17 07:55:16.726708 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:16.726693 2578 scope.go:117] "RemoveContainer" containerID="a5fab8c2661716eb48e2b3ec947feb50cf736e77c952151429eb9d4f2ac5d3e4" Sep 17 07:55:16.729694 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:16.729675 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dgt7q"] Sep 17 07:55:16.734152 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:16.734126 2578 scope.go:117] "RemoveContainer" containerID="d23e9db3dff66498899b74e7efeb47908ff0babf2c5011d67c2e60ce8a7a6e5b" Sep 17 07:55:16.734310 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:16.734295 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dgt7q"] Sep 17 07:55:16.742157 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:16.742074 2578 scope.go:117] "RemoveContainer" containerID="b21d7ea7f2f7f9830290fe64c080cd03f65e61297a8cd4cf9c267eae36d14b01" Sep 17 07:55:16.749140 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:16.749123 2578 scope.go:117] "RemoveContainer" containerID="ce383584b55b38e81363dc3c02c19ce05c6281af7b7435caa832cdedf4ad4ee4" Sep 17 07:55:18.628516 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:18.628484 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="51756a99-098d-4f45-8685-3484422723fe" path="/var/lib/kubelet/pods/51756a99-098d-4f45-8685-3484422723fe/volumes" Sep 17 07:55:18.629049 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:18.629034 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2592b75-852d-4865-8157-9a3a9283a0e0" path="/var/lib/kubelet/pods/f2592b75-852d-4865-8157-9a3a9283a0e0/volumes" Sep 17 07:55:59.737559 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:59.737479 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-7xfz8"] Sep 17 07:55:59.737974 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:59.737791 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2592b75-852d-4865-8157-9a3a9283a0e0" containerName="ui" Sep 17 07:55:59.737974 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:59.737811 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2592b75-852d-4865-8157-9a3a9283a0e0" containerName="ui" Sep 17 07:55:59.737974 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:59.737823 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51756a99-098d-4f45-8685-3484422723fe" containerName="online" Sep 17 07:55:59.737974 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:59.737828 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="51756a99-098d-4f45-8685-3484422723fe" containerName="online" Sep 17 07:55:59.737974 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:59.737842 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2592b75-852d-4865-8157-9a3a9283a0e0" containerName="online" Sep 17 07:55:59.737974 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:59.737848 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2592b75-852d-4865-8157-9a3a9283a0e0" containerName="online" Sep 17 07:55:59.737974 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:59.737854 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2592b75-852d-4865-8157-9a3a9283a0e0" containerName="feast-apply" Sep 17 07:55:59.737974 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:59.737872 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2592b75-852d-4865-8157-9a3a9283a0e0" containerName="feast-apply" Sep 17 07:55:59.737974 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:59.737880 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51756a99-098d-4f45-8685-3484422723fe" containerName="feast-init" Sep 17 07:55:59.737974 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:59.737885 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="51756a99-098d-4f45-8685-3484422723fe" containerName="feast-init" Sep 17 07:55:59.737974 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:59.737893 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2592b75-852d-4865-8157-9a3a9283a0e0" containerName="offline" Sep 17 07:55:59.737974 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:59.737898 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2592b75-852d-4865-8157-9a3a9283a0e0" containerName="offline" Sep 17 07:55:59.737974 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:59.737903 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51756a99-098d-4f45-8685-3484422723fe" containerName="feast-apply" Sep 17 07:55:59.737974 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:59.737909 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="51756a99-098d-4f45-8685-3484422723fe" containerName="feast-apply" Sep 17 07:55:59.737974 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:59.737914 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51756a99-098d-4f45-8685-3484422723fe" containerName="ui" Sep 17 07:55:59.737974 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:59.737919 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="51756a99-098d-4f45-8685-3484422723fe" containerName="ui" Sep 17 07:55:59.737974 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:59.737926 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2592b75-852d-4865-8157-9a3a9283a0e0" containerName="registry" Sep 17 07:55:59.737974 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:59.737930 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2592b75-852d-4865-8157-9a3a9283a0e0" containerName="registry" Sep 17 07:55:59.737974 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:59.737940 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51756a99-098d-4f45-8685-3484422723fe" containerName="offline" Sep 17 07:55:59.737974 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:59.737945 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="51756a99-098d-4f45-8685-3484422723fe" containerName="offline" Sep 17 07:55:59.737974 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:59.737953 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2592b75-852d-4865-8157-9a3a9283a0e0" containerName="feast-init" Sep 17 07:55:59.737974 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:59.737958 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2592b75-852d-4865-8157-9a3a9283a0e0" containerName="feast-init" Sep 17 07:55:59.738620 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:59.738001 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2592b75-852d-4865-8157-9a3a9283a0e0" containerName="ui" Sep 17 07:55:59.738620 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:59.738011 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2592b75-852d-4865-8157-9a3a9283a0e0" containerName="online" Sep 17 07:55:59.738620 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:59.738017 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="51756a99-098d-4f45-8685-3484422723fe" containerName="offline" Sep 17 07:55:59.738620 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:59.738022 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="51756a99-098d-4f45-8685-3484422723fe" containerName="online" Sep 17 07:55:59.738620 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:59.738029 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2592b75-852d-4865-8157-9a3a9283a0e0" containerName="registry" Sep 17 07:55:59.738620 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:59.738034 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2592b75-852d-4865-8157-9a3a9283a0e0" containerName="offline" Sep 17 07:55:59.738620 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:59.738040 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="51756a99-098d-4f45-8685-3484422723fe" containerName="ui" Sep 17 07:55:59.741080 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:59.741064 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7xfz8" Sep 17 07:55:59.746547 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:59.746517 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-ui-tls\"" Sep 17 07:55:59.746703 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:59.746675 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-online-tls\"" Sep 17 07:55:59.746797 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:59.746727 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-offline-tls\"" Sep 17 07:55:59.746980 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:59.746961 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\"" Sep 17 07:55:59.747229 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:59.747205 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-registry-tls\"" Sep 17 07:55:59.747769 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:59.747565 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\"" Sep 17 07:55:59.747769 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:59.747639 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-mzzbc\"" Sep 17 07:55:59.758917 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:59.758888 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-7xfz8"] Sep 17 07:55:59.876626 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:59.876594 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-7xfz8\" (UID: \"ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7xfz8" Sep 17 07:55:59.876784 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:59.876644 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-7xfz8\" (UID: \"ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7xfz8" Sep 17 07:55:59.876784 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:59.876678 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-7xfz8\" (UID: \"ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7xfz8" Sep 17 07:55:59.876784 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:59.876706 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7prvz\" (UniqueName: \"kubernetes.io/projected/ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f-kube-api-access-7prvz\") pod \"feast-simple-feast-setup-57576b8fcf-7xfz8\" (UID: \"ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7xfz8" Sep 17 07:55:59.876784 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:59.876729 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-7xfz8\" (UID: \"ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7xfz8" Sep 17 07:55:59.876784 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:59.876768 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f-feast-data\") pod \"feast-simple-feast-setup-57576b8fcf-7xfz8\" (UID: \"ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7xfz8" Sep 17 07:55:59.977578 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:59.977552 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-7xfz8\" (UID: \"ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7xfz8" Sep 17 07:55:59.977715 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:59.977590 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7prvz\" (UniqueName: \"kubernetes.io/projected/ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f-kube-api-access-7prvz\") pod \"feast-simple-feast-setup-57576b8fcf-7xfz8\" (UID: \"ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7xfz8" Sep 17 07:55:59.977715 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:59.977611 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-7xfz8\" (UID: \"ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7xfz8" Sep 17 07:55:59.977715 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:59.977627 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f-feast-data\") pod \"feast-simple-feast-setup-57576b8fcf-7xfz8\" (UID: \"ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7xfz8" Sep 17 07:55:59.977715 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:59.977651 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-7xfz8\" (UID: \"ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7xfz8" Sep 17 07:55:59.977715 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:59.977702 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-7xfz8\" (UID: \"ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7xfz8" Sep 17 07:55:59.978017 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:55:59.977822 2578 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-online-tls: secret "feast-simple-feast-setup-online-tls" not found Sep 17 07:55:59.978017 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:55:59.977911 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f-online-tls podName:ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f nodeName:}" failed. No retries permitted until 2026-09-17 07:56:00.477889648 +0000 UTC m=+1704.358923990 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f-online-tls") pod "feast-simple-feast-setup-57576b8fcf-7xfz8" (UID: "ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f") : secret "feast-simple-feast-setup-online-tls" not found Sep 17 07:55:59.978134 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:59.978067 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f-feast-data\") pod \"feast-simple-feast-setup-57576b8fcf-7xfz8\" (UID: \"ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7xfz8" Sep 17 07:55:59.980117 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:59.980093 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-7xfz8\" (UID: \"ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7xfz8" Sep 17 07:55:59.980247 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:59.980138 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-7xfz8\" (UID: \"ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7xfz8" Sep 17 07:55:59.980247 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:59.980203 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-7xfz8\" (UID: \"ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7xfz8" Sep 17 07:55:59.988523 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:55:59.988478 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7prvz\" (UniqueName: \"kubernetes.io/projected/ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f-kube-api-access-7prvz\") pod \"feast-simple-feast-setup-57576b8fcf-7xfz8\" (UID: \"ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7xfz8" Sep 17 07:56:00.482665 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:56:00.482634 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-7xfz8\" (UID: \"ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7xfz8" Sep 17 07:56:00.484935 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:56:00.484916 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-7xfz8\" (UID: \"ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7xfz8" Sep 17 07:56:00.660541 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:56:00.660508 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7xfz8" Sep 17 07:56:00.960010 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:56:00.959976 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-7xfz8"] Sep 17 07:56:00.963328 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:56:00.963290 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podac31d52d_bd7f_4bcc_8f88_f1b6c4af957f.slice/crio-ef10ee608b275ec6fcc1292b41c07e2eaf7ad5ca42e9b077169688a473c76e65 WatchSource:0}: Error finding container ef10ee608b275ec6fcc1292b41c07e2eaf7ad5ca42e9b077169688a473c76e65: Status 404 returned error can't find the container with id ef10ee608b275ec6fcc1292b41c07e2eaf7ad5ca42e9b077169688a473c76e65 Sep 17 07:56:01.855133 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:56:01.855091 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7xfz8" event={"ID":"ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f","Type":"ContainerStarted","Data":"c6b51db845e059823debde7fc9c5d96d47413c835795ba7d3bf94221927c0be8"} Sep 17 07:56:01.855305 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:56:01.855142 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7xfz8" event={"ID":"ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f","Type":"ContainerStarted","Data":"ef10ee608b275ec6fcc1292b41c07e2eaf7ad5ca42e9b077169688a473c76e65"} Sep 17 07:56:05.873870 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:56:05.873827 2578 generic.go:356] "Generic (PLEG): container finished" podID="ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f" containerID="c6b51db845e059823debde7fc9c5d96d47413c835795ba7d3bf94221927c0be8" exitCode=0 Sep 17 07:56:05.874173 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:56:05.873894 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7xfz8" event={"ID":"ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f","Type":"ContainerDied","Data":"c6b51db845e059823debde7fc9c5d96d47413c835795ba7d3bf94221927c0be8"} Sep 17 07:56:06.886101 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:56:06.886067 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7xfz8" event={"ID":"ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f","Type":"ContainerStarted","Data":"46d387bffc8be268a3b07ea224e6ede7b62fd7e807152389687d5edc111d192b"} Sep 17 07:56:11.906935 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:56:11.906905 2578 generic.go:356] "Generic (PLEG): container finished" podID="ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f" containerID="46d387bffc8be268a3b07ea224e6ede7b62fd7e807152389687d5edc111d192b" exitCode=0 Sep 17 07:56:11.907340 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:56:11.906959 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7xfz8" event={"ID":"ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f","Type":"ContainerDied","Data":"46d387bffc8be268a3b07ea224e6ede7b62fd7e807152389687d5edc111d192b"} Sep 17 07:56:12.916656 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:56:12.916618 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7xfz8" event={"ID":"ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f","Type":"ContainerStarted","Data":"ff2b942c797d4fa3e6dd269879a814aa228b8433f90f1e17fed317bb19d20aa9"} Sep 17 07:56:12.917187 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:56:12.916682 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7xfz8" event={"ID":"ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f","Type":"ContainerStarted","Data":"45670a94cc8d0d784e540fd722173bd3153dcc9ddc24dfd56fbcb8d4fd778b76"} Sep 17 07:56:12.917187 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:56:12.916703 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7xfz8" event={"ID":"ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f","Type":"ContainerStarted","Data":"5fddfb724823d29a8a250f2349ac93d9be2d7e1e57494ccb59bc206d9bafbdf0"} Sep 17 07:56:12.917187 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:56:12.916723 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7xfz8" event={"ID":"ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f","Type":"ContainerStarted","Data":"2efbd0d28a9a8c8e7d4e6affadf491bd42460b7ab7cf080cb462813aec2ff5ac"} Sep 17 07:56:12.951425 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:56:12.951365 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7xfz8" podStartSLOduration=13.951340839 podStartE2EDuration="13.951340839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 07:55:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 07:56:12.947316181 +0000 UTC m=+1716.828350540" watchObservedRunningTime="2026-09-17 07:56:12.951340839 +0000 UTC m=+1716.832375199" Sep 17 07:56:15.661607 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:56:15.661563 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7xfz8" Sep 17 07:56:15.662131 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:56:15.661633 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7xfz8" Sep 17 07:56:15.662131 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:56:15.661653 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7xfz8" Sep 17 07:56:15.662131 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:56:15.661671 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7xfz8" Sep 17 07:56:15.664004 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:56:15.663966 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7xfz8" podUID="ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.40:8443: connect: connection refused" Sep 17 07:56:15.664139 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:56:15.664009 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7xfz8" podUID="ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.40:8816: connect: connection refused" Sep 17 07:56:15.664139 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:56:15.663966 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7xfz8" podUID="ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.40:6571: connect: connection refused" Sep 17 07:56:15.664328 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:56:15.664287 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7xfz8" podUID="ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f" containerName="online" probeResult="failure" output="Get \"https://10.132.0.40:6567/health\": dial tcp 10.132.0.40:6567: connect: connection refused" Sep 17 07:56:18.661536 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:56:18.661480 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7xfz8" podUID="ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.40:6571: connect: connection refused" Sep 17 07:56:18.662035 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:56:18.661542 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7xfz8" podUID="ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.40:8443: connect: connection refused" Sep 17 07:56:18.662035 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:56:18.661488 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7xfz8" podUID="ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.40:8816: connect: connection refused" Sep 17 07:56:18.662035 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:56:18.661497 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7xfz8" podUID="ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f" containerName="online" probeResult="failure" output="Get \"https://10.132.0.40:6567/health\": dial tcp 10.132.0.40:6567: connect: connection refused" Sep 17 07:56:20.661486 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:56:20.661454 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7xfz8" Sep 17 07:56:20.661854 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:56:20.661502 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7xfz8" Sep 17 07:56:20.661854 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:56:20.661518 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7xfz8" Sep 17 07:56:20.661854 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:56:20.661530 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7xfz8" Sep 17 07:56:21.661822 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:56:21.661765 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7xfz8" Sep 17 07:56:21.662207 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:56:21.661982 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7xfz8" Sep 17 07:56:21.662207 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:56:21.662181 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7xfz8" Sep 17 07:56:21.662851 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:56:21.662827 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7xfz8" Sep 17 07:56:21.662991 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:56:21.662975 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7xfz8" Sep 17 07:56:21.663116 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:56:21.663105 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7xfz8" Sep 17 07:56:21.667389 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:56:21.667371 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7xfz8" Sep 17 07:56:21.967727 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:56:21.967649 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7xfz8" Sep 17 07:57:24.222368 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:57:24.222329 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-7xfz8"] Sep 17 07:57:24.222936 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:57:24.222644 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7xfz8" podUID="ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f" containerName="registry" containerID="cri-o://2efbd0d28a9a8c8e7d4e6affadf491bd42460b7ab7cf080cb462813aec2ff5ac" gracePeriod=30 Sep 17 07:57:24.222936 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:57:24.222739 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7xfz8" podUID="ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f" containerName="offline" containerID="cri-o://45670a94cc8d0d784e540fd722173bd3153dcc9ddc24dfd56fbcb8d4fd778b76" gracePeriod=30 Sep 17 07:57:24.222936 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:57:24.222772 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7xfz8" podUID="ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f" containerName="ui" containerID="cri-o://ff2b942c797d4fa3e6dd269879a814aa228b8433f90f1e17fed317bb19d20aa9" gracePeriod=30 Sep 17 07:57:24.222936 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:57:24.222793 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7xfz8" podUID="ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f" containerName="online" containerID="cri-o://5fddfb724823d29a8a250f2349ac93d9be2d7e1e57494ccb59bc206d9bafbdf0" gracePeriod=30 Sep 17 07:57:24.666612 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:57:24.666575 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-6xfzx"] Sep 17 07:57:24.669593 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:57:24.669574 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6xfzx" Sep 17 07:57:24.672164 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:57:24.672140 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-s9bsj\"" Sep 17 07:57:24.684846 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:57:24.684816 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-6xfzx"] Sep 17 07:57:24.746974 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:57:24.746942 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/b89d5a79-dac1-47a6-a3fd-8e617e696696-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-6xfzx\" (UID: \"b89d5a79-dac1-47a6-a3fd-8e617e696696\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6xfzx" Sep 17 07:57:24.747177 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:57:24.746980 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/b89d5a79-dac1-47a6-a3fd-8e617e696696-feast-data\") pod \"feast-simple-feast-setup-57576b8fcf-6xfzx\" (UID: \"b89d5a79-dac1-47a6-a3fd-8e617e696696\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6xfzx" Sep 17 07:57:24.747177 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:57:24.747035 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqptl\" (UniqueName: \"kubernetes.io/projected/b89d5a79-dac1-47a6-a3fd-8e617e696696-kube-api-access-sqptl\") pod \"feast-simple-feast-setup-57576b8fcf-6xfzx\" (UID: \"b89d5a79-dac1-47a6-a3fd-8e617e696696\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6xfzx" Sep 17 07:57:24.747177 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:57:24.747055 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/b89d5a79-dac1-47a6-a3fd-8e617e696696-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-6xfzx\" (UID: \"b89d5a79-dac1-47a6-a3fd-8e617e696696\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6xfzx" Sep 17 07:57:24.747177 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:57:24.747147 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/b89d5a79-dac1-47a6-a3fd-8e617e696696-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-6xfzx\" (UID: \"b89d5a79-dac1-47a6-a3fd-8e617e696696\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6xfzx" Sep 17 07:57:24.747397 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:57:24.747178 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/b89d5a79-dac1-47a6-a3fd-8e617e696696-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-6xfzx\" (UID: \"b89d5a79-dac1-47a6-a3fd-8e617e696696\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6xfzx" Sep 17 07:57:24.848311 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:57:24.848282 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqptl\" (UniqueName: \"kubernetes.io/projected/b89d5a79-dac1-47a6-a3fd-8e617e696696-kube-api-access-sqptl\") pod \"feast-simple-feast-setup-57576b8fcf-6xfzx\" (UID: \"b89d5a79-dac1-47a6-a3fd-8e617e696696\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6xfzx" Sep 17 07:57:24.848456 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:57:24.848323 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/b89d5a79-dac1-47a6-a3fd-8e617e696696-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-6xfzx\" (UID: \"b89d5a79-dac1-47a6-a3fd-8e617e696696\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6xfzx" Sep 17 07:57:24.848456 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:57:24.848357 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/b89d5a79-dac1-47a6-a3fd-8e617e696696-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-6xfzx\" (UID: \"b89d5a79-dac1-47a6-a3fd-8e617e696696\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6xfzx" Sep 17 07:57:24.848456 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:57:24.848390 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/b89d5a79-dac1-47a6-a3fd-8e617e696696-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-6xfzx\" (UID: \"b89d5a79-dac1-47a6-a3fd-8e617e696696\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6xfzx" Sep 17 07:57:24.848626 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:57:24.848458 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/b89d5a79-dac1-47a6-a3fd-8e617e696696-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-6xfzx\" (UID: \"b89d5a79-dac1-47a6-a3fd-8e617e696696\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6xfzx" Sep 17 07:57:24.848626 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:57:24.848491 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/b89d5a79-dac1-47a6-a3fd-8e617e696696-feast-data\") pod \"feast-simple-feast-setup-57576b8fcf-6xfzx\" (UID: \"b89d5a79-dac1-47a6-a3fd-8e617e696696\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6xfzx" Sep 17 07:57:24.848626 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:57:24.848512 2578 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-offline-tls: secret "feast-simple-feast-setup-offline-tls" not found Sep 17 07:57:24.848626 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:57:24.848594 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b89d5a79-dac1-47a6-a3fd-8e617e696696-offline-tls podName:b89d5a79-dac1-47a6-a3fd-8e617e696696 nodeName:}" failed. No retries permitted until 2026-09-17 07:57:25.348574159 +0000 UTC m=+1789.229608521 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/b89d5a79-dac1-47a6-a3fd-8e617e696696-offline-tls") pod "feast-simple-feast-setup-57576b8fcf-6xfzx" (UID: "b89d5a79-dac1-47a6-a3fd-8e617e696696") : secret "feast-simple-feast-setup-offline-tls" not found Sep 17 07:57:24.848626 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:57:24.848600 2578 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-registry-tls: secret "feast-simple-feast-setup-registry-tls" not found Sep 17 07:57:24.848924 ip-10-0-139-100 kubenswrapper[2578]: E0917 07:57:24.848672 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b89d5a79-dac1-47a6-a3fd-8e617e696696-registry-tls podName:b89d5a79-dac1-47a6-a3fd-8e617e696696 nodeName:}" failed. No retries permitted until 2026-09-17 07:57:25.348652721 +0000 UTC m=+1789.229687059 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/b89d5a79-dac1-47a6-a3fd-8e617e696696-registry-tls") pod "feast-simple-feast-setup-57576b8fcf-6xfzx" (UID: "b89d5a79-dac1-47a6-a3fd-8e617e696696") : secret "feast-simple-feast-setup-registry-tls" not found Sep 17 07:57:24.848924 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:57:24.848900 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/b89d5a79-dac1-47a6-a3fd-8e617e696696-feast-data\") pod \"feast-simple-feast-setup-57576b8fcf-6xfzx\" (UID: \"b89d5a79-dac1-47a6-a3fd-8e617e696696\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6xfzx" Sep 17 07:57:24.850773 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:57:24.850752 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/b89d5a79-dac1-47a6-a3fd-8e617e696696-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-6xfzx\" (UID: \"b89d5a79-dac1-47a6-a3fd-8e617e696696\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6xfzx" Sep 17 07:57:24.850900 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:57:24.850789 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/b89d5a79-dac1-47a6-a3fd-8e617e696696-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-6xfzx\" (UID: \"b89d5a79-dac1-47a6-a3fd-8e617e696696\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6xfzx" Sep 17 07:57:24.857017 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:57:24.856993 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqptl\" (UniqueName: \"kubernetes.io/projected/b89d5a79-dac1-47a6-a3fd-8e617e696696-kube-api-access-sqptl\") pod \"feast-simple-feast-setup-57576b8fcf-6xfzx\" (UID: \"b89d5a79-dac1-47a6-a3fd-8e617e696696\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6xfzx" Sep 17 07:57:25.207252 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:57:25.207182 2578 generic.go:356] "Generic (PLEG): container finished" podID="ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f" containerID="ff2b942c797d4fa3e6dd269879a814aa228b8433f90f1e17fed317bb19d20aa9" exitCode=0 Sep 17 07:57:25.207252 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:57:25.207205 2578 generic.go:356] "Generic (PLEG): container finished" podID="ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f" containerID="5fddfb724823d29a8a250f2349ac93d9be2d7e1e57494ccb59bc206d9bafbdf0" exitCode=0 Sep 17 07:57:25.207414 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:57:25.207254 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7xfz8" event={"ID":"ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f","Type":"ContainerDied","Data":"ff2b942c797d4fa3e6dd269879a814aa228b8433f90f1e17fed317bb19d20aa9"} Sep 17 07:57:25.207414 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:57:25.207294 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7xfz8" event={"ID":"ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f","Type":"ContainerDied","Data":"5fddfb724823d29a8a250f2349ac93d9be2d7e1e57494ccb59bc206d9bafbdf0"} Sep 17 07:57:25.352648 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:57:25.352618 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/b89d5a79-dac1-47a6-a3fd-8e617e696696-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-6xfzx\" (UID: \"b89d5a79-dac1-47a6-a3fd-8e617e696696\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6xfzx" Sep 17 07:57:25.353020 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:57:25.352673 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/b89d5a79-dac1-47a6-a3fd-8e617e696696-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-6xfzx\" (UID: \"b89d5a79-dac1-47a6-a3fd-8e617e696696\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6xfzx" Sep 17 07:57:25.355083 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:57:25.355061 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/b89d5a79-dac1-47a6-a3fd-8e617e696696-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-6xfzx\" (UID: \"b89d5a79-dac1-47a6-a3fd-8e617e696696\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6xfzx" Sep 17 07:57:25.355199 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:57:25.355101 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/b89d5a79-dac1-47a6-a3fd-8e617e696696-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-6xfzx\" (UID: \"b89d5a79-dac1-47a6-a3fd-8e617e696696\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6xfzx" Sep 17 07:57:25.578843 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:57:25.578759 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6xfzx" Sep 17 07:57:25.861566 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:57:25.861535 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-6xfzx"] Sep 17 07:57:25.863048 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:57:25.863024 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb89d5a79_dac1_47a6_a3fd_8e617e696696.slice/crio-98b9014f08c5de6a9f15f0eb7d6b08f5af67d43f250745eefa3ae2c21c62ef27 WatchSource:0}: Error finding container 98b9014f08c5de6a9f15f0eb7d6b08f5af67d43f250745eefa3ae2c21c62ef27: Status 404 returned error can't find the container with id 98b9014f08c5de6a9f15f0eb7d6b08f5af67d43f250745eefa3ae2c21c62ef27 Sep 17 07:57:26.212730 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:57:26.212688 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6xfzx" event={"ID":"b89d5a79-dac1-47a6-a3fd-8e617e696696","Type":"ContainerStarted","Data":"21cb90de4f773a763f8a3b0143d68eaf0f054dd12431932a9bb5e51fde8d041b"} Sep 17 07:57:26.212906 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:57:26.212749 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6xfzx" event={"ID":"b89d5a79-dac1-47a6-a3fd-8e617e696696","Type":"ContainerStarted","Data":"98b9014f08c5de6a9f15f0eb7d6b08f5af67d43f250745eefa3ae2c21c62ef27"} Sep 17 07:57:31.233994 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:57:31.233960 2578 generic.go:356] "Generic (PLEG): container finished" podID="b89d5a79-dac1-47a6-a3fd-8e617e696696" containerID="21cb90de4f773a763f8a3b0143d68eaf0f054dd12431932a9bb5e51fde8d041b" exitCode=0 Sep 17 07:57:31.234344 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:57:31.234000 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6xfzx" event={"ID":"b89d5a79-dac1-47a6-a3fd-8e617e696696","Type":"ContainerDied","Data":"21cb90de4f773a763f8a3b0143d68eaf0f054dd12431932a9bb5e51fde8d041b"} Sep 17 07:57:31.663279 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:57:31.663236 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7xfz8" podUID="ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.40:8443: connect: connection refused" Sep 17 07:57:31.965779 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:57:31.965692 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7xfz8" podUID="ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f" containerName="online" probeResult="failure" output="Get \"https://10.132.0.40:6567/health\": dial tcp 10.132.0.40:6567: connect: connection refused" Sep 17 07:57:32.240202 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:57:32.240125 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6xfzx" event={"ID":"b89d5a79-dac1-47a6-a3fd-8e617e696696","Type":"ContainerStarted","Data":"537d71f8d27be643688a2a4358d90310182821fa2cd50782d865e77a00e1281b"} Sep 17 07:57:37.262614 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:57:37.262522 2578 generic.go:356] "Generic (PLEG): container finished" podID="b89d5a79-dac1-47a6-a3fd-8e617e696696" containerID="537d71f8d27be643688a2a4358d90310182821fa2cd50782d865e77a00e1281b" exitCode=0 Sep 17 07:57:37.262614 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:57:37.262564 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6xfzx" event={"ID":"b89d5a79-dac1-47a6-a3fd-8e617e696696","Type":"ContainerDied","Data":"537d71f8d27be643688a2a4358d90310182821fa2cd50782d865e77a00e1281b"} Sep 17 07:57:38.272953 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:57:38.272900 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6xfzx" event={"ID":"b89d5a79-dac1-47a6-a3fd-8e617e696696","Type":"ContainerStarted","Data":"c8819636a695f2337d933a3916b8a131ea930bc7e6a1afec72915390fce39a35"} Sep 17 07:57:38.273456 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:57:38.272980 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6xfzx" event={"ID":"b89d5a79-dac1-47a6-a3fd-8e617e696696","Type":"ContainerStarted","Data":"ea36862fdd1829e1a4102aac61d75a8b7607730f5fc05df73e034e1f6fc32175"} Sep 17 07:57:38.273456 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:57:38.273000 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6xfzx" event={"ID":"b89d5a79-dac1-47a6-a3fd-8e617e696696","Type":"ContainerStarted","Data":"a37628d02529b2d0a21b6edca29f626ebfac833b4c167142e59bcd65e1a449ae"} Sep 17 07:57:38.273456 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:57:38.273017 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6xfzx" event={"ID":"b89d5a79-dac1-47a6-a3fd-8e617e696696","Type":"ContainerStarted","Data":"963de4287adb54e5aad457deb28b86dd1fac237a00bd5f7de2219403a770cd6a"} Sep 17 07:57:38.300362 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:57:38.300291 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6xfzx" podStartSLOduration=14.300269162 podStartE2EDuration="14.300269162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 07:57:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 07:57:38.297936823 +0000 UTC m=+1802.178971213" watchObservedRunningTime="2026-09-17 07:57:38.300269162 +0000 UTC m=+1802.181303522" Sep 17 07:57:40.579329 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:57:40.579283 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6xfzx" Sep 17 07:57:40.579933 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:57:40.579902 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6xfzx" Sep 17 07:57:40.579933 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:57:40.579936 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6xfzx" Sep 17 07:57:40.580137 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:57:40.579956 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6xfzx" Sep 17 07:57:40.581713 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:57:40.581679 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6xfzx" podUID="b89d5a79-dac1-47a6-a3fd-8e617e696696" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.41:8816: connect: connection refused" Sep 17 07:57:40.581836 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:57:40.581722 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6xfzx" podUID="b89d5a79-dac1-47a6-a3fd-8e617e696696" containerName="online" probeResult="failure" output="Get \"https://10.132.0.41:6567/health\": dial tcp 10.132.0.41:6567: connect: connection refused" Sep 17 07:57:40.581836 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:57:40.581744 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6xfzx" podUID="b89d5a79-dac1-47a6-a3fd-8e617e696696" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.41:8443: connect: connection refused" Sep 17 07:57:40.582048 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:57:40.582008 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6xfzx" podUID="b89d5a79-dac1-47a6-a3fd-8e617e696696" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.41:6571: connect: connection refused" Sep 17 07:57:41.662814 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:57:41.662767 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7xfz8" podUID="ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.40:8443: connect: connection refused" Sep 17 07:57:41.964929 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:57:41.964809 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7xfz8" podUID="ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f" containerName="online" probeResult="failure" output="Get \"https://10.132.0.40:6567/health\": dial tcp 10.132.0.40:6567: connect: connection refused" Sep 17 07:57:43.579628 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:57:43.579579 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6xfzx" podUID="b89d5a79-dac1-47a6-a3fd-8e617e696696" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.41:6571: connect: connection refused" Sep 17 07:57:43.580113 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:57:43.579583 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6xfzx" podUID="b89d5a79-dac1-47a6-a3fd-8e617e696696" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.41:8443: connect: connection refused" Sep 17 07:57:43.580113 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:57:43.579587 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6xfzx" podUID="b89d5a79-dac1-47a6-a3fd-8e617e696696" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.41:8816: connect: connection refused" Sep 17 07:57:43.580113 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:57:43.579873 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6xfzx" podUID="b89d5a79-dac1-47a6-a3fd-8e617e696696" containerName="online" probeResult="failure" output="Get \"https://10.132.0.41:6567/health\": dial tcp 10.132.0.41:6567: connect: connection refused" Sep 17 07:57:45.578932 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:57:45.578889 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6xfzx" Sep 17 07:57:45.579284 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:57:45.578949 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6xfzx" Sep 17 07:57:45.579284 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:57:45.578966 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6xfzx" Sep 17 07:57:45.579284 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:57:45.578980 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6xfzx" Sep 17 07:57:46.580726 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:57:46.580698 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6xfzx" Sep 17 07:57:46.581117 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:57:46.580810 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6xfzx" Sep 17 07:57:46.581117 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:57:46.580995 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6xfzx" Sep 17 07:57:46.585807 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:57:46.585791 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6xfzx" Sep 17 07:57:47.330691 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:57:47.330664 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6xfzx" Sep 17 07:57:47.330848 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:57:47.330731 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6xfzx" Sep 17 07:57:47.330848 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:57:47.330768 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6xfzx" Sep 17 07:57:47.333346 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:57:47.333327 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6xfzx" Sep 17 07:57:51.663536 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:57:51.663485 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7xfz8" podUID="ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.40:8443: connect: connection refused" Sep 17 07:57:51.664005 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:57:51.663655 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7xfz8" Sep 17 07:57:51.964916 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:57:51.964813 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7xfz8" podUID="ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f" containerName="online" probeResult="failure" output="Get \"https://10.132.0.40:6567/health\": dial tcp 10.132.0.40:6567: connect: connection refused" Sep 17 07:57:51.965056 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:57:51.964969 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7xfz8" Sep 17 07:57:54.359818 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:57:54.359793 2578 generic.go:356] "Generic (PLEG): container finished" podID="ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f" containerID="45670a94cc8d0d784e540fd722173bd3153dcc9ddc24dfd56fbcb8d4fd778b76" exitCode=137 Sep 17 07:57:54.359818 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:57:54.359816 2578 generic.go:356] "Generic (PLEG): container finished" podID="ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f" containerID="2efbd0d28a9a8c8e7d4e6affadf491bd42460b7ab7cf080cb462813aec2ff5ac" exitCode=137 Sep 17 07:57:54.360184 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:57:54.359880 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7xfz8" event={"ID":"ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f","Type":"ContainerDied","Data":"45670a94cc8d0d784e540fd722173bd3153dcc9ddc24dfd56fbcb8d4fd778b76"} Sep 17 07:57:54.360184 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:57:54.359921 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7xfz8" event={"ID":"ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f","Type":"ContainerDied","Data":"2efbd0d28a9a8c8e7d4e6affadf491bd42460b7ab7cf080cb462813aec2ff5ac"} Sep 17 07:57:54.863085 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:57:54.863058 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7xfz8" Sep 17 07:57:55.011319 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:57:55.011241 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f-feast-data\") pod \"ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f\" (UID: \"ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f\") " Sep 17 07:57:55.011319 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:57:55.011278 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f-online-tls\") pod \"ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f\" (UID: \"ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f\") " Sep 17 07:57:55.011319 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:57:55.011298 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f-offline-tls\") pod \"ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f\" (UID: \"ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f\") " Sep 17 07:57:55.011549 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:57:55.011346 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7prvz\" (UniqueName: \"kubernetes.io/projected/ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f-kube-api-access-7prvz\") pod \"ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f\" (UID: \"ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f\") " Sep 17 07:57:55.011549 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:57:55.011404 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f-registry-tls\") pod \"ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f\" (UID: \"ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f\") " Sep 17 07:57:55.011549 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:57:55.011428 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f-ui-tls\") pod \"ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f\" (UID: \"ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f\") " Sep 17 07:57:55.011942 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:57:55.011851 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f-feast-data" (OuterVolumeSpecName: "feast-data") pod "ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f" (UID: "ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 07:57:55.013696 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:57:55.013668 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f" (UID: "ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 07:57:55.013811 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:57:55.013689 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f" (UID: "ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 07:57:55.013811 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:57:55.013711 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f-online-tls" (OuterVolumeSpecName: "online-tls") pod "ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f" (UID: "ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 07:57:55.014023 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:57:55.013913 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f" (UID: "ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 07:57:55.014023 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:57:55.013951 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f-kube-api-access-7prvz" (OuterVolumeSpecName: "kube-api-access-7prvz") pod "ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f" (UID: "ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f"). InnerVolumeSpecName "kube-api-access-7prvz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 07:57:55.112787 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:57:55.112762 2578 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f-registry-tls\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Sep 17 07:57:55.112787 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:57:55.112784 2578 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f-ui-tls\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Sep 17 07:57:55.112787 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:57:55.112793 2578 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f-feast-data\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Sep 17 07:57:55.113066 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:57:55.112800 2578 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f-online-tls\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Sep 17 07:57:55.113066 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:57:55.112808 2578 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f-offline-tls\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Sep 17 07:57:55.113066 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:57:55.112816 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7prvz\" (UniqueName: \"kubernetes.io/projected/ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f-kube-api-access-7prvz\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Sep 17 07:57:55.366714 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:57:55.366644 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7xfz8" Sep 17 07:57:55.366714 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:57:55.366651 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7xfz8" event={"ID":"ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f","Type":"ContainerDied","Data":"ef10ee608b275ec6fcc1292b41c07e2eaf7ad5ca42e9b077169688a473c76e65"} Sep 17 07:57:55.366714 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:57:55.366703 2578 scope.go:117] "RemoveContainer" containerID="ff2b942c797d4fa3e6dd269879a814aa228b8433f90f1e17fed317bb19d20aa9" Sep 17 07:57:55.376003 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:57:55.375985 2578 scope.go:117] "RemoveContainer" containerID="45670a94cc8d0d784e540fd722173bd3153dcc9ddc24dfd56fbcb8d4fd778b76" Sep 17 07:57:55.383686 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:57:55.383671 2578 scope.go:117] "RemoveContainer" containerID="5fddfb724823d29a8a250f2349ac93d9be2d7e1e57494ccb59bc206d9bafbdf0" Sep 17 07:57:55.391233 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:57:55.391204 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-7xfz8"] Sep 17 07:57:55.393580 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:57:55.393560 2578 scope.go:117] "RemoveContainer" containerID="2efbd0d28a9a8c8e7d4e6affadf491bd42460b7ab7cf080cb462813aec2ff5ac" Sep 17 07:57:55.394360 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:57:55.394342 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-7xfz8"] Sep 17 07:57:55.400802 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:57:55.400784 2578 scope.go:117] "RemoveContainer" containerID="46d387bffc8be268a3b07ea224e6ede7b62fd7e807152389687d5edc111d192b" Sep 17 07:57:55.407641 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:57:55.407623 2578 scope.go:117] "RemoveContainer" containerID="c6b51db845e059823debde7fc9c5d96d47413c835795ba7d3bf94221927c0be8" Sep 17 07:57:56.628742 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:57:56.628707 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f" path="/var/lib/kubelet/pods/ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f/volumes" Sep 17 07:58:49.655636 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:58:49.655599 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-jwrn8"] Sep 17 07:58:49.656155 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:58:49.656044 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f" containerName="registry" Sep 17 07:58:49.656155 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:58:49.656062 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f" containerName="registry" Sep 17 07:58:49.656155 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:58:49.656082 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f" containerName="feast-init" Sep 17 07:58:49.656155 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:58:49.656090 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f" containerName="feast-init" Sep 17 07:58:49.656155 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:58:49.656102 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f" containerName="feast-apply" Sep 17 07:58:49.656155 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:58:49.656110 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f" containerName="feast-apply" Sep 17 07:58:49.656155 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:58:49.656127 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f" containerName="online" Sep 17 07:58:49.656155 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:58:49.656136 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f" containerName="online" Sep 17 07:58:49.656155 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:58:49.656154 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f" containerName="offline" Sep 17 07:58:49.656616 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:58:49.656162 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f" containerName="offline" Sep 17 07:58:49.656616 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:58:49.656173 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f" containerName="ui" Sep 17 07:58:49.656616 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:58:49.656182 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f" containerName="ui" Sep 17 07:58:49.656616 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:58:49.656263 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f" containerName="registry" Sep 17 07:58:49.656616 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:58:49.656273 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f" containerName="ui" Sep 17 07:58:49.656616 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:58:49.656284 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f" containerName="offline" Sep 17 07:58:49.656616 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:58:49.656296 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac31d52d-bd7f-4bcc-8f88-f1b6c4af957f" containerName="online" Sep 17 07:58:49.660035 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:58:49.660014 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-jwrn8" Sep 17 07:58:49.662888 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:58:49.662853 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-ui-tls\"" Sep 17 07:58:49.663009 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:58:49.662945 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-client-ca\"" Sep 17 07:58:49.663082 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:58:49.663050 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"kube-root-ca.crt\"" Sep 17 07:58:49.663279 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:58:49.663264 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-online-tls\"" Sep 17 07:58:49.663766 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:58:49.663751 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-offline-tls\"" Sep 17 07:58:49.663836 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:58:49.663820 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"openshift-service-ca.crt\"" Sep 17 07:58:49.664086 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:58:49.664071 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-dockercfg-qm5vk\"" Sep 17 07:58:49.673180 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:58:49.673157 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-jwrn8"] Sep 17 07:58:49.724055 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:58:49.724024 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/2b99dbdb-c43b-4c59-84c0-4c52ca82aac6-offline-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-jwrn8\" (UID: \"2b99dbdb-c43b-4c59-84c0-4c52ca82aac6\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-jwrn8" Sep 17 07:58:49.724230 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:58:49.724084 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpn6z\" (UniqueName: \"kubernetes.io/projected/2b99dbdb-c43b-4c59-84c0-4c52ca82aac6-kube-api-access-wpn6z\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-jwrn8\" (UID: \"2b99dbdb-c43b-4c59-84c0-4c52ca82aac6\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-jwrn8" Sep 17 07:58:49.724230 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:58:49.724122 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/2b99dbdb-c43b-4c59-84c0-4c52ca82aac6-ui-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-jwrn8\" (UID: \"2b99dbdb-c43b-4c59-84c0-4c52ca82aac6\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-jwrn8" Sep 17 07:58:49.724230 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:58:49.724170 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/2b99dbdb-c43b-4c59-84c0-4c52ca82aac6-registry-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-jwrn8\" (UID: \"2b99dbdb-c43b-4c59-84c0-4c52ca82aac6\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-jwrn8" Sep 17 07:58:49.724230 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:58:49.724204 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/2b99dbdb-c43b-4c59-84c0-4c52ca82aac6-online-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-jwrn8\" (UID: \"2b99dbdb-c43b-4c59-84c0-4c52ca82aac6\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-jwrn8" Sep 17 07:58:49.724442 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:58:49.724240 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/2b99dbdb-c43b-4c59-84c0-4c52ca82aac6-feast-data\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-jwrn8\" (UID: \"2b99dbdb-c43b-4c59-84c0-4c52ca82aac6\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-jwrn8" Sep 17 07:58:49.824717 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:58:49.824685 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpn6z\" (UniqueName: \"kubernetes.io/projected/2b99dbdb-c43b-4c59-84c0-4c52ca82aac6-kube-api-access-wpn6z\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-jwrn8\" (UID: \"2b99dbdb-c43b-4c59-84c0-4c52ca82aac6\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-jwrn8" Sep 17 07:58:49.824887 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:58:49.824721 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/2b99dbdb-c43b-4c59-84c0-4c52ca82aac6-ui-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-jwrn8\" (UID: \"2b99dbdb-c43b-4c59-84c0-4c52ca82aac6\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-jwrn8" Sep 17 07:58:49.824887 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:58:49.824759 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/2b99dbdb-c43b-4c59-84c0-4c52ca82aac6-registry-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-jwrn8\" (UID: \"2b99dbdb-c43b-4c59-84c0-4c52ca82aac6\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-jwrn8" Sep 17 07:58:49.824887 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:58:49.824778 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/2b99dbdb-c43b-4c59-84c0-4c52ca82aac6-online-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-jwrn8\" (UID: \"2b99dbdb-c43b-4c59-84c0-4c52ca82aac6\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-jwrn8" Sep 17 07:58:49.824887 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:58:49.824794 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/2b99dbdb-c43b-4c59-84c0-4c52ca82aac6-feast-data\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-jwrn8\" (UID: \"2b99dbdb-c43b-4c59-84c0-4c52ca82aac6\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-jwrn8" Sep 17 07:58:49.824887 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:58:49.824815 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/2b99dbdb-c43b-4c59-84c0-4c52ca82aac6-offline-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-jwrn8\" (UID: \"2b99dbdb-c43b-4c59-84c0-4c52ca82aac6\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-jwrn8" Sep 17 07:58:49.825282 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:58:49.825243 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/2b99dbdb-c43b-4c59-84c0-4c52ca82aac6-feast-data\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-jwrn8\" (UID: \"2b99dbdb-c43b-4c59-84c0-4c52ca82aac6\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-jwrn8" Sep 17 07:58:49.825548 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:58:49.825526 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/2b99dbdb-c43b-4c59-84c0-4c52ca82aac6-registry-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-jwrn8\" (UID: \"2b99dbdb-c43b-4c59-84c0-4c52ca82aac6\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-jwrn8" Sep 17 07:58:49.827299 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:58:49.827271 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/2b99dbdb-c43b-4c59-84c0-4c52ca82aac6-ui-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-jwrn8\" (UID: \"2b99dbdb-c43b-4c59-84c0-4c52ca82aac6\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-jwrn8" Sep 17 07:58:49.827402 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:58:49.827344 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/2b99dbdb-c43b-4c59-84c0-4c52ca82aac6-offline-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-jwrn8\" (UID: \"2b99dbdb-c43b-4c59-84c0-4c52ca82aac6\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-jwrn8" Sep 17 07:58:49.827450 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:58:49.827437 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/2b99dbdb-c43b-4c59-84c0-4c52ca82aac6-online-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-jwrn8\" (UID: \"2b99dbdb-c43b-4c59-84c0-4c52ca82aac6\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-jwrn8" Sep 17 07:58:49.836402 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:58:49.836378 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpn6z\" (UniqueName: \"kubernetes.io/projected/2b99dbdb-c43b-4c59-84c0-4c52ca82aac6-kube-api-access-wpn6z\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-jwrn8\" (UID: \"2b99dbdb-c43b-4c59-84c0-4c52ca82aac6\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-jwrn8" Sep 17 07:58:49.969632 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:58:49.969568 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-jwrn8" Sep 17 07:58:50.252749 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:58:50.252677 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-jwrn8"] Sep 17 07:58:50.255943 ip-10-0-139-100 kubenswrapper[2578]: W0917 07:58:50.255914 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2b99dbdb_c43b_4c59_84c0_4c52ca82aac6.slice/crio-e3ef56b61e19f4cbae8aff0976d50561bb501c5b85935251cc2e70ebdb55570d WatchSource:0}: Error finding container e3ef56b61e19f4cbae8aff0976d50561bb501c5b85935251cc2e70ebdb55570d: Status 404 returned error can't find the container with id e3ef56b61e19f4cbae8aff0976d50561bb501c5b85935251cc2e70ebdb55570d Sep 17 07:58:50.589470 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:58:50.589374 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-jwrn8" event={"ID":"2b99dbdb-c43b-4c59-84c0-4c52ca82aac6","Type":"ContainerStarted","Data":"ded5117994624907a5a072b9ff15dbf1903892aa663b4c8805a3ff805eebef2d"} Sep 17 07:58:50.589470 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:58:50.589426 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-jwrn8" event={"ID":"2b99dbdb-c43b-4c59-84c0-4c52ca82aac6","Type":"ContainerStarted","Data":"e3ef56b61e19f4cbae8aff0976d50561bb501c5b85935251cc2e70ebdb55570d"} Sep 17 07:58:55.619019 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:58:55.618986 2578 generic.go:356] "Generic (PLEG): container finished" podID="2b99dbdb-c43b-4c59-84c0-4c52ca82aac6" containerID="ded5117994624907a5a072b9ff15dbf1903892aa663b4c8805a3ff805eebef2d" exitCode=0 Sep 17 07:58:55.619449 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:58:55.619031 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-jwrn8" event={"ID":"2b99dbdb-c43b-4c59-84c0-4c52ca82aac6","Type":"ContainerDied","Data":"ded5117994624907a5a072b9ff15dbf1903892aa663b4c8805a3ff805eebef2d"} Sep 17 07:58:56.628818 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:58:56.628780 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-jwrn8" event={"ID":"2b99dbdb-c43b-4c59-84c0-4c52ca82aac6","Type":"ContainerStarted","Data":"9d285cfb0c04b0f8cf49996089fce985fe255671689f9b796b59b8b6998cf401"} Sep 17 07:59:20.729922 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:59:20.729892 2578 generic.go:356] "Generic (PLEG): container finished" podID="2b99dbdb-c43b-4c59-84c0-4c52ca82aac6" containerID="9d285cfb0c04b0f8cf49996089fce985fe255671689f9b796b59b8b6998cf401" exitCode=0 Sep 17 07:59:20.730367 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:59:20.729934 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-jwrn8" event={"ID":"2b99dbdb-c43b-4c59-84c0-4c52ca82aac6","Type":"ContainerDied","Data":"9d285cfb0c04b0f8cf49996089fce985fe255671689f9b796b59b8b6998cf401"} Sep 17 07:59:21.738062 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:59:21.738024 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-jwrn8" event={"ID":"2b99dbdb-c43b-4c59-84c0-4c52ca82aac6","Type":"ContainerStarted","Data":"707ee5e85c740616faa429b00248d72875e79dfe5292a144d416ed9b52193004"} Sep 17 07:59:21.738427 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:59:21.738080 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-jwrn8" event={"ID":"2b99dbdb-c43b-4c59-84c0-4c52ca82aac6","Type":"ContainerStarted","Data":"2eb11ef299a5d8091abe281c71de6e1afb8248dd4ed6886702057d577724eebe"} Sep 17 07:59:21.738427 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:59:21.738099 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-jwrn8" event={"ID":"2b99dbdb-c43b-4c59-84c0-4c52ca82aac6","Type":"ContainerStarted","Data":"41bf02fa750ee6fda225262289e56606f30eadcaa4cbc4d2ef11acc2492e3caa"} Sep 17 07:59:21.766244 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:59:21.766182 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-jwrn8" podStartSLOduration=32.7661609 podStartE2EDuration="32.7661609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 07:58:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 07:59:21.763139003 +0000 UTC m=+1905.644173363" watchObservedRunningTime="2026-09-17 07:59:21.7661609 +0000 UTC m=+1905.647195262" Sep 17 07:59:22.969971 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:59:22.969928 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-jwrn8" Sep 17 07:59:22.970457 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:59:22.969997 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-jwrn8" Sep 17 07:59:22.970457 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:59:22.970017 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-jwrn8" Sep 17 07:59:22.971500 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:59:22.971473 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-jwrn8" podUID="2b99dbdb-c43b-4c59-84c0-4c52ca82aac6" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.42:8443: connect: connection refused" Sep 17 07:59:22.971590 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:59:22.971472 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-jwrn8" podUID="2b99dbdb-c43b-4c59-84c0-4c52ca82aac6" containerName="online" probeResult="failure" output="Get \"https://10.132.0.42:6567/health\": dial tcp 10.132.0.42:6567: connect: connection refused" Sep 17 07:59:22.971657 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:59:22.971466 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-jwrn8" podUID="2b99dbdb-c43b-4c59-84c0-4c52ca82aac6" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.42:8816: connect: connection refused" Sep 17 07:59:25.969957 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:59:25.969913 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-jwrn8" podUID="2b99dbdb-c43b-4c59-84c0-4c52ca82aac6" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.42:8443: connect: connection refused" Sep 17 07:59:25.970542 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:59:25.969922 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-jwrn8" podUID="2b99dbdb-c43b-4c59-84c0-4c52ca82aac6" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.42:8816: connect: connection refused" Sep 17 07:59:25.970669 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:59:25.970648 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-jwrn8" podUID="2b99dbdb-c43b-4c59-84c0-4c52ca82aac6" containerName="online" probeResult="failure" output="Get \"https://10.132.0.42:6567/health\": dial tcp 10.132.0.42:6567: connect: connection refused" Sep 17 07:59:28.970130 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:59:28.970080 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-jwrn8" podUID="2b99dbdb-c43b-4c59-84c0-4c52ca82aac6" containerName="online" probeResult="failure" output="Get \"https://10.132.0.42:6567/health\": dial tcp 10.132.0.42:6567: connect: connection refused" Sep 17 07:59:28.970929 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:59:28.970903 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-jwrn8" Sep 17 07:59:28.971027 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:59:28.971016 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-jwrn8" Sep 17 07:59:28.971541 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:59:28.971518 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-jwrn8" Sep 17 07:59:28.971637 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:59:28.971553 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-jwrn8" Sep 17 07:59:28.972001 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:59:28.971981 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-jwrn8" Sep 17 07:59:28.972087 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:59:28.972043 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-jwrn8" Sep 17 07:59:29.970514 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:59:29.970473 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-jwrn8" Sep 17 07:59:32.809055 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:59:32.809024 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-jwrn8" Sep 17 07:59:33.578322 ip-10-0-139-100 kubenswrapper[2578]: I0917 07:59:33.578292 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-jwrn8" Sep 17 08:00:03.815005 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:00:03.814959 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-jwrn8" podUID="2b99dbdb-c43b-4c59-84c0-4c52ca82aac6" containerName="online" probeResult="failure" output="Get \"https://10.132.0.42:6567/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 17 08:00:33.814047 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:00:33.813954 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-jwrn8" podUID="2b99dbdb-c43b-4c59-84c0-4c52ca82aac6" containerName="online" probeResult="failure" output="Get \"https://10.132.0.42:6567/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Sep 17 08:00:35.829381 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:00:35.829348 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-jwrn8"] Sep 17 08:00:35.829788 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:00:35.829564 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-jwrn8" podUID="2b99dbdb-c43b-4c59-84c0-4c52ca82aac6" containerName="online" containerID="cri-o://41bf02fa750ee6fda225262289e56606f30eadcaa4cbc4d2ef11acc2492e3caa" gracePeriod=30 Sep 17 08:00:35.829788 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:00:35.829618 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-jwrn8" podUID="2b99dbdb-c43b-4c59-84c0-4c52ca82aac6" containerName="ui" containerID="cri-o://707ee5e85c740616faa429b00248d72875e79dfe5292a144d416ed9b52193004" gracePeriod=30 Sep 17 08:00:35.829788 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:00:35.829633 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-jwrn8" podUID="2b99dbdb-c43b-4c59-84c0-4c52ca82aac6" containerName="offline" containerID="cri-o://2eb11ef299a5d8091abe281c71de6e1afb8248dd4ed6886702057d577724eebe" gracePeriod=30 Sep 17 08:00:35.979950 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:00:35.979916 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-6xfzx"] Sep 17 08:00:35.980151 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:00:35.980130 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6xfzx" podUID="b89d5a79-dac1-47a6-a3fd-8e617e696696" containerName="registry" containerID="cri-o://963de4287adb54e5aad457deb28b86dd1fac237a00bd5f7de2219403a770cd6a" gracePeriod=30 Sep 17 08:00:35.980275 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:00:35.980251 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6xfzx" podUID="b89d5a79-dac1-47a6-a3fd-8e617e696696" containerName="ui" containerID="cri-o://c8819636a695f2337d933a3916b8a131ea930bc7e6a1afec72915390fce39a35" gracePeriod=30 Sep 17 08:00:35.980332 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:00:35.980246 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6xfzx" podUID="b89d5a79-dac1-47a6-a3fd-8e617e696696" containerName="offline" containerID="cri-o://ea36862fdd1829e1a4102aac61d75a8b7607730f5fc05df73e034e1f6fc32175" gracePeriod=30 Sep 17 08:00:35.980332 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:00:35.980290 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6xfzx" podUID="b89d5a79-dac1-47a6-a3fd-8e617e696696" containerName="online" containerID="cri-o://a37628d02529b2d0a21b6edca29f626ebfac833b4c167142e59bcd65e1a449ae" gracePeriod=30 Sep 17 08:00:37.049750 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:00:37.049719 2578 generic.go:356] "Generic (PLEG): container finished" podID="b89d5a79-dac1-47a6-a3fd-8e617e696696" containerID="c8819636a695f2337d933a3916b8a131ea930bc7e6a1afec72915390fce39a35" exitCode=0 Sep 17 08:00:37.049750 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:00:37.049744 2578 generic.go:356] "Generic (PLEG): container finished" podID="b89d5a79-dac1-47a6-a3fd-8e617e696696" containerID="a37628d02529b2d0a21b6edca29f626ebfac833b4c167142e59bcd65e1a449ae" exitCode=0 Sep 17 08:00:37.050235 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:00:37.049789 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6xfzx" event={"ID":"b89d5a79-dac1-47a6-a3fd-8e617e696696","Type":"ContainerDied","Data":"c8819636a695f2337d933a3916b8a131ea930bc7e6a1afec72915390fce39a35"} Sep 17 08:00:37.050235 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:00:37.049837 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6xfzx" event={"ID":"b89d5a79-dac1-47a6-a3fd-8e617e696696","Type":"ContainerDied","Data":"a37628d02529b2d0a21b6edca29f626ebfac833b4c167142e59bcd65e1a449ae"} Sep 17 08:00:37.052261 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:00:37.052238 2578 generic.go:356] "Generic (PLEG): container finished" podID="2b99dbdb-c43b-4c59-84c0-4c52ca82aac6" containerID="707ee5e85c740616faa429b00248d72875e79dfe5292a144d416ed9b52193004" exitCode=0 Sep 17 08:00:37.052261 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:00:37.052258 2578 generic.go:356] "Generic (PLEG): container finished" podID="2b99dbdb-c43b-4c59-84c0-4c52ca82aac6" containerID="41bf02fa750ee6fda225262289e56606f30eadcaa4cbc4d2ef11acc2492e3caa" exitCode=0 Sep 17 08:00:37.052416 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:00:37.052284 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-jwrn8" event={"ID":"2b99dbdb-c43b-4c59-84c0-4c52ca82aac6","Type":"ContainerDied","Data":"707ee5e85c740616faa429b00248d72875e79dfe5292a144d416ed9b52193004"} Sep 17 08:00:37.052416 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:00:37.052327 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-jwrn8" event={"ID":"2b99dbdb-c43b-4c59-84c0-4c52ca82aac6","Type":"ContainerDied","Data":"41bf02fa750ee6fda225262289e56606f30eadcaa4cbc4d2ef11acc2492e3caa"} Sep 17 08:00:37.330820 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:00:37.330713 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6xfzx" podUID="b89d5a79-dac1-47a6-a3fd-8e617e696696" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.41:8443: connect: connection refused" Sep 17 08:00:37.330820 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:00:37.330784 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6xfzx" podUID="b89d5a79-dac1-47a6-a3fd-8e617e696696" containerName="online" probeResult="failure" output="Get \"https://10.132.0.41:6567/health\": dial tcp 10.132.0.41:6567: connect: connection refused" Sep 17 08:00:38.972179 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:00:38.972138 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-jwrn8" podUID="2b99dbdb-c43b-4c59-84c0-4c52ca82aac6" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.42:8443: connect: connection refused" Sep 17 08:00:42.809691 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:00:42.809653 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-jwrn8" podUID="2b99dbdb-c43b-4c59-84c0-4c52ca82aac6" containerName="online" probeResult="failure" output="Get \"https://10.132.0.42:6567/health\": dial tcp 10.132.0.42:6567: connect: connection refused" Sep 17 08:00:47.330763 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:00:47.330711 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6xfzx" podUID="b89d5a79-dac1-47a6-a3fd-8e617e696696" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.41:8443: connect: connection refused" Sep 17 08:00:47.331211 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:00:47.330731 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6xfzx" podUID="b89d5a79-dac1-47a6-a3fd-8e617e696696" containerName="online" probeResult="failure" output="Get \"https://10.132.0.41:6567/health\": dial tcp 10.132.0.41:6567: connect: connection refused" Sep 17 08:00:48.972336 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:00:48.972294 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-jwrn8" podUID="2b99dbdb-c43b-4c59-84c0-4c52ca82aac6" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.42:8443: connect: connection refused" Sep 17 08:00:52.809951 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:00:52.809914 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-jwrn8" podUID="2b99dbdb-c43b-4c59-84c0-4c52ca82aac6" containerName="online" probeResult="failure" output="Get \"https://10.132.0.42:6567/health\": dial tcp 10.132.0.42:6567: connect: connection refused" Sep 17 08:00:52.810396 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:00:52.810025 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-jwrn8" Sep 17 08:00:57.331078 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:00:57.331033 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6xfzx" podUID="b89d5a79-dac1-47a6-a3fd-8e617e696696" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.41:8443: connect: connection refused" Sep 17 08:00:57.331078 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:00:57.331055 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6xfzx" podUID="b89d5a79-dac1-47a6-a3fd-8e617e696696" containerName="online" probeResult="failure" output="Get \"https://10.132.0.41:6567/health\": dial tcp 10.132.0.41:6567: connect: connection refused" Sep 17 08:00:57.331573 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:00:57.331177 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6xfzx" Sep 17 08:00:57.331573 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:00:57.331248 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6xfzx" Sep 17 08:00:58.971782 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:00:58.971732 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-jwrn8" podUID="2b99dbdb-c43b-4c59-84c0-4c52ca82aac6" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.42:8443: connect: connection refused" Sep 17 08:00:58.972193 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:00:58.971852 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-jwrn8" Sep 17 08:01:02.810583 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:02.810540 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-jwrn8" podUID="2b99dbdb-c43b-4c59-84c0-4c52ca82aac6" containerName="online" probeResult="failure" output="Get \"https://10.132.0.42:6567/health\": dial tcp 10.132.0.42:6567: connect: connection refused" Sep 17 08:01:06.169036 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:06.169008 2578 generic.go:356] "Generic (PLEG): container finished" podID="b89d5a79-dac1-47a6-a3fd-8e617e696696" containerID="ea36862fdd1829e1a4102aac61d75a8b7607730f5fc05df73e034e1f6fc32175" exitCode=137 Sep 17 08:01:06.169036 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:06.169031 2578 generic.go:356] "Generic (PLEG): container finished" podID="b89d5a79-dac1-47a6-a3fd-8e617e696696" containerID="963de4287adb54e5aad457deb28b86dd1fac237a00bd5f7de2219403a770cd6a" exitCode=137 Sep 17 08:01:06.169418 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:06.169079 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6xfzx" event={"ID":"b89d5a79-dac1-47a6-a3fd-8e617e696696","Type":"ContainerDied","Data":"ea36862fdd1829e1a4102aac61d75a8b7607730f5fc05df73e034e1f6fc32175"} Sep 17 08:01:06.169418 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:06.169123 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6xfzx" event={"ID":"b89d5a79-dac1-47a6-a3fd-8e617e696696","Type":"ContainerDied","Data":"963de4287adb54e5aad457deb28b86dd1fac237a00bd5f7de2219403a770cd6a"} Sep 17 08:01:06.171572 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:06.171549 2578 generic.go:356] "Generic (PLEG): container finished" podID="2b99dbdb-c43b-4c59-84c0-4c52ca82aac6" containerID="2eb11ef299a5d8091abe281c71de6e1afb8248dd4ed6886702057d577724eebe" exitCode=137 Sep 17 08:01:06.171689 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:06.171624 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-jwrn8" event={"ID":"2b99dbdb-c43b-4c59-84c0-4c52ca82aac6","Type":"ContainerDied","Data":"2eb11ef299a5d8091abe281c71de6e1afb8248dd4ed6886702057d577724eebe"} Sep 17 08:01:06.473383 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:06.473358 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-jwrn8" Sep 17 08:01:06.609539 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:06.609515 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6xfzx" Sep 17 08:01:06.623557 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:06.623530 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/2b99dbdb-c43b-4c59-84c0-4c52ca82aac6-online-tls\") pod \"2b99dbdb-c43b-4c59-84c0-4c52ca82aac6\" (UID: \"2b99dbdb-c43b-4c59-84c0-4c52ca82aac6\") " Sep 17 08:01:06.623687 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:06.623592 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/2b99dbdb-c43b-4c59-84c0-4c52ca82aac6-registry-tls\") pod \"2b99dbdb-c43b-4c59-84c0-4c52ca82aac6\" (UID: \"2b99dbdb-c43b-4c59-84c0-4c52ca82aac6\") " Sep 17 08:01:06.623687 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:06.623624 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/2b99dbdb-c43b-4c59-84c0-4c52ca82aac6-offline-tls\") pod \"2b99dbdb-c43b-4c59-84c0-4c52ca82aac6\" (UID: \"2b99dbdb-c43b-4c59-84c0-4c52ca82aac6\") " Sep 17 08:01:06.623687 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:06.623670 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/2b99dbdb-c43b-4c59-84c0-4c52ca82aac6-ui-tls\") pod \"2b99dbdb-c43b-4c59-84c0-4c52ca82aac6\" (UID: \"2b99dbdb-c43b-4c59-84c0-4c52ca82aac6\") " Sep 17 08:01:06.623883 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:06.623695 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wpn6z\" (UniqueName: \"kubernetes.io/projected/2b99dbdb-c43b-4c59-84c0-4c52ca82aac6-kube-api-access-wpn6z\") pod \"2b99dbdb-c43b-4c59-84c0-4c52ca82aac6\" (UID: \"2b99dbdb-c43b-4c59-84c0-4c52ca82aac6\") " Sep 17 08:01:06.623883 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:06.623733 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/2b99dbdb-c43b-4c59-84c0-4c52ca82aac6-feast-data\") pod \"2b99dbdb-c43b-4c59-84c0-4c52ca82aac6\" (UID: \"2b99dbdb-c43b-4c59-84c0-4c52ca82aac6\") " Sep 17 08:01:06.624021 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:06.623994 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b99dbdb-c43b-4c59-84c0-4c52ca82aac6-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "2b99dbdb-c43b-4c59-84c0-4c52ca82aac6" (UID: "2b99dbdb-c43b-4c59-84c0-4c52ca82aac6"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 08:01:06.624439 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:06.624405 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b99dbdb-c43b-4c59-84c0-4c52ca82aac6-feast-data" (OuterVolumeSpecName: "feast-data") pod "2b99dbdb-c43b-4c59-84c0-4c52ca82aac6" (UID: "2b99dbdb-c43b-4c59-84c0-4c52ca82aac6"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 08:01:06.625927 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:06.625905 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b99dbdb-c43b-4c59-84c0-4c52ca82aac6-online-tls" (OuterVolumeSpecName: "online-tls") pod "2b99dbdb-c43b-4c59-84c0-4c52ca82aac6" (UID: "2b99dbdb-c43b-4c59-84c0-4c52ca82aac6"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 08:01:06.626012 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:06.625926 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b99dbdb-c43b-4c59-84c0-4c52ca82aac6-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "2b99dbdb-c43b-4c59-84c0-4c52ca82aac6" (UID: "2b99dbdb-c43b-4c59-84c0-4c52ca82aac6"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 08:01:06.626012 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:06.625946 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b99dbdb-c43b-4c59-84c0-4c52ca82aac6-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "2b99dbdb-c43b-4c59-84c0-4c52ca82aac6" (UID: "2b99dbdb-c43b-4c59-84c0-4c52ca82aac6"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 08:01:06.626188 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:06.626162 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b99dbdb-c43b-4c59-84c0-4c52ca82aac6-kube-api-access-wpn6z" (OuterVolumeSpecName: "kube-api-access-wpn6z") pod "2b99dbdb-c43b-4c59-84c0-4c52ca82aac6" (UID: "2b99dbdb-c43b-4c59-84c0-4c52ca82aac6"). InnerVolumeSpecName "kube-api-access-wpn6z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 08:01:06.725234 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:06.725151 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/b89d5a79-dac1-47a6-a3fd-8e617e696696-offline-tls\") pod \"b89d5a79-dac1-47a6-a3fd-8e617e696696\" (UID: \"b89d5a79-dac1-47a6-a3fd-8e617e696696\") " Sep 17 08:01:06.725234 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:06.725212 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/b89d5a79-dac1-47a6-a3fd-8e617e696696-registry-tls\") pod \"b89d5a79-dac1-47a6-a3fd-8e617e696696\" (UID: \"b89d5a79-dac1-47a6-a3fd-8e617e696696\") " Sep 17 08:01:06.725429 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:06.725241 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/b89d5a79-dac1-47a6-a3fd-8e617e696696-online-tls\") pod \"b89d5a79-dac1-47a6-a3fd-8e617e696696\" (UID: \"b89d5a79-dac1-47a6-a3fd-8e617e696696\") " Sep 17 08:01:06.725429 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:06.725284 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/b89d5a79-dac1-47a6-a3fd-8e617e696696-ui-tls\") pod \"b89d5a79-dac1-47a6-a3fd-8e617e696696\" (UID: \"b89d5a79-dac1-47a6-a3fd-8e617e696696\") " Sep 17 08:01:06.725429 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:06.725353 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sqptl\" (UniqueName: \"kubernetes.io/projected/b89d5a79-dac1-47a6-a3fd-8e617e696696-kube-api-access-sqptl\") pod \"b89d5a79-dac1-47a6-a3fd-8e617e696696\" (UID: \"b89d5a79-dac1-47a6-a3fd-8e617e696696\") " Sep 17 08:01:06.725429 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:06.725388 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/b89d5a79-dac1-47a6-a3fd-8e617e696696-feast-data\") pod \"b89d5a79-dac1-47a6-a3fd-8e617e696696\" (UID: \"b89d5a79-dac1-47a6-a3fd-8e617e696696\") " Sep 17 08:01:06.725646 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:06.725608 2578 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/2b99dbdb-c43b-4c59-84c0-4c52ca82aac6-offline-tls\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Sep 17 08:01:06.725646 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:06.725624 2578 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/2b99dbdb-c43b-4c59-84c0-4c52ca82aac6-ui-tls\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Sep 17 08:01:06.725646 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:06.725637 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wpn6z\" (UniqueName: \"kubernetes.io/projected/2b99dbdb-c43b-4c59-84c0-4c52ca82aac6-kube-api-access-wpn6z\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Sep 17 08:01:06.725791 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:06.725650 2578 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/2b99dbdb-c43b-4c59-84c0-4c52ca82aac6-feast-data\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Sep 17 08:01:06.725791 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:06.725662 2578 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/2b99dbdb-c43b-4c59-84c0-4c52ca82aac6-online-tls\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Sep 17 08:01:06.725791 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:06.725676 2578 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/2b99dbdb-c43b-4c59-84c0-4c52ca82aac6-registry-tls\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Sep 17 08:01:06.726080 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:06.726028 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b89d5a79-dac1-47a6-a3fd-8e617e696696-feast-data" (OuterVolumeSpecName: "feast-data") pod "b89d5a79-dac1-47a6-a3fd-8e617e696696" (UID: "b89d5a79-dac1-47a6-a3fd-8e617e696696"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 08:01:06.727408 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:06.727380 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b89d5a79-dac1-47a6-a3fd-8e617e696696-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "b89d5a79-dac1-47a6-a3fd-8e617e696696" (UID: "b89d5a79-dac1-47a6-a3fd-8e617e696696"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 08:01:06.727888 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:06.727845 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b89d5a79-dac1-47a6-a3fd-8e617e696696-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "b89d5a79-dac1-47a6-a3fd-8e617e696696" (UID: "b89d5a79-dac1-47a6-a3fd-8e617e696696"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 08:01:06.727888 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:06.727878 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b89d5a79-dac1-47a6-a3fd-8e617e696696-kube-api-access-sqptl" (OuterVolumeSpecName: "kube-api-access-sqptl") pod "b89d5a79-dac1-47a6-a3fd-8e617e696696" (UID: "b89d5a79-dac1-47a6-a3fd-8e617e696696"). InnerVolumeSpecName "kube-api-access-sqptl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 08:01:06.728010 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:06.727902 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b89d5a79-dac1-47a6-a3fd-8e617e696696-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "b89d5a79-dac1-47a6-a3fd-8e617e696696" (UID: "b89d5a79-dac1-47a6-a3fd-8e617e696696"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 08:01:06.728010 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:06.727912 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b89d5a79-dac1-47a6-a3fd-8e617e696696-online-tls" (OuterVolumeSpecName: "online-tls") pod "b89d5a79-dac1-47a6-a3fd-8e617e696696" (UID: "b89d5a79-dac1-47a6-a3fd-8e617e696696"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 08:01:06.826559 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:06.826525 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqptl\" (UniqueName: \"kubernetes.io/projected/b89d5a79-dac1-47a6-a3fd-8e617e696696-kube-api-access-sqptl\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Sep 17 08:01:06.826559 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:06.826553 2578 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/b89d5a79-dac1-47a6-a3fd-8e617e696696-feast-data\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Sep 17 08:01:06.826559 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:06.826563 2578 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/b89d5a79-dac1-47a6-a3fd-8e617e696696-offline-tls\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Sep 17 08:01:06.826751 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:06.826570 2578 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/b89d5a79-dac1-47a6-a3fd-8e617e696696-registry-tls\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Sep 17 08:01:06.826751 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:06.826578 2578 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/b89d5a79-dac1-47a6-a3fd-8e617e696696-online-tls\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Sep 17 08:01:06.826751 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:06.826586 2578 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/b89d5a79-dac1-47a6-a3fd-8e617e696696-ui-tls\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Sep 17 08:01:07.178314 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:07.178284 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-jwrn8" Sep 17 08:01:07.178750 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:07.178289 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-jwrn8" event={"ID":"2b99dbdb-c43b-4c59-84c0-4c52ca82aac6","Type":"ContainerDied","Data":"e3ef56b61e19f4cbae8aff0976d50561bb501c5b85935251cc2e70ebdb55570d"} Sep 17 08:01:07.178750 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:07.178408 2578 scope.go:117] "RemoveContainer" containerID="707ee5e85c740616faa429b00248d72875e79dfe5292a144d416ed9b52193004" Sep 17 08:01:07.181685 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:07.181665 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6xfzx" event={"ID":"b89d5a79-dac1-47a6-a3fd-8e617e696696","Type":"ContainerDied","Data":"98b9014f08c5de6a9f15f0eb7d6b08f5af67d43f250745eefa3ae2c21c62ef27"} Sep 17 08:01:07.181790 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:07.181726 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6xfzx" Sep 17 08:01:07.187082 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:07.187062 2578 scope.go:117] "RemoveContainer" containerID="2eb11ef299a5d8091abe281c71de6e1afb8248dd4ed6886702057d577724eebe" Sep 17 08:01:07.194792 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:07.194774 2578 scope.go:117] "RemoveContainer" containerID="41bf02fa750ee6fda225262289e56606f30eadcaa4cbc4d2ef11acc2492e3caa" Sep 17 08:01:07.201941 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:07.201921 2578 scope.go:117] "RemoveContainer" containerID="9d285cfb0c04b0f8cf49996089fce985fe255671689f9b796b59b8b6998cf401" Sep 17 08:01:07.209412 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:07.209388 2578 scope.go:117] "RemoveContainer" containerID="ded5117994624907a5a072b9ff15dbf1903892aa663b4c8805a3ff805eebef2d" Sep 17 08:01:07.210380 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:07.210350 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-jwrn8"] Sep 17 08:01:07.216733 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:07.216716 2578 scope.go:117] "RemoveContainer" containerID="c8819636a695f2337d933a3916b8a131ea930bc7e6a1afec72915390fce39a35" Sep 17 08:01:07.217087 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:07.217068 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-jwrn8"] Sep 17 08:01:07.223548 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:07.223530 2578 scope.go:117] "RemoveContainer" containerID="ea36862fdd1829e1a4102aac61d75a8b7607730f5fc05df73e034e1f6fc32175" Sep 17 08:01:07.230489 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:07.230473 2578 scope.go:117] "RemoveContainer" containerID="a37628d02529b2d0a21b6edca29f626ebfac833b4c167142e59bcd65e1a449ae" Sep 17 08:01:07.238063 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:07.237471 2578 scope.go:117] "RemoveContainer" containerID="963de4287adb54e5aad457deb28b86dd1fac237a00bd5f7de2219403a770cd6a" Sep 17 08:01:07.239480 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:07.239462 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-6xfzx"] Sep 17 08:01:07.244984 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:07.244965 2578 scope.go:117] "RemoveContainer" containerID="537d71f8d27be643688a2a4358d90310182821fa2cd50782d865e77a00e1281b" Sep 17 08:01:07.249772 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:07.249753 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-6xfzx"] Sep 17 08:01:07.252209 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:07.252195 2578 scope.go:117] "RemoveContainer" containerID="21cb90de4f773a763f8a3b0143d68eaf0f054dd12431932a9bb5e51fde8d041b" Sep 17 08:01:08.628466 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:08.628425 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b99dbdb-c43b-4c59-84c0-4c52ca82aac6" path="/var/lib/kubelet/pods/2b99dbdb-c43b-4c59-84c0-4c52ca82aac6/volumes" Sep 17 08:01:08.628991 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:08.628976 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b89d5a79-dac1-47a6-a3fd-8e617e696696" path="/var/lib/kubelet/pods/b89d5a79-dac1-47a6-a3fd-8e617e696696/volumes" Sep 17 08:01:25.708494 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:25.708457 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-cvr9b/must-gather-qt9qx"] Sep 17 08:01:25.708947 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:25.708763 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b89d5a79-dac1-47a6-a3fd-8e617e696696" containerName="ui" Sep 17 08:01:25.708947 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:25.708774 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b89d5a79-dac1-47a6-a3fd-8e617e696696" containerName="ui" Sep 17 08:01:25.708947 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:25.708784 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b99dbdb-c43b-4c59-84c0-4c52ca82aac6" containerName="ui" Sep 17 08:01:25.708947 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:25.708790 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b99dbdb-c43b-4c59-84c0-4c52ca82aac6" containerName="ui" Sep 17 08:01:25.708947 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:25.708804 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b89d5a79-dac1-47a6-a3fd-8e617e696696" containerName="feast-apply" Sep 17 08:01:25.708947 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:25.708812 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b89d5a79-dac1-47a6-a3fd-8e617e696696" containerName="feast-apply" Sep 17 08:01:25.708947 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:25.708821 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b89d5a79-dac1-47a6-a3fd-8e617e696696" containerName="offline" Sep 17 08:01:25.708947 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:25.708826 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b89d5a79-dac1-47a6-a3fd-8e617e696696" containerName="offline" Sep 17 08:01:25.708947 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:25.708834 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b99dbdb-c43b-4c59-84c0-4c52ca82aac6" containerName="feast-init" Sep 17 08:01:25.708947 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:25.708840 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b99dbdb-c43b-4c59-84c0-4c52ca82aac6" containerName="feast-init" Sep 17 08:01:25.708947 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:25.708849 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b89d5a79-dac1-47a6-a3fd-8e617e696696" containerName="feast-init" Sep 17 08:01:25.708947 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:25.708854 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b89d5a79-dac1-47a6-a3fd-8e617e696696" containerName="feast-init" Sep 17 08:01:25.708947 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:25.708879 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b99dbdb-c43b-4c59-84c0-4c52ca82aac6" containerName="feast-apply" Sep 17 08:01:25.708947 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:25.708887 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b99dbdb-c43b-4c59-84c0-4c52ca82aac6" containerName="feast-apply" Sep 17 08:01:25.708947 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:25.708894 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b99dbdb-c43b-4c59-84c0-4c52ca82aac6" containerName="online" Sep 17 08:01:25.708947 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:25.708899 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b99dbdb-c43b-4c59-84c0-4c52ca82aac6" containerName="online" Sep 17 08:01:25.708947 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:25.708904 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b99dbdb-c43b-4c59-84c0-4c52ca82aac6" containerName="offline" Sep 17 08:01:25.708947 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:25.708909 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b99dbdb-c43b-4c59-84c0-4c52ca82aac6" containerName="offline" Sep 17 08:01:25.708947 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:25.708917 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b89d5a79-dac1-47a6-a3fd-8e617e696696" containerName="registry" Sep 17 08:01:25.708947 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:25.708922 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b89d5a79-dac1-47a6-a3fd-8e617e696696" containerName="registry" Sep 17 08:01:25.708947 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:25.708930 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b89d5a79-dac1-47a6-a3fd-8e617e696696" containerName="online" Sep 17 08:01:25.708947 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:25.708934 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b89d5a79-dac1-47a6-a3fd-8e617e696696" containerName="online" Sep 17 08:01:25.709604 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:25.708987 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b89d5a79-dac1-47a6-a3fd-8e617e696696" containerName="offline" Sep 17 08:01:25.709604 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:25.708995 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b89d5a79-dac1-47a6-a3fd-8e617e696696" containerName="online" Sep 17 08:01:25.709604 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:25.709001 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b89d5a79-dac1-47a6-a3fd-8e617e696696" containerName="registry" Sep 17 08:01:25.709604 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:25.709007 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b99dbdb-c43b-4c59-84c0-4c52ca82aac6" containerName="offline" Sep 17 08:01:25.709604 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:25.709013 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b99dbdb-c43b-4c59-84c0-4c52ca82aac6" containerName="ui" Sep 17 08:01:25.709604 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:25.709019 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b89d5a79-dac1-47a6-a3fd-8e617e696696" containerName="ui" Sep 17 08:01:25.709604 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:25.709025 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b99dbdb-c43b-4c59-84c0-4c52ca82aac6" containerName="online" Sep 17 08:01:25.711954 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:25.711933 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cvr9b/must-gather-qt9qx" Sep 17 08:01:25.715108 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:25.715080 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-cvr9b\"/\"kube-root-ca.crt\"" Sep 17 08:01:25.715108 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:25.715100 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-cvr9b\"/\"openshift-service-ca.crt\"" Sep 17 08:01:25.715108 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:25.715083 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-cvr9b\"/\"default-dockercfg-njwz9\"" Sep 17 08:01:25.724916 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:25.724896 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cvr9b/must-gather-qt9qx"] Sep 17 08:01:25.871288 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:25.871253 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/1dac8fb4-6b34-42b1-bc86-beac29ad3a41-must-gather-output\") pod \"must-gather-qt9qx\" (UID: \"1dac8fb4-6b34-42b1-bc86-beac29ad3a41\") " pod="openshift-must-gather-cvr9b/must-gather-qt9qx" Sep 17 08:01:25.871443 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:25.871301 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbh2f\" (UniqueName: \"kubernetes.io/projected/1dac8fb4-6b34-42b1-bc86-beac29ad3a41-kube-api-access-lbh2f\") pod \"must-gather-qt9qx\" (UID: \"1dac8fb4-6b34-42b1-bc86-beac29ad3a41\") " pod="openshift-must-gather-cvr9b/must-gather-qt9qx" Sep 17 08:01:25.972028 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:25.971963 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbh2f\" (UniqueName: \"kubernetes.io/projected/1dac8fb4-6b34-42b1-bc86-beac29ad3a41-kube-api-access-lbh2f\") pod \"must-gather-qt9qx\" (UID: \"1dac8fb4-6b34-42b1-bc86-beac29ad3a41\") " pod="openshift-must-gather-cvr9b/must-gather-qt9qx" Sep 17 08:01:25.972146 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:25.972025 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/1dac8fb4-6b34-42b1-bc86-beac29ad3a41-must-gather-output\") pod \"must-gather-qt9qx\" (UID: \"1dac8fb4-6b34-42b1-bc86-beac29ad3a41\") " pod="openshift-must-gather-cvr9b/must-gather-qt9qx" Sep 17 08:01:25.972290 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:25.972268 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/1dac8fb4-6b34-42b1-bc86-beac29ad3a41-must-gather-output\") pod \"must-gather-qt9qx\" (UID: \"1dac8fb4-6b34-42b1-bc86-beac29ad3a41\") " pod="openshift-must-gather-cvr9b/must-gather-qt9qx" Sep 17 08:01:25.981190 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:25.981161 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbh2f\" (UniqueName: \"kubernetes.io/projected/1dac8fb4-6b34-42b1-bc86-beac29ad3a41-kube-api-access-lbh2f\") pod \"must-gather-qt9qx\" (UID: \"1dac8fb4-6b34-42b1-bc86-beac29ad3a41\") " pod="openshift-must-gather-cvr9b/must-gather-qt9qx" Sep 17 08:01:26.040185 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:26.040150 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cvr9b/must-gather-qt9qx" Sep 17 08:01:26.309164 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:26.308940 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cvr9b/must-gather-qt9qx"] Sep 17 08:01:26.311913 ip-10-0-139-100 kubenswrapper[2578]: W0917 08:01:26.311877 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1dac8fb4_6b34_42b1_bc86_beac29ad3a41.slice/crio-d2fcce88f6c2d3ff79a9b82fb457dfcfb38162ad357a3c8b8b20bc6b3f2fa142 WatchSource:0}: Error finding container d2fcce88f6c2d3ff79a9b82fb457dfcfb38162ad357a3c8b8b20bc6b3f2fa142: Status 404 returned error can't find the container with id d2fcce88f6c2d3ff79a9b82fb457dfcfb38162ad357a3c8b8b20bc6b3f2fa142 Sep 17 08:01:26.313873 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:26.313839 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 17 08:01:27.262293 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:27.262257 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cvr9b/must-gather-qt9qx" event={"ID":"1dac8fb4-6b34-42b1-bc86-beac29ad3a41","Type":"ContainerStarted","Data":"d2fcce88f6c2d3ff79a9b82fb457dfcfb38162ad357a3c8b8b20bc6b3f2fa142"} Sep 17 08:01:33.289390 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:33.289344 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cvr9b/must-gather-qt9qx" event={"ID":"1dac8fb4-6b34-42b1-bc86-beac29ad3a41","Type":"ContainerStarted","Data":"0ac28d7cbd163d2fc9e4adf9a989ef0c207302002c4baef74190cb1645eaa5e3"} Sep 17 08:01:33.289769 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:33.289404 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cvr9b/must-gather-qt9qx" event={"ID":"1dac8fb4-6b34-42b1-bc86-beac29ad3a41","Type":"ContainerStarted","Data":"a07f4763c52239b0b54aa7c42c495a8da94d93bff94f2c3e11de302c181402ca"} Sep 17 08:01:42.332720 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:42.332627 2578 generic.go:356] "Generic (PLEG): container finished" podID="1dac8fb4-6b34-42b1-bc86-beac29ad3a41" containerID="a07f4763c52239b0b54aa7c42c495a8da94d93bff94f2c3e11de302c181402ca" exitCode=0 Sep 17 08:01:42.333257 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:42.332708 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cvr9b/must-gather-qt9qx" event={"ID":"1dac8fb4-6b34-42b1-bc86-beac29ad3a41","Type":"ContainerDied","Data":"a07f4763c52239b0b54aa7c42c495a8da94d93bff94f2c3e11de302c181402ca"} Sep 17 08:01:42.333257 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:42.333097 2578 scope.go:117] "RemoveContainer" containerID="a07f4763c52239b0b54aa7c42c495a8da94d93bff94f2c3e11de302c181402ca" Sep 17 08:01:42.914515 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:42.914485 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-cvr9b_must-gather-qt9qx_1dac8fb4-6b34-42b1-bc86-beac29ad3a41/gather/0.log" Sep 17 08:01:46.555675 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:46.555641 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-lqcw7_a236d1bd-81b1-4c9e-a14f-997204ad1b98/global-pull-secret-syncer/0.log" Sep 17 08:01:46.831239 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:46.831167 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-qbfvv_bfe3853f-127c-4124-8df0-cdd6439052bb/konnectivity-agent/0.log" Sep 17 08:01:46.940305 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:46.940270 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-100.ec2.internal_9f09f680a0274abd6edf499bc1ec7a29/haproxy/0.log" Sep 17 08:01:48.287318 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:48.287281 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-cvr9b/must-gather-qt9qx"] Sep 17 08:01:48.287726 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:48.287475 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-cvr9b/must-gather-qt9qx" podUID="1dac8fb4-6b34-42b1-bc86-beac29ad3a41" containerName="copy" containerID="cri-o://0ac28d7cbd163d2fc9e4adf9a989ef0c207302002c4baef74190cb1645eaa5e3" gracePeriod=2 Sep 17 08:01:48.294152 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:48.294121 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-cvr9b/must-gather-qt9qx"] Sep 17 08:01:48.518952 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:48.518930 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-cvr9b_must-gather-qt9qx_1dac8fb4-6b34-42b1-bc86-beac29ad3a41/copy/0.log" Sep 17 08:01:48.520545 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:48.520528 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cvr9b/must-gather-qt9qx" Sep 17 08:01:48.522770 ip-10-0-139-100 kubenswrapper[2578]: E0917 08:01:48.522741 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-qt9qx\" is forbidden: User \"system:node:ip-10-0-139-100.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-cvr9b\": no relationship found between node 'ip-10-0-139-100.ec2.internal' and this object" podUID="1dac8fb4-6b34-42b1-bc86-beac29ad3a41" pod="openshift-must-gather-cvr9b/must-gather-qt9qx" Sep 17 08:01:48.652625 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:48.652605 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/1dac8fb4-6b34-42b1-bc86-beac29ad3a41-must-gather-output\") pod \"1dac8fb4-6b34-42b1-bc86-beac29ad3a41\" (UID: \"1dac8fb4-6b34-42b1-bc86-beac29ad3a41\") " Sep 17 08:01:48.652746 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:48.652643 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lbh2f\" (UniqueName: \"kubernetes.io/projected/1dac8fb4-6b34-42b1-bc86-beac29ad3a41-kube-api-access-lbh2f\") pod \"1dac8fb4-6b34-42b1-bc86-beac29ad3a41\" (UID: \"1dac8fb4-6b34-42b1-bc86-beac29ad3a41\") " Sep 17 08:01:48.652932 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:48.652910 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dac8fb4-6b34-42b1-bc86-beac29ad3a41-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "1dac8fb4-6b34-42b1-bc86-beac29ad3a41" (UID: "1dac8fb4-6b34-42b1-bc86-beac29ad3a41"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 08:01:48.654669 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:48.654646 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dac8fb4-6b34-42b1-bc86-beac29ad3a41-kube-api-access-lbh2f" (OuterVolumeSpecName: "kube-api-access-lbh2f") pod "1dac8fb4-6b34-42b1-bc86-beac29ad3a41" (UID: "1dac8fb4-6b34-42b1-bc86-beac29ad3a41"). InnerVolumeSpecName "kube-api-access-lbh2f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 08:01:48.753938 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:48.753914 2578 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/1dac8fb4-6b34-42b1-bc86-beac29ad3a41-must-gather-output\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Sep 17 08:01:48.753938 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:48.753936 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbh2f\" (UniqueName: \"kubernetes.io/projected/1dac8fb4-6b34-42b1-bc86-beac29ad3a41-kube-api-access-lbh2f\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Sep 17 08:01:49.361159 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:49.361132 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-cvr9b_must-gather-qt9qx_1dac8fb4-6b34-42b1-bc86-beac29ad3a41/copy/0.log" Sep 17 08:01:49.361560 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:49.361507 2578 generic.go:356] "Generic (PLEG): container finished" podID="1dac8fb4-6b34-42b1-bc86-beac29ad3a41" containerID="0ac28d7cbd163d2fc9e4adf9a989ef0c207302002c4baef74190cb1645eaa5e3" exitCode=143 Sep 17 08:01:49.361610 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:49.361564 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cvr9b/must-gather-qt9qx" Sep 17 08:01:49.361610 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:49.361595 2578 scope.go:117] "RemoveContainer" containerID="0ac28d7cbd163d2fc9e4adf9a989ef0c207302002c4baef74190cb1645eaa5e3" Sep 17 08:01:49.376361 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:49.376249 2578 scope.go:117] "RemoveContainer" containerID="a07f4763c52239b0b54aa7c42c495a8da94d93bff94f2c3e11de302c181402ca" Sep 17 08:01:49.387531 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:49.387510 2578 scope.go:117] "RemoveContainer" containerID="0ac28d7cbd163d2fc9e4adf9a989ef0c207302002c4baef74190cb1645eaa5e3" Sep 17 08:01:49.387810 ip-10-0-139-100 kubenswrapper[2578]: E0917 08:01:49.387791 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ac28d7cbd163d2fc9e4adf9a989ef0c207302002c4baef74190cb1645eaa5e3\": container with ID starting with 0ac28d7cbd163d2fc9e4adf9a989ef0c207302002c4baef74190cb1645eaa5e3 not found: ID does not exist" containerID="0ac28d7cbd163d2fc9e4adf9a989ef0c207302002c4baef74190cb1645eaa5e3" Sep 17 08:01:49.387892 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:49.387817 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ac28d7cbd163d2fc9e4adf9a989ef0c207302002c4baef74190cb1645eaa5e3"} err="failed to get container status \"0ac28d7cbd163d2fc9e4adf9a989ef0c207302002c4baef74190cb1645eaa5e3\": rpc error: code = NotFound desc = could not find container \"0ac28d7cbd163d2fc9e4adf9a989ef0c207302002c4baef74190cb1645eaa5e3\": container with ID starting with 0ac28d7cbd163d2fc9e4adf9a989ef0c207302002c4baef74190cb1645eaa5e3 not found: ID does not exist" Sep 17 08:01:49.387892 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:49.387838 2578 scope.go:117] "RemoveContainer" containerID="a07f4763c52239b0b54aa7c42c495a8da94d93bff94f2c3e11de302c181402ca" Sep 17 08:01:49.388102 ip-10-0-139-100 kubenswrapper[2578]: E0917 08:01:49.388084 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a07f4763c52239b0b54aa7c42c495a8da94d93bff94f2c3e11de302c181402ca\": container with ID starting with a07f4763c52239b0b54aa7c42c495a8da94d93bff94f2c3e11de302c181402ca not found: ID does not exist" containerID="a07f4763c52239b0b54aa7c42c495a8da94d93bff94f2c3e11de302c181402ca" Sep 17 08:01:49.388158 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:49.388108 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a07f4763c52239b0b54aa7c42c495a8da94d93bff94f2c3e11de302c181402ca"} err="failed to get container status \"a07f4763c52239b0b54aa7c42c495a8da94d93bff94f2c3e11de302c181402ca\": rpc error: code = NotFound desc = could not find container \"a07f4763c52239b0b54aa7c42c495a8da94d93bff94f2c3e11de302c181402ca\": container with ID starting with a07f4763c52239b0b54aa7c42c495a8da94d93bff94f2c3e11de302c181402ca not found: ID does not exist" Sep 17 08:01:50.629059 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:50.628982 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1dac8fb4-6b34-42b1-bc86-beac29ad3a41" path="/var/lib/kubelet/pods/1dac8fb4-6b34-42b1-bc86-beac29ad3a41/volumes" Sep 17 08:01:50.899122 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:50.899099 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_55b5d3f9-e42c-40be-a9a4-3cc3c81ad7e2/alertmanager/0.log" Sep 17 08:01:50.975908 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:50.975881 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_55b5d3f9-e42c-40be-a9a4-3cc3c81ad7e2/config-reloader/0.log" Sep 17 08:01:51.042145 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:51.042121 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_55b5d3f9-e42c-40be-a9a4-3cc3c81ad7e2/kube-rbac-proxy-web/0.log" Sep 17 08:01:51.083514 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:51.083493 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_55b5d3f9-e42c-40be-a9a4-3cc3c81ad7e2/kube-rbac-proxy/0.log" Sep 17 08:01:51.151101 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:51.151042 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_55b5d3f9-e42c-40be-a9a4-3cc3c81ad7e2/kube-rbac-proxy-metric/0.log" Sep 17 08:01:51.208271 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:51.208242 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_55b5d3f9-e42c-40be-a9a4-3cc3c81ad7e2/prom-label-proxy/0.log" Sep 17 08:01:51.246906 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:51.246887 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_55b5d3f9-e42c-40be-a9a4-3cc3c81ad7e2/init-config-reloader/0.log" Sep 17 08:01:51.425191 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:51.425119 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-8pn4t_b6f2509c-cdcc-4938-a577-f9d7d409c95d/kube-state-metrics/0.log" Sep 17 08:01:51.481034 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:51.480999 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-8pn4t_b6f2509c-cdcc-4938-a577-f9d7d409c95d/kube-rbac-proxy-main/0.log" Sep 17 08:01:51.521223 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:51.521199 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-8pn4t_b6f2509c-cdcc-4938-a577-f9d7d409c95d/kube-rbac-proxy-self/0.log" Sep 17 08:01:51.604337 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:51.604314 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-665f5f65f8-sdxtt_38179dd9-b912-4fb0-9dd0-f4c57f4c5ac1/metrics-server/0.log" Sep 17 08:01:51.708695 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:51.708630 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fczng_cc77ee52-e75e-4485-8bfb-40dbfcaf83e9/node-exporter/0.log" Sep 17 08:01:51.758596 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:51.758575 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fczng_cc77ee52-e75e-4485-8bfb-40dbfcaf83e9/kube-rbac-proxy/0.log" Sep 17 08:01:51.814411 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:51.814390 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fczng_cc77ee52-e75e-4485-8bfb-40dbfcaf83e9/init-textfile/0.log" Sep 17 08:01:52.095619 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:52.095546 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-qd64r_8dadfa8e-9e62-4a67-8aeb-8eb92e2410ba/kube-rbac-proxy-main/0.log" Sep 17 08:01:52.135334 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:52.135309 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-qd64r_8dadfa8e-9e62-4a67-8aeb-8eb92e2410ba/kube-rbac-proxy-self/0.log" Sep 17 08:01:52.162916 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:52.162895 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-qd64r_8dadfa8e-9e62-4a67-8aeb-8eb92e2410ba/openshift-state-metrics/0.log" Sep 17 08:01:52.423199 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:52.423170 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-m4894_a7d2fc30-22e9-4052-82f1-83ec99c2e632/prometheus-operator/0.log" Sep 17 08:01:52.448802 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:52.448776 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-m4894_a7d2fc30-22e9-4052-82f1-83ec99c2e632/kube-rbac-proxy/0.log" Sep 17 08:01:52.481142 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:52.481121 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5864cd99d7-9d2qq_a63c0215-fb21-4077-bc08-1f227674a5b7/prometheus-operator-admission-webhook/0.log" Sep 17 08:01:52.525029 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:52.525005 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-68dddff84-wggvb_c6869870-c751-48a4-8f01-bc70ae552eb4/telemeter-client/0.log" Sep 17 08:01:52.554011 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:52.553995 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-68dddff84-wggvb_c6869870-c751-48a4-8f01-bc70ae552eb4/reload/0.log" Sep 17 08:01:52.595191 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:52.595168 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-68dddff84-wggvb_c6869870-c751-48a4-8f01-bc70ae552eb4/kube-rbac-proxy/0.log" Sep 17 08:01:53.584902 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:53.584804 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-7n9ch/perf-node-gather-daemonset-n9fhx"] Sep 17 08:01:53.585342 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:53.585273 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1dac8fb4-6b34-42b1-bc86-beac29ad3a41" containerName="copy" Sep 17 08:01:53.585342 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:53.585292 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1dac8fb4-6b34-42b1-bc86-beac29ad3a41" containerName="copy" Sep 17 08:01:53.585342 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:53.585316 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1dac8fb4-6b34-42b1-bc86-beac29ad3a41" containerName="gather" Sep 17 08:01:53.585342 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:53.585324 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1dac8fb4-6b34-42b1-bc86-beac29ad3a41" containerName="gather" Sep 17 08:01:53.585557 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:53.585394 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="1dac8fb4-6b34-42b1-bc86-beac29ad3a41" containerName="copy" Sep 17 08:01:53.585557 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:53.585409 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="1dac8fb4-6b34-42b1-bc86-beac29ad3a41" containerName="gather" Sep 17 08:01:53.589406 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:53.589383 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7n9ch/perf-node-gather-daemonset-n9fhx" Sep 17 08:01:53.593000 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:53.592979 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-7n9ch\"/\"openshift-service-ca.crt\"" Sep 17 08:01:53.593721 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:53.593706 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-7n9ch\"/\"default-dockercfg-qsmmh\"" Sep 17 08:01:53.593786 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:53.593761 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-7n9ch\"/\"kube-root-ca.crt\"" Sep 17 08:01:53.601259 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:53.601238 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7n9ch/perf-node-gather-daemonset-n9fhx"] Sep 17 08:01:53.689344 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:53.689318 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/13491b6a-bf3b-4b21-8456-279d537003b6-podres\") pod \"perf-node-gather-daemonset-n9fhx\" (UID: \"13491b6a-bf3b-4b21-8456-279d537003b6\") " pod="openshift-must-gather-7n9ch/perf-node-gather-daemonset-n9fhx" Sep 17 08:01:53.689461 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:53.689351 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/13491b6a-bf3b-4b21-8456-279d537003b6-sys\") pod \"perf-node-gather-daemonset-n9fhx\" (UID: \"13491b6a-bf3b-4b21-8456-279d537003b6\") " pod="openshift-must-gather-7n9ch/perf-node-gather-daemonset-n9fhx" Sep 17 08:01:53.689461 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:53.689373 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/13491b6a-bf3b-4b21-8456-279d537003b6-lib-modules\") pod \"perf-node-gather-daemonset-n9fhx\" (UID: \"13491b6a-bf3b-4b21-8456-279d537003b6\") " pod="openshift-must-gather-7n9ch/perf-node-gather-daemonset-n9fhx" Sep 17 08:01:53.689461 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:53.689396 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52zzn\" (UniqueName: \"kubernetes.io/projected/13491b6a-bf3b-4b21-8456-279d537003b6-kube-api-access-52zzn\") pod \"perf-node-gather-daemonset-n9fhx\" (UID: \"13491b6a-bf3b-4b21-8456-279d537003b6\") " pod="openshift-must-gather-7n9ch/perf-node-gather-daemonset-n9fhx" Sep 17 08:01:53.689576 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:53.689472 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/13491b6a-bf3b-4b21-8456-279d537003b6-proc\") pod \"perf-node-gather-daemonset-n9fhx\" (UID: \"13491b6a-bf3b-4b21-8456-279d537003b6\") " pod="openshift-must-gather-7n9ch/perf-node-gather-daemonset-n9fhx" Sep 17 08:01:53.790774 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:53.790748 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/13491b6a-bf3b-4b21-8456-279d537003b6-podres\") pod \"perf-node-gather-daemonset-n9fhx\" (UID: \"13491b6a-bf3b-4b21-8456-279d537003b6\") " pod="openshift-must-gather-7n9ch/perf-node-gather-daemonset-n9fhx" Sep 17 08:01:53.790906 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:53.790794 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/13491b6a-bf3b-4b21-8456-279d537003b6-sys\") pod \"perf-node-gather-daemonset-n9fhx\" (UID: \"13491b6a-bf3b-4b21-8456-279d537003b6\") " pod="openshift-must-gather-7n9ch/perf-node-gather-daemonset-n9fhx" Sep 17 08:01:53.790906 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:53.790813 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/13491b6a-bf3b-4b21-8456-279d537003b6-lib-modules\") pod \"perf-node-gather-daemonset-n9fhx\" (UID: \"13491b6a-bf3b-4b21-8456-279d537003b6\") " pod="openshift-must-gather-7n9ch/perf-node-gather-daemonset-n9fhx" Sep 17 08:01:53.790906 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:53.790820 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/13491b6a-bf3b-4b21-8456-279d537003b6-podres\") pod \"perf-node-gather-daemonset-n9fhx\" (UID: \"13491b6a-bf3b-4b21-8456-279d537003b6\") " pod="openshift-must-gather-7n9ch/perf-node-gather-daemonset-n9fhx" Sep 17 08:01:53.790906 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:53.790829 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52zzn\" (UniqueName: \"kubernetes.io/projected/13491b6a-bf3b-4b21-8456-279d537003b6-kube-api-access-52zzn\") pod \"perf-node-gather-daemonset-n9fhx\" (UID: \"13491b6a-bf3b-4b21-8456-279d537003b6\") " pod="openshift-must-gather-7n9ch/perf-node-gather-daemonset-n9fhx" Sep 17 08:01:53.790906 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:53.790898 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/13491b6a-bf3b-4b21-8456-279d537003b6-proc\") pod \"perf-node-gather-daemonset-n9fhx\" (UID: \"13491b6a-bf3b-4b21-8456-279d537003b6\") " pod="openshift-must-gather-7n9ch/perf-node-gather-daemonset-n9fhx" Sep 17 08:01:53.791098 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:53.790936 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/13491b6a-bf3b-4b21-8456-279d537003b6-sys\") pod \"perf-node-gather-daemonset-n9fhx\" (UID: \"13491b6a-bf3b-4b21-8456-279d537003b6\") " pod="openshift-must-gather-7n9ch/perf-node-gather-daemonset-n9fhx" Sep 17 08:01:53.791098 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:53.790950 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/13491b6a-bf3b-4b21-8456-279d537003b6-proc\") pod \"perf-node-gather-daemonset-n9fhx\" (UID: \"13491b6a-bf3b-4b21-8456-279d537003b6\") " pod="openshift-must-gather-7n9ch/perf-node-gather-daemonset-n9fhx" Sep 17 08:01:53.791098 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:53.790985 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/13491b6a-bf3b-4b21-8456-279d537003b6-lib-modules\") pod \"perf-node-gather-daemonset-n9fhx\" (UID: \"13491b6a-bf3b-4b21-8456-279d537003b6\") " pod="openshift-must-gather-7n9ch/perf-node-gather-daemonset-n9fhx" Sep 17 08:01:53.800610 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:53.800588 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-52zzn\" (UniqueName: \"kubernetes.io/projected/13491b6a-bf3b-4b21-8456-279d537003b6-kube-api-access-52zzn\") pod \"perf-node-gather-daemonset-n9fhx\" (UID: \"13491b6a-bf3b-4b21-8456-279d537003b6\") " pod="openshift-must-gather-7n9ch/perf-node-gather-daemonset-n9fhx" Sep 17 08:01:53.843776 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:53.843717 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-5dcbc7df6b-8hwpw_0a5a4190-dc08-4f8e-ab3c-af69dea0d11f/networking-console-plugin/0.log" Sep 17 08:01:53.899460 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:53.899441 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7n9ch/perf-node-gather-daemonset-n9fhx" Sep 17 08:01:54.180688 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:54.180644 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7n9ch/perf-node-gather-daemonset-n9fhx"] Sep 17 08:01:54.182968 ip-10-0-139-100 kubenswrapper[2578]: W0917 08:01:54.182939 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod13491b6a_bf3b_4b21_8456_279d537003b6.slice/crio-94052206a015a1362772ad4f6cd820dccfd18e08ac371010215b84c2fd6cd841 WatchSource:0}: Error finding container 94052206a015a1362772ad4f6cd820dccfd18e08ac371010215b84c2fd6cd841: Status 404 returned error can't find the container with id 94052206a015a1362772ad4f6cd820dccfd18e08ac371010215b84c2fd6cd841 Sep 17 08:01:54.383234 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:54.383196 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7n9ch/perf-node-gather-daemonset-n9fhx" event={"ID":"13491b6a-bf3b-4b21-8456-279d537003b6","Type":"ContainerStarted","Data":"0ddd5dded9e708175301d87ca5c8f3e7b155fa81679807670471cec4a81eddb6"} Sep 17 08:01:54.383364 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:54.383257 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7n9ch/perf-node-gather-daemonset-n9fhx" event={"ID":"13491b6a-bf3b-4b21-8456-279d537003b6","Type":"ContainerStarted","Data":"94052206a015a1362772ad4f6cd820dccfd18e08ac371010215b84c2fd6cd841"} Sep 17 08:01:54.383364 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:54.383309 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-7n9ch/perf-node-gather-daemonset-n9fhx" Sep 17 08:01:54.406805 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:54.406763 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-7n9ch/perf-node-gather-daemonset-n9fhx" podStartSLOduration=1.406750799 podStartE2EDuration="1.406750799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 08:01:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 08:01:54.405328434 +0000 UTC m=+2058.286362794" watchObservedRunningTime="2026-09-17 08:01:54.406750799 +0000 UTC m=+2058.287785170" Sep 17 08:01:54.818515 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:54.818486 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-76f99584f4-z7dwl_ef341fec-b3aa-45f8-9697-70bbf084fc7f/download-server/0.log" Sep 17 08:01:56.090717 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:56.090687 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-nfjvh_2fcfd678-3207-4e47-a6d4-3d1416568d1c/dns/0.log" Sep 17 08:01:56.116595 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:56.116563 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-nfjvh_2fcfd678-3207-4e47-a6d4-3d1416568d1c/kube-rbac-proxy/0.log" Sep 17 08:01:56.242281 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:56.242246 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xfggl_d9df1c13-2108-44d8-9137-730e35cf2165/dns-node-resolver/0.log" Sep 17 08:01:56.775319 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:56.775290 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-fwdd5_f50cb832-b497-4a6a-bbe6-b758ec939688/node-ca/0.log" Sep 17 08:01:57.723098 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:57.723069 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-764969ff56-mgc25_67157086-7b61-49c2-8862-e594a9881821/router/0.log" Sep 17 08:01:58.259383 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:58.259353 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-kcfcf_fdc3a585-47be-40de-bfa4-9503c7e28701/serve-healthcheck-canary/0.log" Sep 17 08:01:58.923662 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:58.923640 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-zkvmd_ed949cd2-5046-4ef7-870c-57671c2843f7/kube-rbac-proxy/0.log" Sep 17 08:01:58.995232 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:58.995170 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-zkvmd_ed949cd2-5046-4ef7-870c-57671c2843f7/exporter/0.log" Sep 17 08:01:59.045649 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:01:59.045626 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-zkvmd_ed949cd2-5046-4ef7-870c-57671c2843f7/extractor/0.log" Sep 17 08:02:00.397021 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:02:00.396996 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-7n9ch/perf-node-gather-daemonset-n9fhx" Sep 17 08:02:05.727749 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:02:05.727717 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lh5z8_6642bc78-77e5-44af-af76-a8f87d25c89a/kube-multus-additional-cni-plugins/0.log" Sep 17 08:02:05.761945 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:02:05.761920 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lh5z8_6642bc78-77e5-44af-af76-a8f87d25c89a/egress-router-binary-copy/0.log" Sep 17 08:02:05.792000 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:02:05.791979 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lh5z8_6642bc78-77e5-44af-af76-a8f87d25c89a/cni-plugins/0.log" Sep 17 08:02:05.818270 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:02:05.818247 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lh5z8_6642bc78-77e5-44af-af76-a8f87d25c89a/bond-cni-plugin/0.log" Sep 17 08:02:05.855006 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:02:05.854988 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lh5z8_6642bc78-77e5-44af-af76-a8f87d25c89a/routeoverride-cni/0.log" Sep 17 08:02:05.880665 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:02:05.880646 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lh5z8_6642bc78-77e5-44af-af76-a8f87d25c89a/whereabouts-cni-bincopy/0.log" Sep 17 08:02:05.908240 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:02:05.908218 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lh5z8_6642bc78-77e5-44af-af76-a8f87d25c89a/whereabouts-cni/0.log" Sep 17 08:02:06.245370 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:02:06.245347 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-zpwkj_60c84ae0-1da1-4845-8d12-d9656a247588/kube-multus/0.log" Sep 17 08:02:06.270562 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:02:06.270536 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-lnpjx_737e0243-b8eb-47e4-bf0f-c6b31e6ba787/network-metrics-daemon/0.log" Sep 17 08:02:06.294111 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:02:06.294091 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-lnpjx_737e0243-b8eb-47e4-bf0f-c6b31e6ba787/kube-rbac-proxy/0.log" Sep 17 08:02:07.716076 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:02:07.716045 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hnzgv_eeb1a9ba-ece4-4e09-8bc7-de92eb38f029/ovn-controller/0.log" Sep 17 08:02:07.742356 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:02:07.742328 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hnzgv_eeb1a9ba-ece4-4e09-8bc7-de92eb38f029/ovn-acl-logging/0.log" Sep 17 08:02:07.772204 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:02:07.772179 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hnzgv_eeb1a9ba-ece4-4e09-8bc7-de92eb38f029/kube-rbac-proxy-node/0.log" Sep 17 08:02:07.801202 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:02:07.801181 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hnzgv_eeb1a9ba-ece4-4e09-8bc7-de92eb38f029/kube-rbac-proxy-ovn-metrics/0.log" Sep 17 08:02:07.827556 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:02:07.827533 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hnzgv_eeb1a9ba-ece4-4e09-8bc7-de92eb38f029/northd/0.log" Sep 17 08:02:07.851538 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:02:07.851511 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hnzgv_eeb1a9ba-ece4-4e09-8bc7-de92eb38f029/nbdb/0.log" Sep 17 08:02:07.874721 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:02:07.874702 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hnzgv_eeb1a9ba-ece4-4e09-8bc7-de92eb38f029/sbdb/0.log" Sep 17 08:02:07.921133 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:02:07.921105 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hnzgv_eeb1a9ba-ece4-4e09-8bc7-de92eb38f029/ovnkube-controller/0.log" Sep 17 08:02:08.941894 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:02:08.941852 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-source-6f8b768455-bjh22_4f97baec-aba5-4b9f-afd8-717a5ad6c84e/check-endpoints/0.log" Sep 17 08:02:08.991040 ip-10-0-139-100 kubenswrapper[2578]: I0917 08:02:08.991015 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-bvxfq_82555a36-653a-4106-a31e-b4da6445b838/network-check-target-container/0.log"