Sep 17 10:08:40.035804 ip-10-0-137-203 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Sep 17 10:08:40.035815 ip-10-0-137-203 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Sep 17 10:08:40.035821 ip-10-0-137-203 systemd[1]: kubelet.service: Failed with result 'resources'. Sep 17 10:08:40.035974 ip-10-0-137-203 systemd[1]: Failed to start Kubernetes Kubelet. Sep 17 10:08:50.043896 ip-10-0-137-203 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Sep 17 10:08:50.043913 ip-10-0-137-203 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot ddca51df781448628b5fd3731affbdfb -- Sep 17 10:10:45.339117 ip-10-0-137-203 systemd[1]: Starting Kubernetes Kubelet... Sep 17 10:10:45.723870 ip-10-0-137-203 kubenswrapper[2581]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 17 10:10:45.723870 ip-10-0-137-203 kubenswrapper[2581]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 17 10:10:45.723870 ip-10-0-137-203 kubenswrapper[2581]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 17 10:10:45.723870 ip-10-0-137-203 kubenswrapper[2581]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 17 10:10:45.723870 ip-10-0-137-203 kubenswrapper[2581]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 17 10:10:45.724768 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.724675 2581 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 17 10:10:45.729317 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.729295 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 17 10:10:45.729317 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.729313 2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 17 10:10:45.729317 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.729318 2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 17 10:10:45.729317 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.729322 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 17 10:10:45.729560 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.729326 2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 17 10:10:45.729560 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.729331 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 17 10:10:45.729560 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.729335 2581 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 17 10:10:45.729560 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.729339 2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 17 10:10:45.729560 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.729343 2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 17 10:10:45.729560 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.729347 2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 17 10:10:45.729560 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.729350 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 17 10:10:45.729560 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.729354 2581 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 17 10:10:45.729560 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.729358 2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 17 10:10:45.729560 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.729362 2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 17 10:10:45.729560 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.729366 2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 17 10:10:45.729560 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.729370 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 17 10:10:45.729560 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.729373 2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 17 10:10:45.729560 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.729387 2581 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 17 10:10:45.729560 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.729391 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 17 10:10:45.729560 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.729395 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 17 10:10:45.729560 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.729399 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 17 10:10:45.729560 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.729403 2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 17 10:10:45.729560 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.729407 2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 17 10:10:45.730369 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.729411 2581 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 17 10:10:45.730369 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.729415 2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 17 10:10:45.730369 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.729419 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 17 10:10:45.730369 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.729423 2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 17 10:10:45.730369 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.729429 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 17 10:10:45.730369 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.729432 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 17 10:10:45.730369 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.729436 2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 17 10:10:45.730369 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.729441 2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 17 10:10:45.730369 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.729445 2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 17 10:10:45.730369 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.729449 2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 17 10:10:45.730369 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.729454 2581 feature_gate.go:328] unrecognized feature gate: Example Sep 17 10:10:45.730369 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.729458 2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 17 10:10:45.730369 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.729463 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 17 10:10:45.730369 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.729467 2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 17 10:10:45.730369 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.729472 2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 17 10:10:45.730369 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.729476 2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 17 10:10:45.730369 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.729479 2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 17 10:10:45.730369 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.729484 2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 17 10:10:45.730369 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.729489 2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 17 10:10:45.730369 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.729493 2581 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 17 10:10:45.731294 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.729523 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 17 10:10:45.731294 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.729531 2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 17 10:10:45.731294 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.729536 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 17 10:10:45.731294 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.729541 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 17 10:10:45.731294 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.729548 2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 17 10:10:45.731294 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.729555 2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 17 10:10:45.731294 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.729560 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 17 10:10:45.731294 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.729564 2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 17 10:10:45.731294 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.729568 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 17 10:10:45.731294 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.729572 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 17 10:10:45.731294 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.729578 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 17 10:10:45.731294 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.729582 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 17 10:10:45.731294 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.729586 2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 17 10:10:45.731294 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.729591 2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 17 10:10:45.731294 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.729595 2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 17 10:10:45.731294 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.729599 2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 17 10:10:45.731294 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.729604 2581 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 17 10:10:45.731294 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.729608 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 17 10:10:45.731294 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.729612 2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 17 10:10:45.731294 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.729616 2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 17 10:10:45.732103 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.729620 2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 17 10:10:45.732103 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.729624 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 17 10:10:45.732103 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.729629 2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 17 10:10:45.732103 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.729636 2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 17 10:10:45.732103 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.729643 2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 17 10:10:45.732103 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.729650 2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 17 10:10:45.732103 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.729655 2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 17 10:10:45.732103 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.729660 2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 17 10:10:45.732103 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.729666 2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 17 10:10:45.732103 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.729671 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 17 10:10:45.732103 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.729675 2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 17 10:10:45.732103 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.729681 2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 17 10:10:45.732103 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.729686 2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 17 10:10:45.732103 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.729690 2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 17 10:10:45.732103 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.729694 2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 17 10:10:45.732103 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.729699 2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 17 10:10:45.732103 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.729703 2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 17 10:10:45.732103 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.729710 2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 17 10:10:45.732103 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.729717 2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 17 10:10:45.732610 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.729721 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 17 10:10:45.732610 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.729726 2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 17 10:10:45.732610 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.729731 2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 17 10:10:45.732610 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.729736 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 17 10:10:45.732610 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.729740 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 17 10:10:45.732610 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.729747 2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 17 10:10:45.732610 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.729752 2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 17 10:10:45.732610 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.729756 2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 17 10:10:45.732610 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.729761 2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 17 10:10:45.732610 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.729765 2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 17 10:10:45.732610 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.729769 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 17 10:10:45.732610 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.729773 2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 17 10:10:45.732610 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.729795 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 17 10:10:45.732610 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.729800 2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 17 10:10:45.732610 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.729804 2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 17 10:10:45.732610 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.729809 2581 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 17 10:10:45.732610 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.729814 2581 feature_gate.go:328] unrecognized feature gate: Example2 Sep 17 10:10:45.732610 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.731057 2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 17 10:10:45.732610 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.731068 2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 17 10:10:45.732610 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.731073 2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 17 10:10:45.733477 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.731077 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 17 10:10:45.733477 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.731081 2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 17 10:10:45.733477 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.731087 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 17 10:10:45.733477 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.731093 2581 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 17 10:10:45.733477 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.731097 2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 17 10:10:45.733477 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.731102 2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 17 10:10:45.733477 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.731107 2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 17 10:10:45.733477 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.731111 2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 17 10:10:45.733477 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.731115 2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 17 10:10:45.733477 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.731120 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 17 10:10:45.733477 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.731124 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 17 10:10:45.733477 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.731128 2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 17 10:10:45.733477 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.731133 2581 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 17 10:10:45.733477 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.731138 2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 17 10:10:45.733477 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.731143 2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 17 10:10:45.733477 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.731147 2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 17 10:10:45.733477 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.731151 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 17 10:10:45.733477 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.731155 2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 17 10:10:45.733477 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.731160 2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 17 10:10:45.733477 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.731164 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 17 10:10:45.734178 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.731168 2581 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 17 10:10:45.734178 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.731172 2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 17 10:10:45.734178 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.731176 2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 17 10:10:45.734178 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.731181 2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 17 10:10:45.734178 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.731185 2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 17 10:10:45.734178 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.731190 2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 17 10:10:45.734178 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.731194 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 17 10:10:45.734178 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.731198 2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 17 10:10:45.734178 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.731203 2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 17 10:10:45.734178 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.731207 2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 17 10:10:45.734178 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.731211 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 17 10:10:45.734178 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.731216 2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 17 10:10:45.734178 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.731220 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 17 10:10:45.734178 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.731224 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 17 10:10:45.734178 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.731228 2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 17 10:10:45.734178 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.731233 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 17 10:10:45.734178 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.731238 2581 feature_gate.go:328] unrecognized feature gate: Example Sep 17 10:10:45.734178 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.731242 2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 17 10:10:45.734178 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.731246 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 17 10:10:45.734178 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.731251 2581 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 17 10:10:45.734725 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.731255 2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 17 10:10:45.734725 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.731260 2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 17 10:10:45.734725 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.731264 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 17 10:10:45.734725 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.731268 2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 17 10:10:45.734725 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.731272 2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 17 10:10:45.734725 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.731277 2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 17 10:10:45.734725 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.731282 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 17 10:10:45.734725 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.731296 2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 17 10:10:45.734725 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.731302 2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 17 10:10:45.734725 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.731308 2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 17 10:10:45.734725 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.731314 2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 17 10:10:45.734725 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.731320 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 17 10:10:45.734725 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.731325 2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 17 10:10:45.734725 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.731330 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 17 10:10:45.734725 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.731334 2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 17 10:10:45.734725 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.731338 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 17 10:10:45.734725 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.731343 2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 17 10:10:45.734725 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.731347 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 17 10:10:45.734725 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.731351 2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 17 10:10:45.735220 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.731356 2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 17 10:10:45.735220 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.731360 2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 17 10:10:45.735220 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.731364 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 17 10:10:45.735220 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.731369 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 17 10:10:45.735220 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.731373 2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 17 10:10:45.735220 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.731378 2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 17 10:10:45.735220 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.731382 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 17 10:10:45.735220 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.731386 2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 17 10:10:45.735220 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.731392 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 17 10:10:45.735220 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.731404 2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 17 10:10:45.735220 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.731409 2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 17 10:10:45.735220 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.731413 2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 17 10:10:45.735220 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.731418 2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 17 10:10:45.735220 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.731422 2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 17 10:10:45.735220 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.731426 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 17 10:10:45.735220 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.731430 2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 17 10:10:45.735220 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.731434 2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 17 10:10:45.735220 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.731438 2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 17 10:10:45.735220 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.731445 2581 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 17 10:10:45.735220 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.731449 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 17 10:10:45.735717 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.731456 2581 feature_gate.go:328] unrecognized feature gate: Example2 Sep 17 10:10:45.735717 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.731460 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 17 10:10:45.735717 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.731464 2581 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 17 10:10:45.735717 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.731468 2581 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 17 10:10:45.735717 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.731472 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 17 10:10:45.735717 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.731476 2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 17 10:10:45.735717 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.731480 2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 17 10:10:45.735717 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.731484 2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 17 10:10:45.735717 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.731489 2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 17 10:10:45.735717 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.731493 2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 17 10:10:45.735717 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.731499 2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 17 10:10:45.735717 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.731505 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 17 10:10:45.735717 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.731510 2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 17 10:10:45.735717 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.731515 2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 17 10:10:45.735717 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.731520 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 17 10:10:45.735717 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.731524 2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 17 10:10:45.735717 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.731529 2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 17 10:10:45.735717 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.732748 2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 17 10:10:45.735717 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.732758 2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 17 10:10:45.735717 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.732763 2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 17 10:10:45.736228 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.732771 2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 17 10:10:45.736228 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.732801 2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 17 10:10:45.736228 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.732808 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 17 10:10:45.736228 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.732812 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 17 10:10:45.736228 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.732825 2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 17 10:10:45.736228 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.732829 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 17 10:10:45.736228 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.732834 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 17 10:10:45.736228 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.732838 2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 17 10:10:45.736228 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.732843 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 17 10:10:45.736228 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.732847 2581 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 17 10:10:45.736228 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.732852 2581 feature_gate.go:328] unrecognized feature gate: Example Sep 17 10:10:45.736228 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.732856 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 17 10:10:45.736228 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.732860 2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 17 10:10:45.736228 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.732865 2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 17 10:10:45.736228 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.732869 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 17 10:10:45.736228 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.732873 2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 17 10:10:45.736228 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.732877 2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 17 10:10:45.736228 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.732882 2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 17 10:10:45.736228 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.732886 2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 17 10:10:45.736228 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.732891 2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 17 10:10:45.736706 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.732895 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 17 10:10:45.736706 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.732899 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 17 10:10:45.736706 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.732904 2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 17 10:10:45.736706 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.732909 2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 17 10:10:45.736706 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.732913 2581 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 17 10:10:45.736706 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.732918 2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 17 10:10:45.736706 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.732922 2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 17 10:10:45.736706 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.732926 2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 17 10:10:45.736706 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.732931 2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 17 10:10:45.736706 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.732935 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 17 10:10:45.736706 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.732940 2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 17 10:10:45.736706 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.732943 2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 17 10:10:45.736706 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.732947 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 17 10:10:45.736706 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.732951 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 17 10:10:45.736706 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.732964 2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 17 10:10:45.736706 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.732969 2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 17 10:10:45.736706 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.732973 2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 17 10:10:45.736706 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.732977 2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 17 10:10:45.736706 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.732983 2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 17 10:10:45.737190 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.732990 2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 17 10:10:45.737190 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.732997 2581 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 17 10:10:45.737190 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.733002 2581 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 17 10:10:45.737190 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.733007 2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 17 10:10:45.737190 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.733011 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 17 10:10:45.737190 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.733016 2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 17 10:10:45.737190 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.733020 2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 17 10:10:45.737190 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.733024 2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 17 10:10:45.737190 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.733028 2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 17 10:10:45.737190 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.733032 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 17 10:10:45.737190 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.733036 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 17 10:10:45.737190 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.733040 2581 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 17 10:10:45.737190 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.733044 2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 17 10:10:45.737190 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.733050 2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 17 10:10:45.737190 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.733054 2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 17 10:10:45.737190 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.733058 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 17 10:10:45.737190 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.733063 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 17 10:10:45.737190 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.733067 2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 17 10:10:45.737190 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.733071 2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 17 10:10:45.737190 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.733075 2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 17 10:10:45.737691 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.733079 2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 17 10:10:45.737691 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.733084 2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 17 10:10:45.737691 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.733088 2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 17 10:10:45.737691 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.733093 2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 17 10:10:45.737691 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.733097 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 17 10:10:45.737691 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.733101 2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 17 10:10:45.737691 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.733105 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 17 10:10:45.737691 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.733109 2581 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 17 10:10:45.737691 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.733123 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 17 10:10:45.737691 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.733129 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 17 10:10:45.737691 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.733133 2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 17 10:10:45.737691 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.733137 2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 17 10:10:45.737691 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.733141 2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 17 10:10:45.737691 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.733145 2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 17 10:10:45.737691 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.733149 2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 17 10:10:45.737691 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.733154 2581 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 17 10:10:45.737691 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.733158 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 17 10:10:45.737691 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.733162 2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 17 10:10:45.737691 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.733166 2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 17 10:10:45.737691 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.733170 2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 17 10:10:45.738198 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.733175 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 17 10:10:45.738198 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.733179 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 17 10:10:45.738198 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.733183 2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 17 10:10:45.738198 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.733187 2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 17 10:10:45.738198 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.733191 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 17 10:10:45.738198 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.733196 2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 17 10:10:45.738198 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.733199 2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 17 10:10:45.738198 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.733204 2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 17 10:10:45.738198 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.733208 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 17 10:10:45.738198 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.733212 2581 feature_gate.go:328] unrecognized feature gate: Example2 Sep 17 10:10:45.738198 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.733217 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 17 10:10:45.738198 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.733221 2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 17 10:10:45.738198 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.733225 2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 17 10:10:45.738198 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.733229 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 17 10:10:45.738198 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.733233 2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 17 10:10:45.738198 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.733237 2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 17 10:10:45.738198 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.733241 2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 17 10:10:45.738198 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.733341 2581 flags.go:64] FLAG: --address="0.0.0.0" Sep 17 10:10:45.738198 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.733392 2581 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 17 10:10:45.738198 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.733403 2581 flags.go:64] FLAG: --anonymous-auth="true" Sep 17 10:10:45.738686 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.733410 2581 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 17 10:10:45.738686 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.733422 2581 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 17 10:10:45.738686 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.733428 2581 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 17 10:10:45.738686 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.733434 2581 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 17 10:10:45.738686 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.733442 2581 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 17 10:10:45.738686 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.733447 2581 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 17 10:10:45.738686 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.733452 2581 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 17 10:10:45.738686 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.733458 2581 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 17 10:10:45.738686 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.733463 2581 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 17 10:10:45.738686 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.733468 2581 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 17 10:10:45.738686 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.733473 2581 flags.go:64] FLAG: --cgroup-root="" Sep 17 10:10:45.738686 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.733477 2581 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 17 10:10:45.738686 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.733482 2581 flags.go:64] FLAG: --client-ca-file="" Sep 17 10:10:45.738686 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.733486 2581 flags.go:64] FLAG: --cloud-provider="external" Sep 17 10:10:45.738686 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.733491 2581 flags.go:64] FLAG: --cluster-dns="[]" Sep 17 10:10:45.738686 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.733498 2581 flags.go:64] FLAG: --cluster-domain="" Sep 17 10:10:45.738686 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.733503 2581 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 17 10:10:45.738686 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.733508 2581 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 17 10:10:45.738686 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.733513 2581 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 17 10:10:45.738686 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.733518 2581 flags.go:64] FLAG: --container-log-max-files="5" Sep 17 10:10:45.738686 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.733525 2581 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 17 10:10:45.738686 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.733530 2581 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 17 10:10:45.738686 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.733535 2581 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 17 10:10:45.739262 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.733540 2581 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 17 10:10:45.739262 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.733545 2581 flags.go:64] FLAG: --contention-profiling="false" Sep 17 10:10:45.739262 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.733550 2581 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 17 10:10:45.739262 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.733554 2581 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 17 10:10:45.739262 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.733560 2581 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 17 10:10:45.739262 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.733564 2581 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 17 10:10:45.739262 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.733571 2581 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 17 10:10:45.739262 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.733576 2581 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 17 10:10:45.739262 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.733580 2581 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 17 10:10:45.739262 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.733585 2581 flags.go:64] FLAG: --enable-load-reader="false" Sep 17 10:10:45.739262 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.733590 2581 flags.go:64] FLAG: --enable-server="true" Sep 17 10:10:45.739262 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.733600 2581 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 17 10:10:45.739262 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.733608 2581 flags.go:64] FLAG: --event-burst="100" Sep 17 10:10:45.739262 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.733613 2581 flags.go:64] FLAG: --event-qps="50" Sep 17 10:10:45.739262 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.733620 2581 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 17 10:10:45.739262 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.733625 2581 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 17 10:10:45.739262 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.733630 2581 flags.go:64] FLAG: --eviction-hard="" Sep 17 10:10:45.739262 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.733636 2581 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 17 10:10:45.739262 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.733641 2581 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 17 10:10:45.739262 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.733646 2581 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 17 10:10:45.739262 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.733651 2581 flags.go:64] FLAG: --eviction-soft="" Sep 17 10:10:45.739262 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.733655 2581 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 17 10:10:45.739262 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.733660 2581 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 17 10:10:45.739262 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.733665 2581 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 17 10:10:45.739262 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.733670 2581 flags.go:64] FLAG: --experimental-mounter-path="" Sep 17 10:10:45.739933 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.733674 2581 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 17 10:10:45.739933 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.733679 2581 flags.go:64] FLAG: --fail-swap-on="true" Sep 17 10:10:45.739933 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.733684 2581 flags.go:64] FLAG: --feature-gates="" Sep 17 10:10:45.739933 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.733691 2581 flags.go:64] FLAG: --file-check-frequency="20s" Sep 17 10:10:45.739933 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.733696 2581 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 17 10:10:45.739933 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.733701 2581 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 17 10:10:45.739933 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.733706 2581 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 17 10:10:45.739933 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.733711 2581 flags.go:64] FLAG: --healthz-port="10248" Sep 17 10:10:45.739933 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.733716 2581 flags.go:64] FLAG: --help="false" Sep 17 10:10:45.739933 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.733720 2581 flags.go:64] FLAG: --hostname-override="ip-10-0-137-203.ec2.internal" Sep 17 10:10:45.739933 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.733726 2581 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 17 10:10:45.739933 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.733731 2581 flags.go:64] FLAG: --http-check-frequency="20s" Sep 17 10:10:45.739933 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.733735 2581 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 17 10:10:45.739933 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.733741 2581 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 17 10:10:45.739933 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.733747 2581 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 17 10:10:45.739933 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.733752 2581 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 17 10:10:45.739933 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.733756 2581 flags.go:64] FLAG: --image-service-endpoint="" Sep 17 10:10:45.739933 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.733760 2581 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 17 10:10:45.739933 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.733765 2581 flags.go:64] FLAG: --kube-api-burst="100" Sep 17 10:10:45.739933 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.733771 2581 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 17 10:10:45.739933 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.733794 2581 flags.go:64] FLAG: --kube-api-qps="50" Sep 17 10:10:45.739933 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.733802 2581 flags.go:64] FLAG: --kube-reserved="" Sep 17 10:10:45.739933 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.733807 2581 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 17 10:10:45.739933 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.733811 2581 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 17 10:10:45.740528 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.733816 2581 flags.go:64] FLAG: --kubelet-cgroups="" Sep 17 10:10:45.740528 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.733821 2581 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 17 10:10:45.740528 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.733826 2581 flags.go:64] FLAG: --lock-file="" Sep 17 10:10:45.740528 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.733831 2581 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 17 10:10:45.740528 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.733836 2581 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 17 10:10:45.740528 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.733841 2581 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 17 10:10:45.740528 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.733850 2581 flags.go:64] FLAG: --log-json-split-stream="false" Sep 17 10:10:45.740528 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.733854 2581 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 17 10:10:45.740528 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.733861 2581 flags.go:64] FLAG: --log-text-split-stream="false" Sep 17 10:10:45.740528 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.733865 2581 flags.go:64] FLAG: --logging-format="text" Sep 17 10:10:45.740528 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.733870 2581 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 17 10:10:45.740528 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.733875 2581 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 17 10:10:45.740528 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.733880 2581 flags.go:64] FLAG: --manifest-url="" Sep 17 10:10:45.740528 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.733915 2581 flags.go:64] FLAG: --manifest-url-header="" Sep 17 10:10:45.740528 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.733942 2581 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 17 10:10:45.740528 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.733947 2581 flags.go:64] FLAG: --max-open-files="1000000" Sep 17 10:10:45.740528 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.733953 2581 flags.go:64] FLAG: --max-pods="110" Sep 17 10:10:45.740528 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.733956 2581 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 17 10:10:45.740528 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.733960 2581 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 17 10:10:45.740528 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.733964 2581 flags.go:64] FLAG: --memory-manager-policy="None" Sep 17 10:10:45.740528 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.733968 2581 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 17 10:10:45.740528 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.733971 2581 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 17 10:10:45.740528 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.733975 2581 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 17 10:10:45.740528 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.733978 2581 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 17 10:10:45.740528 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.733988 2581 flags.go:64] FLAG: --node-status-max-images="50" Sep 17 10:10:45.741192 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.733991 2581 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 17 10:10:45.741192 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.733994 2581 flags.go:64] FLAG: --oom-score-adj="-999" Sep 17 10:10:45.741192 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.733998 2581 flags.go:64] FLAG: --pod-cidr="" Sep 17 10:10:45.741192 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.734001 2581 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10" Sep 17 10:10:45.741192 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.734007 2581 flags.go:64] FLAG: --pod-manifest-path="" Sep 17 10:10:45.741192 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.734010 2581 flags.go:64] FLAG: --pod-max-pids="-1" Sep 17 10:10:45.741192 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.734013 2581 flags.go:64] FLAG: --pods-per-core="0" Sep 17 10:10:45.741192 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.734016 2581 flags.go:64] FLAG: --port="10250" Sep 17 10:10:45.741192 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.734020 2581 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 17 10:10:45.741192 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.734023 2581 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-08b3a484ca1b9897f" Sep 17 10:10:45.741192 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.734027 2581 flags.go:64] FLAG: --qos-reserved="" Sep 17 10:10:45.741192 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.734030 2581 flags.go:64] FLAG: --read-only-port="10255" Sep 17 10:10:45.741192 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.734033 2581 flags.go:64] FLAG: --register-node="true" Sep 17 10:10:45.741192 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.734036 2581 flags.go:64] FLAG: --register-with-taints="" Sep 17 10:10:45.741192 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.734045 2581 flags.go:64] FLAG: --registry-burst="10" Sep 17 10:10:45.741192 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.734048 2581 flags.go:64] FLAG: --registry-qps="5" Sep 17 10:10:45.741192 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.734051 2581 flags.go:64] FLAG: --reserved-cpus="" Sep 17 10:10:45.741192 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.734054 2581 flags.go:64] FLAG: --reserved-memory="" Sep 17 10:10:45.741192 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.734058 2581 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 17 10:10:45.741192 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.734061 2581 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 17 10:10:45.741192 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.734064 2581 flags.go:64] FLAG: --rotate-certificates="false" Sep 17 10:10:45.741192 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.734074 2581 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 17 10:10:45.741192 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.734077 2581 flags.go:64] FLAG: --runonce="false" Sep 17 10:10:45.741192 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.734080 2581 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 17 10:10:45.741192 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.734084 2581 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 17 10:10:45.741943 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.734087 2581 flags.go:64] FLAG: --seccomp-default="false" Sep 17 10:10:45.741943 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.734090 2581 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 17 10:10:45.741943 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.734093 2581 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 17 10:10:45.741943 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.734096 2581 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 17 10:10:45.741943 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.734099 2581 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 17 10:10:45.741943 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.734108 2581 flags.go:64] FLAG: --storage-driver-password="root" Sep 17 10:10:45.741943 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.734112 2581 flags.go:64] FLAG: --storage-driver-secure="false" Sep 17 10:10:45.741943 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.734115 2581 flags.go:64] FLAG: --storage-driver-table="stats" Sep 17 10:10:45.741943 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.734118 2581 flags.go:64] FLAG: --storage-driver-user="root" Sep 17 10:10:45.741943 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.734121 2581 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 17 10:10:45.741943 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.734124 2581 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 17 10:10:45.741943 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.734127 2581 flags.go:64] FLAG: --system-cgroups="" Sep 17 10:10:45.741943 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.734130 2581 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 17 10:10:45.741943 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.734145 2581 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 17 10:10:45.741943 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.734148 2581 flags.go:64] FLAG: --tls-cert-file="" Sep 17 10:10:45.741943 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.734151 2581 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 17 10:10:45.741943 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.734159 2581 flags.go:64] FLAG: --tls-min-version="" Sep 17 10:10:45.741943 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.734162 2581 flags.go:64] FLAG: --tls-private-key-file="" Sep 17 10:10:45.741943 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.734165 2581 flags.go:64] FLAG: --topology-manager-policy="none" Sep 17 10:10:45.741943 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.734168 2581 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 17 10:10:45.741943 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.734171 2581 flags.go:64] FLAG: --topology-manager-scope="container" Sep 17 10:10:45.741943 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.734175 2581 flags.go:64] FLAG: --v="2" Sep 17 10:10:45.741943 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.734180 2581 flags.go:64] FLAG: --version="false" Sep 17 10:10:45.741943 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.734185 2581 flags.go:64] FLAG: --vmodule="" Sep 17 10:10:45.741943 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.734189 2581 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 17 10:10:45.742573 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.734192 2581 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 17 10:10:45.742573 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.734374 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 17 10:10:45.742573 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.734386 2581 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 17 10:10:45.742573 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.734391 2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 17 10:10:45.742573 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.734401 2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 17 10:10:45.742573 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.734405 2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 17 10:10:45.742573 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.734407 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 17 10:10:45.742573 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.734411 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 17 10:10:45.742573 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.734414 2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 17 10:10:45.742573 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.734417 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 17 10:10:45.742573 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.734420 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 17 10:10:45.742573 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.734423 2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 17 10:10:45.742573 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.734427 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 17 10:10:45.742573 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.734430 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 17 10:10:45.742573 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.734433 2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 17 10:10:45.742573 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.734436 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 17 10:10:45.742573 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.734438 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 17 10:10:45.742573 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.734441 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 17 10:10:45.742573 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.734444 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 17 10:10:45.742573 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.734446 2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 17 10:10:45.743093 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.734449 2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 17 10:10:45.743093 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.734452 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 17 10:10:45.743093 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.734454 2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 17 10:10:45.743093 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.734457 2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 17 10:10:45.743093 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.734460 2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 17 10:10:45.743093 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.734463 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 17 10:10:45.743093 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.734465 2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 17 10:10:45.743093 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.734468 2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 17 10:10:45.743093 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.734471 2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 17 10:10:45.743093 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.734474 2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 17 10:10:45.743093 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.734476 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 17 10:10:45.743093 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.734479 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 17 10:10:45.743093 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.734481 2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 17 10:10:45.743093 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.734484 2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 17 10:10:45.743093 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.734487 2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 17 10:10:45.743093 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.734490 2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 17 10:10:45.743093 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.734492 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 17 10:10:45.743093 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.734501 2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 17 10:10:45.743093 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.734504 2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 17 10:10:45.743093 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.734507 2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 17 10:10:45.743601 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.734509 2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 17 10:10:45.743601 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.734512 2581 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 17 10:10:45.743601 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.734514 2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 17 10:10:45.743601 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.734517 2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 17 10:10:45.743601 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.734520 2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 17 10:10:45.743601 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.734522 2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 17 10:10:45.743601 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.734525 2581 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 17 10:10:45.743601 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.734527 2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 17 10:10:45.743601 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.734530 2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 17 10:10:45.743601 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.734532 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 17 10:10:45.743601 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.734535 2581 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 17 10:10:45.743601 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.734538 2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 17 10:10:45.743601 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.734540 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 17 10:10:45.743601 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.734543 2581 feature_gate.go:328] unrecognized feature gate: Example Sep 17 10:10:45.743601 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.734546 2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 17 10:10:45.743601 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.734548 2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 17 10:10:45.743601 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.734551 2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 17 10:10:45.743601 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.734553 2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 17 10:10:45.743601 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.734556 2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 17 10:10:45.743601 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.734561 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 17 10:10:45.744144 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.734564 2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 17 10:10:45.744144 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.734567 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 17 10:10:45.744144 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.734569 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 17 10:10:45.744144 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.734572 2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 17 10:10:45.744144 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.734576 2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 17 10:10:45.744144 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.734580 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 17 10:10:45.744144 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.734583 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 17 10:10:45.744144 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.734588 2581 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 17 10:10:45.744144 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.734591 2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 17 10:10:45.744144 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.734594 2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 17 10:10:45.744144 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.734603 2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 17 10:10:45.744144 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.734606 2581 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 17 10:10:45.744144 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.734609 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 17 10:10:45.744144 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.734611 2581 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 17 10:10:45.744144 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.734614 2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 17 10:10:45.744144 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.734617 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 17 10:10:45.744144 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.734619 2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 17 10:10:45.744144 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.734622 2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 17 10:10:45.744144 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.734625 2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 17 10:10:45.744620 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.734628 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 17 10:10:45.744620 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.734630 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 17 10:10:45.744620 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.734633 2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 17 10:10:45.744620 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.734635 2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 17 10:10:45.744620 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.734638 2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 17 10:10:45.744620 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.734641 2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 17 10:10:45.744620 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.734643 2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 17 10:10:45.744620 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.734646 2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 17 10:10:45.744620 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.734649 2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 17 10:10:45.744620 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.734651 2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 17 10:10:45.744620 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.734654 2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 17 10:10:45.744620 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.734656 2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 17 10:10:45.744620 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.734662 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 17 10:10:45.744620 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.734666 2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 17 10:10:45.744620 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.734670 2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 17 10:10:45.744620 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.734674 2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 17 10:10:45.744620 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.734678 2581 feature_gate.go:328] unrecognized feature gate: Example2 Sep 17 10:10:45.744620 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.734681 2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 17 10:10:45.744620 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.734685 2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 17 10:10:45.745102 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.734688 2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 17 10:10:45.745102 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.734694 2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 17 10:10:45.745102 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.735509 2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 17 10:10:45.745102 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.742642 2581 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 17 10:10:45.745102 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.742662 2581 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 17 10:10:45.745102 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.743383 2581 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 17 10:10:45.745102 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.743390 2581 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 17 10:10:45.745102 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.743470 2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 17 10:10:45.745102 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.743476 2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 17 10:10:45.745102 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.743479 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 17 10:10:45.745102 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.743482 2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 17 10:10:45.745102 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.743485 2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 17 10:10:45.745102 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.743489 2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 17 10:10:45.745102 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.743491 2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 17 10:10:45.745102 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.743494 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 17 10:10:45.745486 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.743497 2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 17 10:10:45.745486 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.743500 2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 17 10:10:45.745486 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.743502 2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 17 10:10:45.745486 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.743505 2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 17 10:10:45.745486 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.743508 2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 17 10:10:45.745486 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.743511 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 17 10:10:45.745486 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.743515 2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 17 10:10:45.745486 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.743517 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 17 10:10:45.745486 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.743521 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 17 10:10:45.745486 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.743524 2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 17 10:10:45.745486 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.743526 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 17 10:10:45.745486 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.743529 2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 17 10:10:45.745486 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.743532 2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 17 10:10:45.745486 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.743536 2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 17 10:10:45.745486 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.743541 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 17 10:10:45.745486 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.743544 2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 17 10:10:45.745486 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.743547 2581 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 17 10:10:45.745486 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.743549 2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 17 10:10:45.745486 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.743552 2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 17 10:10:45.746020 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.743555 2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 17 10:10:45.746020 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.743558 2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 17 10:10:45.746020 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.743561 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 17 10:10:45.746020 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.743565 2581 feature_gate.go:328] unrecognized feature gate: Example2 Sep 17 10:10:45.746020 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.743568 2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 17 10:10:45.746020 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.743571 2581 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 17 10:10:45.746020 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.743573 2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 17 10:10:45.746020 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.743576 2581 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 17 10:10:45.746020 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.743579 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 17 10:10:45.746020 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.743581 2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 17 10:10:45.746020 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.743584 2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 17 10:10:45.746020 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.743586 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 17 10:10:45.746020 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.743589 2581 feature_gate.go:328] unrecognized feature gate: Example Sep 17 10:10:45.746020 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.743591 2581 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 17 10:10:45.746020 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.743594 2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 17 10:10:45.746020 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.743597 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 17 10:10:45.746020 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.743599 2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 17 10:10:45.746020 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.743602 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 17 10:10:45.746020 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.743605 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 17 10:10:45.746020 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.743608 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 17 10:10:45.746517 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.743611 2581 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 17 10:10:45.746517 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.743614 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 17 10:10:45.746517 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.743616 2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 17 10:10:45.746517 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.743619 2581 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 17 10:10:45.746517 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.743622 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 17 10:10:45.746517 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.743624 2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 17 10:10:45.746517 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.743627 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 17 10:10:45.746517 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.743629 2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 17 10:10:45.746517 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.743632 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 17 10:10:45.746517 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.743634 2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 17 10:10:45.746517 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.743637 2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 17 10:10:45.746517 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.743639 2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 17 10:10:45.746517 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.743642 2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 17 10:10:45.746517 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.743645 2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 17 10:10:45.746517 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.743650 2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 17 10:10:45.746517 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.743653 2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 17 10:10:45.746517 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.743656 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 17 10:10:45.746517 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.743658 2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 17 10:10:45.746517 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.743661 2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 17 10:10:45.747005 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.743665 2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 17 10:10:45.747005 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.743668 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 17 10:10:45.747005 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.743671 2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 17 10:10:45.747005 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.743674 2581 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 17 10:10:45.747005 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.743677 2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 17 10:10:45.747005 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.743679 2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 17 10:10:45.747005 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.743682 2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 17 10:10:45.747005 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.743684 2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 17 10:10:45.747005 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.743687 2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 17 10:10:45.747005 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.743689 2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 17 10:10:45.747005 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.743692 2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 17 10:10:45.747005 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.743694 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 17 10:10:45.747005 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.743697 2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 17 10:10:45.747005 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.743700 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 17 10:10:45.747005 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.743703 2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 17 10:10:45.747005 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.743705 2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 17 10:10:45.747005 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.743708 2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 17 10:10:45.747005 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.743710 2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 17 10:10:45.747005 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.743713 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 17 10:10:45.747516 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.743715 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 17 10:10:45.747516 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.743717 2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 17 10:10:45.747516 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.743720 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 17 10:10:45.747516 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.743722 2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 17 10:10:45.747516 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.743725 2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 17 10:10:45.747516 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.743728 2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 17 10:10:45.747516 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.743731 2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 17 10:10:45.747516 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.743733 2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 17 10:10:45.747516 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.743737 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 17 10:10:45.747516 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.743739 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 17 10:10:45.747516 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.743742 2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 17 10:10:45.747516 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.743745 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 17 10:10:45.747516 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.743748 2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 17 10:10:45.747516 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.743750 2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 17 10:10:45.747516 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.743755 2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 17 10:10:45.747897 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.743883 2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 17 10:10:45.747897 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.743888 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 17 10:10:45.747897 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.743891 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 17 10:10:45.747897 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.743894 2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 17 10:10:45.747897 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.743897 2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 17 10:10:45.747897 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.743900 2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 17 10:10:45.747897 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.743903 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 17 10:10:45.747897 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.743906 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 17 10:10:45.747897 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.743909 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 17 10:10:45.747897 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.743912 2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 17 10:10:45.747897 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.743914 2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 17 10:10:45.747897 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.743917 2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 17 10:10:45.747897 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.743920 2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 17 10:10:45.747897 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.743922 2581 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 17 10:10:45.747897 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.743925 2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 17 10:10:45.747897 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.743928 2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 17 10:10:45.747897 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.743931 2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 17 10:10:45.747897 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.743933 2581 feature_gate.go:328] unrecognized feature gate: Example Sep 17 10:10:45.747897 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.743936 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 17 10:10:45.747897 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.743939 2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 17 10:10:45.748399 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.743950 2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 17 10:10:45.748399 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.743952 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 17 10:10:45.748399 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.743955 2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 17 10:10:45.748399 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.743958 2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 17 10:10:45.748399 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.743961 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 17 10:10:45.748399 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.743965 2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 17 10:10:45.748399 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.743968 2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 17 10:10:45.748399 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.743971 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 17 10:10:45.748399 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.743974 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 17 10:10:45.748399 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.743977 2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 17 10:10:45.748399 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.743979 2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 17 10:10:45.748399 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.743982 2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 17 10:10:45.748399 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.743985 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 17 10:10:45.748399 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.743987 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 17 10:10:45.748399 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.743990 2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 17 10:10:45.748399 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.743992 2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 17 10:10:45.748399 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.743995 2581 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 17 10:10:45.748399 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.743997 2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 17 10:10:45.748399 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.744000 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 17 10:10:45.748399 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.744003 2581 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 17 10:10:45.748399 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.744005 2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 17 10:10:45.748932 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.744008 2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 17 10:10:45.748932 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.744010 2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 17 10:10:45.748932 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.744013 2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 17 10:10:45.748932 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.744015 2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 17 10:10:45.748932 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.744018 2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 17 10:10:45.748932 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.744021 2581 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 17 10:10:45.748932 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.744023 2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 17 10:10:45.748932 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.744026 2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 17 10:10:45.748932 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.744028 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 17 10:10:45.748932 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.744031 2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 17 10:10:45.748932 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.744033 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 17 10:10:45.748932 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.744035 2581 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 17 10:10:45.748932 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.744038 2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 17 10:10:45.748932 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.744040 2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 17 10:10:45.748932 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.744043 2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 17 10:10:45.748932 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.744045 2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 17 10:10:45.748932 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.744049 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 17 10:10:45.748932 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.744052 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 17 10:10:45.748932 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.744054 2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 17 10:10:45.748932 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.744057 2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 17 10:10:45.749415 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.744059 2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 17 10:10:45.749415 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.744063 2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 17 10:10:45.749415 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.744066 2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 17 10:10:45.749415 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.744069 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 17 10:10:45.749415 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.744072 2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 17 10:10:45.749415 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.744074 2581 feature_gate.go:328] unrecognized feature gate: Example2 Sep 17 10:10:45.749415 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.744077 2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 17 10:10:45.749415 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.744079 2581 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 17 10:10:45.749415 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.744082 2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 17 10:10:45.749415 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.744084 2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 17 10:10:45.749415 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.744087 2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 17 10:10:45.749415 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.744089 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 17 10:10:45.749415 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.744092 2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 17 10:10:45.749415 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.744094 2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 17 10:10:45.749415 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.744098 2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 17 10:10:45.749415 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.744102 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 17 10:10:45.749415 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.744105 2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 17 10:10:45.749415 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.744109 2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 17 10:10:45.749415 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.744112 2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 17 10:10:45.750022 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.744115 2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 17 10:10:45.750022 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.744118 2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 17 10:10:45.750022 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.744120 2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 17 10:10:45.750022 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.744123 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 17 10:10:45.750022 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.744125 2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 17 10:10:45.750022 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.744129 2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 17 10:10:45.750022 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.744132 2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 17 10:10:45.750022 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.744135 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 17 10:10:45.750022 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.744138 2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 17 10:10:45.750022 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.744142 2581 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 17 10:10:45.750022 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.744145 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 17 10:10:45.750022 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.744148 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 17 10:10:45.750022 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.744151 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 17 10:10:45.750022 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.744154 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 17 10:10:45.750022 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.744156 2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 17 10:10:45.750022 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.744160 2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 17 10:10:45.750022 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.744163 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 17 10:10:45.750022 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.744165 2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 17 10:10:45.750022 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:45.744168 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 17 10:10:45.750490 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.744173 2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 17 10:10:45.750490 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.744296 2581 server.go:962] "Client rotation is on, will bootstrap in background" Sep 17 10:10:45.750490 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.748094 2581 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 17 10:10:45.750490 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.748871 2581 server.go:1019] "Starting client certificate rotation" Sep 17 10:10:45.750490 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.748979 2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 17 10:10:45.750490 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.749035 2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 17 10:10:45.751474 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.751458 2581 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 17 10:10:45.754513 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.754487 2581 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 17 10:10:45.764138 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.764115 2581 log.go:25] "Validated CRI v1 runtime API" Sep 17 10:10:45.768552 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.768535 2581 log.go:25] "Validated CRI v1 image API" Sep 17 10:10:45.769538 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.769516 2581 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 17 10:10:45.772460 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.772441 2581 fs.go:135] Filesystem UUIDs: map[23c2b94d-a7d4-4eea-a9d0-3fd78397f4a0:/dev/nvme0n1p3 737a9e72-337c-4360-9ed4-656de88ebae7:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Sep 17 10:10:45.772534 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.772460 2581 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 17 10:10:45.778290 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.778168 2581 manager.go:216] Machine: {Timestamp:2026-09-17 10:10:45.776621082 +0000 UTC m=+0.325178183 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101665 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2a8aa583a81c002e116b24e481a999 SystemUUID:ec2a8aa5-83a8-1c00-2e11-6b24e481a999 BootID:ddca51df-7814-4862-8b5f-d3731affbdfb Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:de:2a:ad:67 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:de:2a:ad:67 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:16:69:f1:42:61:52 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 17 10:10:45.778290 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.778282 2581 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 17 10:10:45.778443 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.778369 2581 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 17 10:10:45.778634 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.778618 2581 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 17 10:10:45.782155 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.782129 2581 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 17 10:10:45.782313 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.782160 2581 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-203.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 17 10:10:45.783465 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.783452 2581 topology_manager.go:138] "Creating topology manager with none policy" Sep 17 10:10:45.783502 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.783468 2581 container_manager_linux.go:309] "Creating device plugin manager" Sep 17 10:10:45.783502 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.783476 2581 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 17 10:10:45.783502 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.783490 2581 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 17 10:10:45.784848 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.784836 2581 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 17 10:10:45.788409 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.788396 2581 state_mem.go:36] "Initialized new in-memory state store" Sep 17 10:10:45.788537 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.788527 2581 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 17 10:10:45.788575 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.788558 2581 kubelet.go:486] "Attempting to sync node with API server" Sep 17 10:10:45.788606 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.788578 2581 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 17 10:10:45.788606 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.788598 2581 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 17 10:10:45.788661 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.788616 2581 kubelet.go:398] "Adding apiserver pod source" Sep 17 10:10:45.788661 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.788628 2581 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 17 10:10:45.789106 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.789094 2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 17 10:10:45.789162 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.789115 2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 17 10:10:45.793132 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.793111 2581 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 17 10:10:45.795548 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.795534 2581 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 17 10:10:45.795613 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.795563 2581 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 17 10:10:45.797167 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.797151 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 17 10:10:45.797167 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.797169 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 17 10:10:45.797248 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.797176 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 17 10:10:45.797248 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.797181 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 17 10:10:45.797248 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.797193 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 17 10:10:45.797248 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.797198 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 17 10:10:45.797248 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.797211 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 17 10:10:45.797248 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.797217 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 17 10:10:45.797248 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.797223 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 17 10:10:45.797248 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.797228 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 17 10:10:45.797248 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.797248 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 17 10:10:45.797482 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.797254 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 17 10:10:45.797703 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:10:45.797682 2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-137-203.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 17 10:10:45.797750 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:10:45.797683 2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 17 10:10:45.798126 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.798116 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 17 10:10:45.798164 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.798126 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 17 10:10:45.801981 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.801967 2581 server.go:1268] "Started kubelet" Sep 17 10:10:45.802098 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.802043 2581 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 17 10:10:45.802900 ip-10-0-137-203 systemd[1]: Started Kubernetes Kubelet. Sep 17 10:10:45.804941 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.804693 2581 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 17 10:10:45.804998 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.804968 2581 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 17 10:10:45.806104 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.806083 2581 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 17 10:10:45.806759 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.806741 2581 server.go:310] "Adding debug handlers to kubelet server" Sep 17 10:10:45.808138 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:10:45.808103 2581 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 17 10:10:45.811283 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.811264 2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-tc49q" Sep 17 10:10:45.812649 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.812632 2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 17 10:10:45.812793 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.812759 2581 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-137-203.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 17 10:10:45.813293 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.813280 2581 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 17 10:10:45.813428 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:10:45.813413 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-203.ec2.internal\" not found" Sep 17 10:10:45.813874 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:10:45.812803 2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-203.ec2.internal.18d613f534e8c33d default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-203.ec2.internal,UID:ip-10-0-137-203.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-137-203.ec2.internal,},FirstTimestamp:2026-09-17 10:10:45.801935677 +0000 UTC m=+0.350492779,LastTimestamp:2026-09-17 10:10:45.801935677 +0000 UTC m=+0.350492779,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-203.ec2.internal,}" Sep 17 10:10:45.814014 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.813863 2581 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 17 10:10:45.814014 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.814013 2581 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 17 10:10:45.814173 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.814091 2581 factory.go:55] Registering systemd factory Sep 17 10:10:45.814173 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.814106 2581 factory.go:223] Registration of the systemd container factory successfully Sep 17 10:10:45.814173 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.814132 2581 reconstruct.go:97] "Volume reconstruction finished" Sep 17 10:10:45.814173 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.814138 2581 reconciler.go:29] "Reconciler: start to sync state" Sep 17 10:10:45.814173 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.814159 2581 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 17 10:10:45.814399 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.814331 2581 factory.go:153] Registering CRI-O factory Sep 17 10:10:45.814399 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.814348 2581 factory.go:223] Registration of the crio container factory successfully Sep 17 10:10:45.814492 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.814403 2581 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 17 10:10:45.814492 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.814421 2581 factory.go:103] Registering Raw factory Sep 17 10:10:45.814492 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.814432 2581 manager.go:1196] Started watching for new ooms in manager Sep 17 10:10:45.814840 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.814826 2581 manager.go:349] Starting recovery of all containers Sep 17 10:10:45.818502 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:10:45.818472 2581 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 17 10:10:45.818625 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:10:45.818502 2581 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-137-203.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 17 10:10:45.824000 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.823976 2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-tc49q" Sep 17 10:10:45.826023 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.826005 2581 manager.go:354] Recovery completed Sep 17 10:10:45.831741 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.831724 2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 17 10:10:45.834292 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.834273 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-203.ec2.internal" event="NodeHasSufficientMemory" Sep 17 10:10:45.834385 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.834304 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-203.ec2.internal" event="NodeHasNoDiskPressure" Sep 17 10:10:45.834385 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.834316 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-203.ec2.internal" event="NodeHasSufficientPID" Sep 17 10:10:45.834970 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.834955 2581 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 17 10:10:45.834970 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.834967 2581 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 17 10:10:45.835107 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.834972 2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 17 10:10:45.835107 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.834982 2581 state_mem.go:36] "Initialized new in-memory state store" Sep 17 10:10:45.835933 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.835910 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-203.ec2.internal" event="NodeHasSufficientMemory" Sep 17 10:10:45.836003 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.835944 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-203.ec2.internal" event="NodeHasNoDiskPressure" Sep 17 10:10:45.836003 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.835959 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-203.ec2.internal" event="NodeHasSufficientPID" Sep 17 10:10:45.836169 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:10:45.836084 2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-203.ec2.internal.18d613f536d677be default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-203.ec2.internal,UID:ip-10-0-137-203.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-137-203.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-137-203.ec2.internal,},FirstTimestamp:2026-09-17 10:10:45.834291134 +0000 UTC m=+0.382848237,LastTimestamp:2026-09-17 10:10:45.834291134 +0000 UTC m=+0.382848237,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-203.ec2.internal,}" Sep 17 10:10:45.836378 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.836365 2581 policy_none.go:49] "None policy: Start" Sep 17 10:10:45.836425 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.836382 2581 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 17 10:10:45.836425 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.836392 2581 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 17 10:10:45.837648 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.837636 2581 policy_none.go:47] "Start" Sep 17 10:10:45.878294 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.872434 2581 manager.go:337] "Starting Device Plugin manager" Sep 17 10:10:45.878294 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:10:45.872480 2581 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 17 10:10:45.878294 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.872493 2581 server.go:87] "Starting device plugin registration server" Sep 17 10:10:45.878294 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.872820 2581 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 17 10:10:45.878294 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.872833 2581 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 17 10:10:45.878294 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.872921 2581 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 17 10:10:45.878294 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.873007 2581 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 17 10:10:45.878294 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.873015 2581 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 17 10:10:45.878294 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:10:45.873493 2581 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 17 10:10:45.878294 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:10:45.873546 2581 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-203.ec2.internal\" not found" Sep 17 10:10:45.936734 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.936691 2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 17 10:10:45.937946 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.937924 2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 17 10:10:45.938060 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.937991 2581 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 17 10:10:45.938060 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.938017 2581 kubelet.go:2448] "Starting kubelet main sync loop" Sep 17 10:10:45.938146 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:10:45.938059 2581 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 17 10:10:45.940343 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.940315 2581 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 17 10:10:45.973012 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.972978 2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 17 10:10:45.975306 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.975242 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-203.ec2.internal" event="NodeHasSufficientMemory" Sep 17 10:10:45.975306 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.975287 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-203.ec2.internal" event="NodeHasNoDiskPressure" Sep 17 10:10:45.975306 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.975298 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-203.ec2.internal" event="NodeHasSufficientPID" Sep 17 10:10:45.975471 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.975323 2581 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-137-203.ec2.internal" Sep 17 10:10:45.984177 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:45.984156 2581 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-137-203.ec2.internal" Sep 17 10:10:45.984308 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:10:45.984195 2581 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-137-203.ec2.internal\": node \"ip-10-0-137-203.ec2.internal\" not found" Sep 17 10:10:46.005857 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:10:46.005833 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-203.ec2.internal\" not found" Sep 17 10:10:46.039047 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:46.039008 2581 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-137-203.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-203.ec2.internal"] Sep 17 10:10:46.039212 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:46.039093 2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 17 10:10:46.041001 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:46.040983 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-203.ec2.internal" event="NodeHasSufficientMemory" Sep 17 10:10:46.041090 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:46.041015 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-203.ec2.internal" event="NodeHasNoDiskPressure" Sep 17 10:10:46.041090 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:46.041025 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-203.ec2.internal" event="NodeHasSufficientPID" Sep 17 10:10:46.042211 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:46.042197 2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 17 10:10:46.042307 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:46.042293 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-203.ec2.internal" Sep 17 10:10:46.042375 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:46.042321 2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 17 10:10:46.043007 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:46.042992 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-203.ec2.internal" event="NodeHasSufficientMemory" Sep 17 10:10:46.043097 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:46.043018 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-203.ec2.internal" event="NodeHasNoDiskPressure" Sep 17 10:10:46.043097 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:46.042993 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-203.ec2.internal" event="NodeHasSufficientMemory" Sep 17 10:10:46.043097 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:46.043059 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-203.ec2.internal" event="NodeHasNoDiskPressure" Sep 17 10:10:46.043097 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:46.043070 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-203.ec2.internal" event="NodeHasSufficientPID" Sep 17 10:10:46.043097 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:46.043029 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-203.ec2.internal" event="NodeHasSufficientPID" Sep 17 10:10:46.044285 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:46.044270 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-203.ec2.internal" Sep 17 10:10:46.044334 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:46.044296 2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 17 10:10:46.045046 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:46.045031 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-203.ec2.internal" event="NodeHasSufficientMemory" Sep 17 10:10:46.045128 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:46.045057 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-203.ec2.internal" event="NodeHasNoDiskPressure" Sep 17 10:10:46.045128 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:46.045072 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-203.ec2.internal" event="NodeHasSufficientPID" Sep 17 10:10:46.073723 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:10:46.073695 2581 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-203.ec2.internal\" not found" node="ip-10-0-137-203.ec2.internal" Sep 17 10:10:46.078390 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:10:46.078361 2581 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-203.ec2.internal\" not found" node="ip-10-0-137-203.ec2.internal" Sep 17 10:10:46.106065 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:10:46.106033 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-203.ec2.internal\" not found" Sep 17 10:10:46.115330 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:46.115306 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fcac5a3f4a7d9d4b6710d0af7b8c7711-config\") pod \"kube-apiserver-proxy-ip-10-0-137-203.ec2.internal\" (UID: \"fcac5a3f4a7d9d4b6710d0af7b8c7711\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-203.ec2.internal" Sep 17 10:10:46.115387 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:46.115338 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c075fc41f60411f931a6e82e74eff54a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-203.ec2.internal\" (UID: \"c075fc41f60411f931a6e82e74eff54a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-203.ec2.internal" Sep 17 10:10:46.115387 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:46.115357 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c075fc41f60411f931a6e82e74eff54a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-203.ec2.internal\" (UID: \"c075fc41f60411f931a6e82e74eff54a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-203.ec2.internal" Sep 17 10:10:46.207116 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:10:46.207080 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-203.ec2.internal\" not found" Sep 17 10:10:46.216430 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:46.216398 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fcac5a3f4a7d9d4b6710d0af7b8c7711-config\") pod \"kube-apiserver-proxy-ip-10-0-137-203.ec2.internal\" (UID: \"fcac5a3f4a7d9d4b6710d0af7b8c7711\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-203.ec2.internal" Sep 17 10:10:46.216430 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:46.216437 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c075fc41f60411f931a6e82e74eff54a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-203.ec2.internal\" (UID: \"c075fc41f60411f931a6e82e74eff54a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-203.ec2.internal" Sep 17 10:10:46.216541 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:46.216456 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c075fc41f60411f931a6e82e74eff54a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-203.ec2.internal\" (UID: \"c075fc41f60411f931a6e82e74eff54a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-203.ec2.internal" Sep 17 10:10:46.216541 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:46.216503 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c075fc41f60411f931a6e82e74eff54a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-203.ec2.internal\" (UID: \"c075fc41f60411f931a6e82e74eff54a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-203.ec2.internal" Sep 17 10:10:46.216541 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:46.216521 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fcac5a3f4a7d9d4b6710d0af7b8c7711-config\") pod \"kube-apiserver-proxy-ip-10-0-137-203.ec2.internal\" (UID: \"fcac5a3f4a7d9d4b6710d0af7b8c7711\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-203.ec2.internal" Sep 17 10:10:46.216632 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:46.216504 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c075fc41f60411f931a6e82e74eff54a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-203.ec2.internal\" (UID: \"c075fc41f60411f931a6e82e74eff54a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-203.ec2.internal" Sep 17 10:10:46.308136 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:10:46.307897 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-203.ec2.internal\" not found" Sep 17 10:10:46.376473 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:46.376445 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-203.ec2.internal" Sep 17 10:10:46.382178 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:46.382156 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-203.ec2.internal" Sep 17 10:10:46.408800 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:10:46.408748 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-203.ec2.internal\" not found" Sep 17 10:10:46.509358 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:10:46.509320 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-203.ec2.internal\" not found" Sep 17 10:10:46.609909 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:10:46.609826 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-203.ec2.internal\" not found" Sep 17 10:10:46.688324 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:46.688291 2581 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 17 10:10:46.710827 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:10:46.710799 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-203.ec2.internal\" not found" Sep 17 10:10:46.749360 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:46.749325 2581 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 17 10:10:46.749879 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:46.749520 2581 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 17 10:10:46.749879 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:46.749521 2581 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 17 10:10:46.811791 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:10:46.811746 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-203.ec2.internal\" not found" Sep 17 10:10:46.826633 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:46.826589 2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-16 10:05:46 +0000 UTC" deadline="2028-03-03 09:40:15.693768138 +0000 UTC" Sep 17 10:10:46.826633 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:46.826630 2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12791h29m28.867143215s" Sep 17 10:10:46.889265 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:46.889182 2581 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 17 10:10:46.913619 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:46.913590 2581 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-203.ec2.internal" Sep 17 10:10:46.923160 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:46.923141 2581 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 17 10:10:46.923941 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:46.923923 2581 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-203.ec2.internal" Sep 17 10:10:46.926556 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:46.926530 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc075fc41f60411f931a6e82e74eff54a.slice/crio-3b6bd9edcd51acc4ef2522fc8832abd35ce3df6620935120f6436342a2896850 WatchSource:0}: Error finding container 3b6bd9edcd51acc4ef2522fc8832abd35ce3df6620935120f6436342a2896850: Status 404 returned error can't find the container with id 3b6bd9edcd51acc4ef2522fc8832abd35ce3df6620935120f6436342a2896850 Sep 17 10:10:46.926821 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:46.926797 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcac5a3f4a7d9d4b6710d0af7b8c7711.slice/crio-fabf36907d68c56567f2ec4ae9114b346002fd16e638f97967536bacb770e055 WatchSource:0}: Error finding container fabf36907d68c56567f2ec4ae9114b346002fd16e638f97967536bacb770e055: Status 404 returned error can't find the container with id fabf36907d68c56567f2ec4ae9114b346002fd16e638f97967536bacb770e055 Sep 17 10:10:46.931391 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:46.931378 2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 17 10:10:46.933871 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:46.933857 2581 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 17 10:10:46.942389 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:46.942334 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-203.ec2.internal" event={"ID":"c075fc41f60411f931a6e82e74eff54a","Type":"ContainerStarted","Data":"3b6bd9edcd51acc4ef2522fc8832abd35ce3df6620935120f6436342a2896850"} Sep 17 10:10:46.943403 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:46.943382 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-203.ec2.internal" event={"ID":"fcac5a3f4a7d9d4b6710d0af7b8c7711","Type":"ContainerStarted","Data":"fabf36907d68c56567f2ec4ae9114b346002fd16e638f97967536bacb770e055"} Sep 17 10:10:47.035497 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.035461 2581 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 17 10:10:47.730536 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.730495 2581 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 17 10:10:47.790046 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.790009 2581 apiserver.go:52] "Watching apiserver" Sep 17 10:10:47.800522 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.800493 2581 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 17 10:10:47.801053 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.801028 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kcztl","openshift-dns/node-resolver-7f64n","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-203.ec2.internal","openshift-multus/multus-additional-cni-plugins-4w7ff","openshift-multus/multus-nxbjh","openshift-multus/network-metrics-daemon-wfccr","kube-system/konnectivity-agent-jkx62","openshift-cluster-node-tuning-operator/tuned-kg2hz","openshift-image-registry/node-ca-r2dcq","openshift-network-diagnostics/network-check-target-fppwm","openshift-network-operator/iptables-alerter-g8lzm","openshift-ovn-kubernetes/ovnkube-node-9zbjf","kube-system/global-pull-secret-syncer-v9xpw","kube-system/kube-apiserver-proxy-ip-10-0-137-203.ec2.internal"] Sep 17 10:10:47.803452 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.803362 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-jkx62" Sep 17 10:10:47.805426 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.805238 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v9xpw" Sep 17 10:10:47.805426 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:10:47.805313 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v9xpw" podUID="c304d2b0-b1ce-4a29-86b1-341d5db70d4f" Sep 17 10:10:47.806543 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.806419 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-brc6b\"" Sep 17 10:10:47.806543 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.806428 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 17 10:10:47.806543 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.806455 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 17 10:10:47.807139 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.807119 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-kg2hz" Sep 17 10:10:47.809165 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.809147 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-4w7ff" Sep 17 10:10:47.810319 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.810296 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 17 10:10:47.810774 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.810599 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 17 10:10:47.810774 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.810655 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-lh6sl\"" Sep 17 10:10:47.811247 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.811156 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-nxbjh" Sep 17 10:10:47.813088 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.811813 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 17 10:10:47.813088 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.811965 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 17 10:10:47.813088 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.812028 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 17 10:10:47.813088 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.812227 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 17 10:10:47.813088 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.812308 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-r98jc\"" Sep 17 10:10:47.813088 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.812421 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 17 10:10:47.813673 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.813413 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wfccr" Sep 17 10:10:47.813673 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:10:47.813469 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wfccr" podUID="c84a7003-1537-4470-8094-ea10a89a86a2" Sep 17 10:10:47.813673 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.813599 2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 17 10:10:47.813859 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.813724 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-h5f7z\"" Sep 17 10:10:47.814321 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.814302 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 17 10:10:47.815518 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.815491 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-r2dcq" Sep 17 10:10:47.818373 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.817823 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kcztl" Sep 17 10:10:47.818373 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.818114 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 17 10:10:47.818373 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.818208 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 17 10:10:47.818373 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.818245 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 17 10:10:47.818754 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.818603 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-4kt2b\"" Sep 17 10:10:47.821208 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.819918 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-7f64n" Sep 17 10:10:47.821208 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.820585 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 17 10:10:47.821208 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.820849 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 17 10:10:47.821208 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.820895 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 17 10:10:47.821208 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.821057 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-qr9pv\"" Sep 17 10:10:47.822298 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.822245 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fppwm" Sep 17 10:10:47.823060 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:10:47.823022 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fppwm" podUID="f6509b04-a69d-49d5-a08c-aabdd8bf6c86" Sep 17 10:10:47.823920 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.823898 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-d48pd\"" Sep 17 10:10:47.824444 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.824423 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 17 10:10:47.825589 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.825567 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 17 10:10:47.826995 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.826965 2581 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 17 10:10:47.827224 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.827206 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e4b73c92-0ac6-4d8f-90d8-249cfbcadf4b-etc-tuned\") pod \"tuned-kg2hz\" (UID: \"e4b73c92-0ac6-4d8f-90d8-249cfbcadf4b\") " pod="openshift-cluster-node-tuning-operator/tuned-kg2hz" Sep 17 10:10:47.827293 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.827247 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a80abea0-a53c-4d9e-8934-7a7f073d71e3-system-cni-dir\") pod \"multus-additional-cni-plugins-4w7ff\" (UID: \"a80abea0-a53c-4d9e-8934-7a7f073d71e3\") " pod="openshift-multus/multus-additional-cni-plugins-4w7ff" Sep 17 10:10:47.827293 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.827279 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ae147838-ff96-4652-bf25-ca99bc25be02-multus-daemon-config\") pod \"multus-nxbjh\" (UID: \"ae147838-ff96-4652-bf25-ca99bc25be02\") " pod="openshift-multus/multus-nxbjh" Sep 17 10:10:47.827403 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.827309 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgzjj\" (UniqueName: \"kubernetes.io/projected/ae147838-ff96-4652-bf25-ca99bc25be02-kube-api-access-mgzjj\") pod \"multus-nxbjh\" (UID: \"ae147838-ff96-4652-bf25-ca99bc25be02\") " pod="openshift-multus/multus-nxbjh" Sep 17 10:10:47.827403 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.827339 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxp6q\" (UniqueName: \"kubernetes.io/projected/e4b73c92-0ac6-4d8f-90d8-249cfbcadf4b-kube-api-access-gxp6q\") pod \"tuned-kg2hz\" (UID: \"e4b73c92-0ac6-4d8f-90d8-249cfbcadf4b\") " pod="openshift-cluster-node-tuning-operator/tuned-kg2hz" Sep 17 10:10:47.827403 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.827370 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a80abea0-a53c-4d9e-8934-7a7f073d71e3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4w7ff\" (UID: \"a80abea0-a53c-4d9e-8934-7a7f073d71e3\") " pod="openshift-multus/multus-additional-cni-plugins-4w7ff" Sep 17 10:10:47.827546 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.827402 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a80abea0-a53c-4d9e-8934-7a7f073d71e3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4w7ff\" (UID: \"a80abea0-a53c-4d9e-8934-7a7f073d71e3\") " pod="openshift-multus/multus-additional-cni-plugins-4w7ff" Sep 17 10:10:47.827546 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.827437 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rn9xb\" (UniqueName: \"kubernetes.io/projected/a80abea0-a53c-4d9e-8934-7a7f073d71e3-kube-api-access-rn9xb\") pod \"multus-additional-cni-plugins-4w7ff\" (UID: \"a80abea0-a53c-4d9e-8934-7a7f073d71e3\") " pod="openshift-multus/multus-additional-cni-plugins-4w7ff" Sep 17 10:10:47.827546 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.827515 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ae147838-ff96-4652-bf25-ca99bc25be02-host-var-lib-cni-multus\") pod \"multus-nxbjh\" (UID: \"ae147838-ff96-4652-bf25-ca99bc25be02\") " pod="openshift-multus/multus-nxbjh" Sep 17 10:10:47.827691 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.827577 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ae147838-ff96-4652-bf25-ca99bc25be02-host-run-multus-certs\") pod \"multus-nxbjh\" (UID: \"ae147838-ff96-4652-bf25-ca99bc25be02\") " pod="openshift-multus/multus-nxbjh" Sep 17 10:10:47.827691 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.827608 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2c409c2a-64a4-4164-8a14-c2a87597dd93-host\") pod \"node-ca-r2dcq\" (UID: \"2c409c2a-64a4-4164-8a14-c2a87597dd93\") " pod="openshift-image-registry/node-ca-r2dcq" Sep 17 10:10:47.827691 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.827638 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e4b73c92-0ac6-4d8f-90d8-249cfbcadf4b-etc-kubernetes\") pod \"tuned-kg2hz\" (UID: \"e4b73c92-0ac6-4d8f-90d8-249cfbcadf4b\") " pod="openshift-cluster-node-tuning-operator/tuned-kg2hz" Sep 17 10:10:47.827691 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.827665 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e4b73c92-0ac6-4d8f-90d8-249cfbcadf4b-etc-sysctl-conf\") pod \"tuned-kg2hz\" (UID: \"e4b73c92-0ac6-4d8f-90d8-249cfbcadf4b\") " pod="openshift-cluster-node-tuning-operator/tuned-kg2hz" Sep 17 10:10:47.827906 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.827695 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ae147838-ff96-4652-bf25-ca99bc25be02-hostroot\") pod \"multus-nxbjh\" (UID: \"ae147838-ff96-4652-bf25-ca99bc25be02\") " pod="openshift-multus/multus-nxbjh" Sep 17 10:10:47.827906 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.827740 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ae147838-ff96-4652-bf25-ca99bc25be02-multus-conf-dir\") pod \"multus-nxbjh\" (UID: \"ae147838-ff96-4652-bf25-ca99bc25be02\") " pod="openshift-multus/multus-nxbjh" Sep 17 10:10:47.827906 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.827803 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/de933a7f-7eae-4e7b-870f-aa9c56565481-konnectivity-ca\") pod \"konnectivity-agent-jkx62\" (UID: \"de933a7f-7eae-4e7b-870f-aa9c56565481\") " pod="kube-system/konnectivity-agent-jkx62" Sep 17 10:10:47.827906 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.827839 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-g8lzm" Sep 17 10:10:47.827906 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.827847 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e4b73c92-0ac6-4d8f-90d8-249cfbcadf4b-host\") pod \"tuned-kg2hz\" (UID: \"e4b73c92-0ac6-4d8f-90d8-249cfbcadf4b\") " pod="openshift-cluster-node-tuning-operator/tuned-kg2hz" Sep 17 10:10:47.827906 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.827896 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a80abea0-a53c-4d9e-8934-7a7f073d71e3-cnibin\") pod \"multus-additional-cni-plugins-4w7ff\" (UID: \"a80abea0-a53c-4d9e-8934-7a7f073d71e3\") " pod="openshift-multus/multus-additional-cni-plugins-4w7ff" Sep 17 10:10:47.828168 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.827928 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ae147838-ff96-4652-bf25-ca99bc25be02-multus-cni-dir\") pod \"multus-nxbjh\" (UID: \"ae147838-ff96-4652-bf25-ca99bc25be02\") " pod="openshift-multus/multus-nxbjh" Sep 17 10:10:47.828168 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.827959 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ae147838-ff96-4652-bf25-ca99bc25be02-host-run-k8s-cni-cncf-io\") pod \"multus-nxbjh\" (UID: \"ae147838-ff96-4652-bf25-ca99bc25be02\") " pod="openshift-multus/multus-nxbjh" Sep 17 10:10:47.828168 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.827990 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ae147838-ff96-4652-bf25-ca99bc25be02-host-run-netns\") pod \"multus-nxbjh\" (UID: \"ae147838-ff96-4652-bf25-ca99bc25be02\") " pod="openshift-multus/multus-nxbjh" Sep 17 10:10:47.828168 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.828052 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ae147838-ff96-4652-bf25-ca99bc25be02-etc-kubernetes\") pod \"multus-nxbjh\" (UID: \"ae147838-ff96-4652-bf25-ca99bc25be02\") " pod="openshift-multus/multus-nxbjh" Sep 17 10:10:47.828168 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.828083 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c84a7003-1537-4470-8094-ea10a89a86a2-metrics-certs\") pod \"network-metrics-daemon-wfccr\" (UID: \"c84a7003-1537-4470-8094-ea10a89a86a2\") " pod="openshift-multus/network-metrics-daemon-wfccr" Sep 17 10:10:47.828168 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.828108 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2c409c2a-64a4-4164-8a14-c2a87597dd93-serviceca\") pod \"node-ca-r2dcq\" (UID: \"2c409c2a-64a4-4164-8a14-c2a87597dd93\") " pod="openshift-image-registry/node-ca-r2dcq" Sep 17 10:10:47.828168 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.828137 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e4b73c92-0ac6-4d8f-90d8-249cfbcadf4b-sys\") pod \"tuned-kg2hz\" (UID: \"e4b73c92-0ac6-4d8f-90d8-249cfbcadf4b\") " pod="openshift-cluster-node-tuning-operator/tuned-kg2hz" Sep 17 10:10:47.828562 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.828178 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c304d2b0-b1ce-4a29-86b1-341d5db70d4f-original-pull-secret\") pod \"global-pull-secret-syncer-v9xpw\" (UID: \"c304d2b0-b1ce-4a29-86b1-341d5db70d4f\") " pod="kube-system/global-pull-secret-syncer-v9xpw" Sep 17 10:10:47.828562 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.828209 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e4b73c92-0ac6-4d8f-90d8-249cfbcadf4b-etc-sysconfig\") pod \"tuned-kg2hz\" (UID: \"e4b73c92-0ac6-4d8f-90d8-249cfbcadf4b\") " pod="openshift-cluster-node-tuning-operator/tuned-kg2hz" Sep 17 10:10:47.828562 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.828272 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e4b73c92-0ac6-4d8f-90d8-249cfbcadf4b-etc-systemd\") pod \"tuned-kg2hz\" (UID: \"e4b73c92-0ac6-4d8f-90d8-249cfbcadf4b\") " pod="openshift-cluster-node-tuning-operator/tuned-kg2hz" Sep 17 10:10:47.828562 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.828378 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e4b73c92-0ac6-4d8f-90d8-249cfbcadf4b-lib-modules\") pod \"tuned-kg2hz\" (UID: \"e4b73c92-0ac6-4d8f-90d8-249cfbcadf4b\") " pod="openshift-cluster-node-tuning-operator/tuned-kg2hz" Sep 17 10:10:47.828562 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.828407 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a80abea0-a53c-4d9e-8934-7a7f073d71e3-cni-binary-copy\") pod \"multus-additional-cni-plugins-4w7ff\" (UID: \"a80abea0-a53c-4d9e-8934-7a7f073d71e3\") " pod="openshift-multus/multus-additional-cni-plugins-4w7ff" Sep 17 10:10:47.828562 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.828475 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ae147838-ff96-4652-bf25-ca99bc25be02-cnibin\") pod \"multus-nxbjh\" (UID: \"ae147838-ff96-4652-bf25-ca99bc25be02\") " pod="openshift-multus/multus-nxbjh" Sep 17 10:10:47.828562 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.828507 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ae147838-ff96-4652-bf25-ca99bc25be02-multus-socket-dir-parent\") pod \"multus-nxbjh\" (UID: \"ae147838-ff96-4652-bf25-ca99bc25be02\") " pod="openshift-multus/multus-nxbjh" Sep 17 10:10:47.828918 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.828736 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c304d2b0-b1ce-4a29-86b1-341d5db70d4f-dbus\") pod \"global-pull-secret-syncer-v9xpw\" (UID: \"c304d2b0-b1ce-4a29-86b1-341d5db70d4f\") " pod="kube-system/global-pull-secret-syncer-v9xpw" Sep 17 10:10:47.828918 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.828824 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e4b73c92-0ac6-4d8f-90d8-249cfbcadf4b-var-lib-kubelet\") pod \"tuned-kg2hz\" (UID: \"e4b73c92-0ac6-4d8f-90d8-249cfbcadf4b\") " pod="openshift-cluster-node-tuning-operator/tuned-kg2hz" Sep 17 10:10:47.828918 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.828865 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a80abea0-a53c-4d9e-8934-7a7f073d71e3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4w7ff\" (UID: \"a80abea0-a53c-4d9e-8934-7a7f073d71e3\") " pod="openshift-multus/multus-additional-cni-plugins-4w7ff" Sep 17 10:10:47.828918 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.828902 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ae147838-ff96-4652-bf25-ca99bc25be02-system-cni-dir\") pod \"multus-nxbjh\" (UID: \"ae147838-ff96-4652-bf25-ca99bc25be02\") " pod="openshift-multus/multus-nxbjh" Sep 17 10:10:47.829090 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.828933 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ae147838-ff96-4652-bf25-ca99bc25be02-host-var-lib-cni-bin\") pod \"multus-nxbjh\" (UID: \"ae147838-ff96-4652-bf25-ca99bc25be02\") " pod="openshift-multus/multus-nxbjh" Sep 17 10:10:47.829090 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.828996 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/de933a7f-7eae-4e7b-870f-aa9c56565481-agent-certs\") pod \"konnectivity-agent-jkx62\" (UID: \"de933a7f-7eae-4e7b-870f-aa9c56565481\") " pod="kube-system/konnectivity-agent-jkx62" Sep 17 10:10:47.829090 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.829023 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c304d2b0-b1ce-4a29-86b1-341d5db70d4f-kubelet-config\") pod \"global-pull-secret-syncer-v9xpw\" (UID: \"c304d2b0-b1ce-4a29-86b1-341d5db70d4f\") " pod="kube-system/global-pull-secret-syncer-v9xpw" Sep 17 10:10:47.829090 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.829056 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e4b73c92-0ac6-4d8f-90d8-249cfbcadf4b-etc-modprobe-d\") pod \"tuned-kg2hz\" (UID: \"e4b73c92-0ac6-4d8f-90d8-249cfbcadf4b\") " pod="openshift-cluster-node-tuning-operator/tuned-kg2hz" Sep 17 10:10:47.829090 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.829088 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ae147838-ff96-4652-bf25-ca99bc25be02-cni-binary-copy\") pod \"multus-nxbjh\" (UID: \"ae147838-ff96-4652-bf25-ca99bc25be02\") " pod="openshift-multus/multus-nxbjh" Sep 17 10:10:47.829311 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.829121 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzz8c\" (UniqueName: \"kubernetes.io/projected/c84a7003-1537-4470-8094-ea10a89a86a2-kube-api-access-kzz8c\") pod \"network-metrics-daemon-wfccr\" (UID: \"c84a7003-1537-4470-8094-ea10a89a86a2\") " pod="openshift-multus/network-metrics-daemon-wfccr" Sep 17 10:10:47.829311 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.829149 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qchf4\" (UniqueName: \"kubernetes.io/projected/2c409c2a-64a4-4164-8a14-c2a87597dd93-kube-api-access-qchf4\") pod \"node-ca-r2dcq\" (UID: \"2c409c2a-64a4-4164-8a14-c2a87597dd93\") " pod="openshift-image-registry/node-ca-r2dcq" Sep 17 10:10:47.829311 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.829178 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e4b73c92-0ac6-4d8f-90d8-249cfbcadf4b-etc-sysctl-d\") pod \"tuned-kg2hz\" (UID: \"e4b73c92-0ac6-4d8f-90d8-249cfbcadf4b\") " pod="openshift-cluster-node-tuning-operator/tuned-kg2hz" Sep 17 10:10:47.829311 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.829220 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e4b73c92-0ac6-4d8f-90d8-249cfbcadf4b-tmp\") pod \"tuned-kg2hz\" (UID: \"e4b73c92-0ac6-4d8f-90d8-249cfbcadf4b\") " pod="openshift-cluster-node-tuning-operator/tuned-kg2hz" Sep 17 10:10:47.829311 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.829251 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ae147838-ff96-4652-bf25-ca99bc25be02-host-var-lib-kubelet\") pod \"multus-nxbjh\" (UID: \"ae147838-ff96-4652-bf25-ca99bc25be02\") " pod="openshift-multus/multus-nxbjh" Sep 17 10:10:47.829311 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.829277 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e4b73c92-0ac6-4d8f-90d8-249cfbcadf4b-run\") pod \"tuned-kg2hz\" (UID: \"e4b73c92-0ac6-4d8f-90d8-249cfbcadf4b\") " pod="openshift-cluster-node-tuning-operator/tuned-kg2hz" Sep 17 10:10:47.830099 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.830076 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-9zbjf" Sep 17 10:10:47.830750 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.830246 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 17 10:10:47.830970 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.830949 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-sz2qm\"" Sep 17 10:10:47.831063 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.830983 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 17 10:10:47.831392 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.831374 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 17 10:10:47.833377 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.833354 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 17 10:10:47.835192 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.834900 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 17 10:10:47.835192 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.834916 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 17 10:10:47.835192 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.834956 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-4vgrg\"" Sep 17 10:10:47.835192 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.834906 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 17 10:10:47.835192 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.835072 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 17 10:10:47.835192 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.835136 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 17 10:10:47.850158 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.850137 2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-d5h8v" Sep 17 10:10:47.859377 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.859353 2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-d5h8v" Sep 17 10:10:47.916050 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.916028 2581 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 17 10:10:47.929603 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.929572 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a80abea0-a53c-4d9e-8934-7a7f073d71e3-cni-binary-copy\") pod \"multus-additional-cni-plugins-4w7ff\" (UID: \"a80abea0-a53c-4d9e-8934-7a7f073d71e3\") " pod="openshift-multus/multus-additional-cni-plugins-4w7ff" Sep 17 10:10:47.929603 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.929608 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ae147838-ff96-4652-bf25-ca99bc25be02-multus-socket-dir-parent\") pod \"multus-nxbjh\" (UID: \"ae147838-ff96-4652-bf25-ca99bc25be02\") " pod="openshift-multus/multus-nxbjh" Sep 17 10:10:47.929878 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.929631 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/de933a7f-7eae-4e7b-870f-aa9c56565481-agent-certs\") pod \"konnectivity-agent-jkx62\" (UID: \"de933a7f-7eae-4e7b-870f-aa9c56565481\") " pod="kube-system/konnectivity-agent-jkx62" Sep 17 10:10:47.929878 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.929662 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/606f20f7-c795-451a-b146-03f3a4f92684-etc-selinux\") pod \"aws-ebs-csi-driver-node-kcztl\" (UID: \"606f20f7-c795-451a-b146-03f3a4f92684\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kcztl" Sep 17 10:10:47.929878 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.929687 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b506216d-c159-420e-9111-56acab7568df-host-slash\") pod \"iptables-alerter-g8lzm\" (UID: \"b506216d-c159-420e-9111-56acab7568df\") " pod="openshift-network-operator/iptables-alerter-g8lzm" Sep 17 10:10:47.929878 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.929806 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c304d2b0-b1ce-4a29-86b1-341d5db70d4f-dbus\") pod \"global-pull-secret-syncer-v9xpw\" (UID: \"c304d2b0-b1ce-4a29-86b1-341d5db70d4f\") " pod="kube-system/global-pull-secret-syncer-v9xpw" Sep 17 10:10:47.929878 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.929849 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ae147838-ff96-4652-bf25-ca99bc25be02-host-var-lib-cni-bin\") pod \"multus-nxbjh\" (UID: \"ae147838-ff96-4652-bf25-ca99bc25be02\") " pod="openshift-multus/multus-nxbjh" Sep 17 10:10:47.929878 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.929856 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ae147838-ff96-4652-bf25-ca99bc25be02-multus-socket-dir-parent\") pod \"multus-nxbjh\" (UID: \"ae147838-ff96-4652-bf25-ca99bc25be02\") " pod="openshift-multus/multus-nxbjh" Sep 17 10:10:47.930164 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.929881 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cnjs\" (UniqueName: \"kubernetes.io/projected/b506216d-c159-420e-9111-56acab7568df-kube-api-access-6cnjs\") pod \"iptables-alerter-g8lzm\" (UID: \"b506216d-c159-420e-9111-56acab7568df\") " pod="openshift-network-operator/iptables-alerter-g8lzm" Sep 17 10:10:47.930164 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.929894 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ae147838-ff96-4652-bf25-ca99bc25be02-host-var-lib-cni-bin\") pod \"multus-nxbjh\" (UID: \"ae147838-ff96-4652-bf25-ca99bc25be02\") " pod="openshift-multus/multus-nxbjh" Sep 17 10:10:47.930164 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.929908 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c304d2b0-b1ce-4a29-86b1-341d5db70d4f-kubelet-config\") pod \"global-pull-secret-syncer-v9xpw\" (UID: \"c304d2b0-b1ce-4a29-86b1-341d5db70d4f\") " pod="kube-system/global-pull-secret-syncer-v9xpw" Sep 17 10:10:47.930164 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.929934 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ae147838-ff96-4652-bf25-ca99bc25be02-cni-binary-copy\") pod \"multus-nxbjh\" (UID: \"ae147838-ff96-4652-bf25-ca99bc25be02\") " pod="openshift-multus/multus-nxbjh" Sep 17 10:10:47.930164 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.929957 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzz8c\" (UniqueName: \"kubernetes.io/projected/c84a7003-1537-4470-8094-ea10a89a86a2-kube-api-access-kzz8c\") pod \"network-metrics-daemon-wfccr\" (UID: \"c84a7003-1537-4470-8094-ea10a89a86a2\") " pod="openshift-multus/network-metrics-daemon-wfccr" Sep 17 10:10:47.930164 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.929997 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qchf4\" (UniqueName: \"kubernetes.io/projected/2c409c2a-64a4-4164-8a14-c2a87597dd93-kube-api-access-qchf4\") pod \"node-ca-r2dcq\" (UID: \"2c409c2a-64a4-4164-8a14-c2a87597dd93\") " pod="openshift-image-registry/node-ca-r2dcq" Sep 17 10:10:47.930164 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.930005 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c304d2b0-b1ce-4a29-86b1-341d5db70d4f-dbus\") pod \"global-pull-secret-syncer-v9xpw\" (UID: \"c304d2b0-b1ce-4a29-86b1-341d5db70d4f\") " pod="kube-system/global-pull-secret-syncer-v9xpw" Sep 17 10:10:47.930164 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.930023 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/606f20f7-c795-451a-b146-03f3a4f92684-sys-fs\") pod \"aws-ebs-csi-driver-node-kcztl\" (UID: \"606f20f7-c795-451a-b146-03f3a4f92684\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kcztl" Sep 17 10:10:47.930164 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.930022 2581 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 17 10:10:47.930660 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.930000 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c304d2b0-b1ce-4a29-86b1-341d5db70d4f-kubelet-config\") pod \"global-pull-secret-syncer-v9xpw\" (UID: \"c304d2b0-b1ce-4a29-86b1-341d5db70d4f\") " pod="kube-system/global-pull-secret-syncer-v9xpw" Sep 17 10:10:47.930660 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.930049 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f7323dd-d769-4f58-92d7-e8a36c2f2854-tmp-dir\") pod \"node-resolver-7f64n\" (UID: \"6f7323dd-d769-4f58-92d7-e8a36c2f2854\") " pod="openshift-dns/node-resolver-7f64n" Sep 17 10:10:47.930660 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.930226 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/896c7cf8-3f20-4809-8f6d-a6852a068ad2-host-cni-netd\") pod \"ovnkube-node-9zbjf\" (UID: \"896c7cf8-3f20-4809-8f6d-a6852a068ad2\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zbjf" Sep 17 10:10:47.930660 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.930255 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/896c7cf8-3f20-4809-8f6d-a6852a068ad2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9zbjf\" (UID: \"896c7cf8-3f20-4809-8f6d-a6852a068ad2\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zbjf" Sep 17 10:10:47.930660 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.930282 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e4b73c92-0ac6-4d8f-90d8-249cfbcadf4b-etc-sysctl-d\") pod \"tuned-kg2hz\" (UID: \"e4b73c92-0ac6-4d8f-90d8-249cfbcadf4b\") " pod="openshift-cluster-node-tuning-operator/tuned-kg2hz" Sep 17 10:10:47.930660 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.930305 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e4b73c92-0ac6-4d8f-90d8-249cfbcadf4b-tmp\") pod \"tuned-kg2hz\" (UID: \"e4b73c92-0ac6-4d8f-90d8-249cfbcadf4b\") " pod="openshift-cluster-node-tuning-operator/tuned-kg2hz" Sep 17 10:10:47.930660 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.930328 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ae147838-ff96-4652-bf25-ca99bc25be02-host-var-lib-kubelet\") pod \"multus-nxbjh\" (UID: \"ae147838-ff96-4652-bf25-ca99bc25be02\") " pod="openshift-multus/multus-nxbjh" Sep 17 10:10:47.930660 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.930352 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/896c7cf8-3f20-4809-8f6d-a6852a068ad2-node-log\") pod \"ovnkube-node-9zbjf\" (UID: \"896c7cf8-3f20-4809-8f6d-a6852a068ad2\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zbjf" Sep 17 10:10:47.930660 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.930383 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/896c7cf8-3f20-4809-8f6d-a6852a068ad2-host-cni-bin\") pod \"ovnkube-node-9zbjf\" (UID: \"896c7cf8-3f20-4809-8f6d-a6852a068ad2\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zbjf" Sep 17 10:10:47.930660 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.930433 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/896c7cf8-3f20-4809-8f6d-a6852a068ad2-ovnkube-config\") pod \"ovnkube-node-9zbjf\" (UID: \"896c7cf8-3f20-4809-8f6d-a6852a068ad2\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zbjf" Sep 17 10:10:47.930660 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.930448 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ae147838-ff96-4652-bf25-ca99bc25be02-host-var-lib-kubelet\") pod \"multus-nxbjh\" (UID: \"ae147838-ff96-4652-bf25-ca99bc25be02\") " pod="openshift-multus/multus-nxbjh" Sep 17 10:10:47.930660 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.930459 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wj8z\" (UniqueName: \"kubernetes.io/projected/896c7cf8-3f20-4809-8f6d-a6852a068ad2-kube-api-access-2wj8z\") pod \"ovnkube-node-9zbjf\" (UID: \"896c7cf8-3f20-4809-8f6d-a6852a068ad2\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zbjf" Sep 17 10:10:47.930660 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.930462 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e4b73c92-0ac6-4d8f-90d8-249cfbcadf4b-etc-sysctl-d\") pod \"tuned-kg2hz\" (UID: \"e4b73c92-0ac6-4d8f-90d8-249cfbcadf4b\") " pod="openshift-cluster-node-tuning-operator/tuned-kg2hz" Sep 17 10:10:47.930660 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.930484 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e4b73c92-0ac6-4d8f-90d8-249cfbcadf4b-run\") pod \"tuned-kg2hz\" (UID: \"e4b73c92-0ac6-4d8f-90d8-249cfbcadf4b\") " pod="openshift-cluster-node-tuning-operator/tuned-kg2hz" Sep 17 10:10:47.930660 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.930504 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ae147838-ff96-4652-bf25-ca99bc25be02-cni-binary-copy\") pod \"multus-nxbjh\" (UID: \"ae147838-ff96-4652-bf25-ca99bc25be02\") " pod="openshift-multus/multus-nxbjh" Sep 17 10:10:47.930660 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.930522 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a80abea0-a53c-4d9e-8934-7a7f073d71e3-system-cni-dir\") pod \"multus-additional-cni-plugins-4w7ff\" (UID: \"a80abea0-a53c-4d9e-8934-7a7f073d71e3\") " pod="openshift-multus/multus-additional-cni-plugins-4w7ff" Sep 17 10:10:47.930660 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.930550 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e4b73c92-0ac6-4d8f-90d8-249cfbcadf4b-run\") pod \"tuned-kg2hz\" (UID: \"e4b73c92-0ac6-4d8f-90d8-249cfbcadf4b\") " pod="openshift-cluster-node-tuning-operator/tuned-kg2hz" Sep 17 10:10:47.931373 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.930551 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ae147838-ff96-4652-bf25-ca99bc25be02-multus-daemon-config\") pod \"multus-nxbjh\" (UID: \"ae147838-ff96-4652-bf25-ca99bc25be02\") " pod="openshift-multus/multus-nxbjh" Sep 17 10:10:47.931373 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.930580 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a80abea0-a53c-4d9e-8934-7a7f073d71e3-system-cni-dir\") pod \"multus-additional-cni-plugins-4w7ff\" (UID: \"a80abea0-a53c-4d9e-8934-7a7f073d71e3\") " pod="openshift-multus/multus-additional-cni-plugins-4w7ff" Sep 17 10:10:47.931373 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.930588 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgzjj\" (UniqueName: \"kubernetes.io/projected/ae147838-ff96-4652-bf25-ca99bc25be02-kube-api-access-mgzjj\") pod \"multus-nxbjh\" (UID: \"ae147838-ff96-4652-bf25-ca99bc25be02\") " pod="openshift-multus/multus-nxbjh" Sep 17 10:10:47.931373 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.930612 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/896c7cf8-3f20-4809-8f6d-a6852a068ad2-host-slash\") pod \"ovnkube-node-9zbjf\" (UID: \"896c7cf8-3f20-4809-8f6d-a6852a068ad2\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zbjf" Sep 17 10:10:47.931373 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.930633 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/896c7cf8-3f20-4809-8f6d-a6852a068ad2-run-systemd\") pod \"ovnkube-node-9zbjf\" (UID: \"896c7cf8-3f20-4809-8f6d-a6852a068ad2\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zbjf" Sep 17 10:10:47.931373 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.930652 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/896c7cf8-3f20-4809-8f6d-a6852a068ad2-var-lib-openvswitch\") pod \"ovnkube-node-9zbjf\" (UID: \"896c7cf8-3f20-4809-8f6d-a6852a068ad2\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zbjf" Sep 17 10:10:47.931373 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.930684 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a80abea0-a53c-4d9e-8934-7a7f073d71e3-cni-binary-copy\") pod \"multus-additional-cni-plugins-4w7ff\" (UID: \"a80abea0-a53c-4d9e-8934-7a7f073d71e3\") " pod="openshift-multus/multus-additional-cni-plugins-4w7ff" Sep 17 10:10:47.931373 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.930694 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxp6q\" (UniqueName: \"kubernetes.io/projected/e4b73c92-0ac6-4d8f-90d8-249cfbcadf4b-kube-api-access-gxp6q\") pod \"tuned-kg2hz\" (UID: \"e4b73c92-0ac6-4d8f-90d8-249cfbcadf4b\") " pod="openshift-cluster-node-tuning-operator/tuned-kg2hz" Sep 17 10:10:47.931373 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.930811 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a80abea0-a53c-4d9e-8934-7a7f073d71e3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4w7ff\" (UID: \"a80abea0-a53c-4d9e-8934-7a7f073d71e3\") " pod="openshift-multus/multus-additional-cni-plugins-4w7ff" Sep 17 10:10:47.931373 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.930865 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2c409c2a-64a4-4164-8a14-c2a87597dd93-host\") pod \"node-ca-r2dcq\" (UID: \"2c409c2a-64a4-4164-8a14-c2a87597dd93\") " pod="openshift-image-registry/node-ca-r2dcq" Sep 17 10:10:47.931373 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.930885 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e4b73c92-0ac6-4d8f-90d8-249cfbcadf4b-etc-kubernetes\") pod \"tuned-kg2hz\" (UID: \"e4b73c92-0ac6-4d8f-90d8-249cfbcadf4b\") " pod="openshift-cluster-node-tuning-operator/tuned-kg2hz" Sep 17 10:10:47.931373 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.930908 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/de933a7f-7eae-4e7b-870f-aa9c56565481-konnectivity-ca\") pod \"konnectivity-agent-jkx62\" (UID: \"de933a7f-7eae-4e7b-870f-aa9c56565481\") " pod="kube-system/konnectivity-agent-jkx62" Sep 17 10:10:47.931373 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.930930 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2c409c2a-64a4-4164-8a14-c2a87597dd93-host\") pod \"node-ca-r2dcq\" (UID: \"2c409c2a-64a4-4164-8a14-c2a87597dd93\") " pod="openshift-image-registry/node-ca-r2dcq" Sep 17 10:10:47.931373 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.930995 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e4b73c92-0ac6-4d8f-90d8-249cfbcadf4b-etc-kubernetes\") pod \"tuned-kg2hz\" (UID: \"e4b73c92-0ac6-4d8f-90d8-249cfbcadf4b\") " pod="openshift-cluster-node-tuning-operator/tuned-kg2hz" Sep 17 10:10:47.931373 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.931011 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a80abea0-a53c-4d9e-8934-7a7f073d71e3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4w7ff\" (UID: \"a80abea0-a53c-4d9e-8934-7a7f073d71e3\") " pod="openshift-multus/multus-additional-cni-plugins-4w7ff" Sep 17 10:10:47.931373 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.931032 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ae147838-ff96-4652-bf25-ca99bc25be02-multus-daemon-config\") pod \"multus-nxbjh\" (UID: \"ae147838-ff96-4652-bf25-ca99bc25be02\") " pod="openshift-multus/multus-nxbjh" Sep 17 10:10:47.932164 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.931443 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e4b73c92-0ac6-4d8f-90d8-249cfbcadf4b-host\") pod \"tuned-kg2hz\" (UID: \"e4b73c92-0ac6-4d8f-90d8-249cfbcadf4b\") " pod="openshift-cluster-node-tuning-operator/tuned-kg2hz" Sep 17 10:10:47.932164 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.931475 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a80abea0-a53c-4d9e-8934-7a7f073d71e3-cnibin\") pod \"multus-additional-cni-plugins-4w7ff\" (UID: \"a80abea0-a53c-4d9e-8934-7a7f073d71e3\") " pod="openshift-multus/multus-additional-cni-plugins-4w7ff" Sep 17 10:10:47.932164 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.931506 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ae147838-ff96-4652-bf25-ca99bc25be02-multus-cni-dir\") pod \"multus-nxbjh\" (UID: \"ae147838-ff96-4652-bf25-ca99bc25be02\") " pod="openshift-multus/multus-nxbjh" Sep 17 10:10:47.932164 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.931535 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ae147838-ff96-4652-bf25-ca99bc25be02-host-run-netns\") pod \"multus-nxbjh\" (UID: \"ae147838-ff96-4652-bf25-ca99bc25be02\") " pod="openshift-multus/multus-nxbjh" Sep 17 10:10:47.932164 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.931561 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ae147838-ff96-4652-bf25-ca99bc25be02-etc-kubernetes\") pod \"multus-nxbjh\" (UID: \"ae147838-ff96-4652-bf25-ca99bc25be02\") " pod="openshift-multus/multus-nxbjh" Sep 17 10:10:47.932164 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.931596 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c84a7003-1537-4470-8094-ea10a89a86a2-metrics-certs\") pod \"network-metrics-daemon-wfccr\" (UID: \"c84a7003-1537-4470-8094-ea10a89a86a2\") " pod="openshift-multus/network-metrics-daemon-wfccr" Sep 17 10:10:47.932164 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.931627 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/896c7cf8-3f20-4809-8f6d-a6852a068ad2-host-run-netns\") pod \"ovnkube-node-9zbjf\" (UID: \"896c7cf8-3f20-4809-8f6d-a6852a068ad2\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zbjf" Sep 17 10:10:47.932164 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.931663 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e4b73c92-0ac6-4d8f-90d8-249cfbcadf4b-host\") pod \"tuned-kg2hz\" (UID: \"e4b73c92-0ac6-4d8f-90d8-249cfbcadf4b\") " pod="openshift-cluster-node-tuning-operator/tuned-kg2hz" Sep 17 10:10:47.932164 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.931674 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e4b73c92-0ac6-4d8f-90d8-249cfbcadf4b-sys\") pod \"tuned-kg2hz\" (UID: \"e4b73c92-0ac6-4d8f-90d8-249cfbcadf4b\") " pod="openshift-cluster-node-tuning-operator/tuned-kg2hz" Sep 17 10:10:47.932164 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.931718 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a80abea0-a53c-4d9e-8934-7a7f073d71e3-cnibin\") pod \"multus-additional-cni-plugins-4w7ff\" (UID: \"a80abea0-a53c-4d9e-8934-7a7f073d71e3\") " pod="openshift-multus/multus-additional-cni-plugins-4w7ff" Sep 17 10:10:47.932164 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.931746 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c304d2b0-b1ce-4a29-86b1-341d5db70d4f-original-pull-secret\") pod \"global-pull-secret-syncer-v9xpw\" (UID: \"c304d2b0-b1ce-4a29-86b1-341d5db70d4f\") " pod="kube-system/global-pull-secret-syncer-v9xpw" Sep 17 10:10:47.932164 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.931793 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e4b73c92-0ac6-4d8f-90d8-249cfbcadf4b-etc-sysconfig\") pod \"tuned-kg2hz\" (UID: \"e4b73c92-0ac6-4d8f-90d8-249cfbcadf4b\") " pod="openshift-cluster-node-tuning-operator/tuned-kg2hz" Sep 17 10:10:47.932164 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.931804 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ae147838-ff96-4652-bf25-ca99bc25be02-host-run-netns\") pod \"multus-nxbjh\" (UID: \"ae147838-ff96-4652-bf25-ca99bc25be02\") " pod="openshift-multus/multus-nxbjh" Sep 17 10:10:47.932164 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.931817 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e4b73c92-0ac6-4d8f-90d8-249cfbcadf4b-lib-modules\") pod \"tuned-kg2hz\" (UID: \"e4b73c92-0ac6-4d8f-90d8-249cfbcadf4b\") " pod="openshift-cluster-node-tuning-operator/tuned-kg2hz" Sep 17 10:10:47.932164 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.931848 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ae147838-ff96-4652-bf25-ca99bc25be02-etc-kubernetes\") pod \"multus-nxbjh\" (UID: \"ae147838-ff96-4652-bf25-ca99bc25be02\") " pod="openshift-multus/multus-nxbjh" Sep 17 10:10:47.932164 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.931868 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ae147838-ff96-4652-bf25-ca99bc25be02-cnibin\") pod \"multus-nxbjh\" (UID: \"ae147838-ff96-4652-bf25-ca99bc25be02\") " pod="openshift-multus/multus-nxbjh" Sep 17 10:10:47.932164 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.931895 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxnn2\" (UniqueName: \"kubernetes.io/projected/f6509b04-a69d-49d5-a08c-aabdd8bf6c86-kube-api-access-nxnn2\") pod \"network-check-target-fppwm\" (UID: \"f6509b04-a69d-49d5-a08c-aabdd8bf6c86\") " pod="openshift-network-diagnostics/network-check-target-fppwm" Sep 17 10:10:47.932164 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.931921 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/896c7cf8-3f20-4809-8f6d-a6852a068ad2-host-kubelet\") pod \"ovnkube-node-9zbjf\" (UID: \"896c7cf8-3f20-4809-8f6d-a6852a068ad2\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zbjf" Sep 17 10:10:47.933052 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:10:47.931936 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 10:10:47.933052 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.931949 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e4b73c92-0ac6-4d8f-90d8-249cfbcadf4b-var-lib-kubelet\") pod \"tuned-kg2hz\" (UID: \"e4b73c92-0ac6-4d8f-90d8-249cfbcadf4b\") " pod="openshift-cluster-node-tuning-operator/tuned-kg2hz" Sep 17 10:10:47.933052 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.931972 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a80abea0-a53c-4d9e-8934-7a7f073d71e3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4w7ff\" (UID: \"a80abea0-a53c-4d9e-8934-7a7f073d71e3\") " pod="openshift-multus/multus-additional-cni-plugins-4w7ff" Sep 17 10:10:47.933052 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:10:47.932008 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c84a7003-1537-4470-8094-ea10a89a86a2-metrics-certs podName:c84a7003-1537-4470-8094-ea10a89a86a2 nodeName:}" failed. No retries permitted until 2026-09-17 10:10:48.431975107 +0000 UTC m=+2.980532218 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c84a7003-1537-4470-8094-ea10a89a86a2-metrics-certs") pod "network-metrics-daemon-wfccr" (UID: "c84a7003-1537-4470-8094-ea10a89a86a2") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 10:10:47.933052 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.932037 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ae147838-ff96-4652-bf25-ca99bc25be02-system-cni-dir\") pod \"multus-nxbjh\" (UID: \"ae147838-ff96-4652-bf25-ca99bc25be02\") " pod="openshift-multus/multus-nxbjh" Sep 17 10:10:47.933052 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.932080 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2x84p\" (UniqueName: \"kubernetes.io/projected/606f20f7-c795-451a-b146-03f3a4f92684-kube-api-access-2x84p\") pod \"aws-ebs-csi-driver-node-kcztl\" (UID: \"606f20f7-c795-451a-b146-03f3a4f92684\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kcztl" Sep 17 10:10:47.933052 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.932104 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/896c7cf8-3f20-4809-8f6d-a6852a068ad2-systemd-units\") pod \"ovnkube-node-9zbjf\" (UID: \"896c7cf8-3f20-4809-8f6d-a6852a068ad2\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zbjf" Sep 17 10:10:47.933052 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.932118 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/de933a7f-7eae-4e7b-870f-aa9c56565481-konnectivity-ca\") pod \"konnectivity-agent-jkx62\" (UID: \"de933a7f-7eae-4e7b-870f-aa9c56565481\") " pod="kube-system/konnectivity-agent-jkx62" Sep 17 10:10:47.933052 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.932129 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e4b73c92-0ac6-4d8f-90d8-249cfbcadf4b-etc-modprobe-d\") pod \"tuned-kg2hz\" (UID: \"e4b73c92-0ac6-4d8f-90d8-249cfbcadf4b\") " pod="openshift-cluster-node-tuning-operator/tuned-kg2hz" Sep 17 10:10:47.933052 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.932152 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/606f20f7-c795-451a-b146-03f3a4f92684-device-dir\") pod \"aws-ebs-csi-driver-node-kcztl\" (UID: \"606f20f7-c795-451a-b146-03f3a4f92684\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kcztl" Sep 17 10:10:47.933052 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.932176 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/896c7cf8-3f20-4809-8f6d-a6852a068ad2-host-run-ovn-kubernetes\") pod \"ovnkube-node-9zbjf\" (UID: \"896c7cf8-3f20-4809-8f6d-a6852a068ad2\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zbjf" Sep 17 10:10:47.933052 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.932196 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/896c7cf8-3f20-4809-8f6d-a6852a068ad2-ovn-node-metrics-cert\") pod \"ovnkube-node-9zbjf\" (UID: \"896c7cf8-3f20-4809-8f6d-a6852a068ad2\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zbjf" Sep 17 10:10:47.933052 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.932217 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/896c7cf8-3f20-4809-8f6d-a6852a068ad2-ovnkube-script-lib\") pod \"ovnkube-node-9zbjf\" (UID: \"896c7cf8-3f20-4809-8f6d-a6852a068ad2\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zbjf" Sep 17 10:10:47.933052 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.932263 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e4b73c92-0ac6-4d8f-90d8-249cfbcadf4b-etc-tuned\") pod \"tuned-kg2hz\" (UID: \"e4b73c92-0ac6-4d8f-90d8-249cfbcadf4b\") " pod="openshift-cluster-node-tuning-operator/tuned-kg2hz" Sep 17 10:10:47.933052 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.932285 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/606f20f7-c795-451a-b146-03f3a4f92684-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kcztl\" (UID: \"606f20f7-c795-451a-b146-03f3a4f92684\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kcztl" Sep 17 10:10:47.933052 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.932309 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6f7323dd-d769-4f58-92d7-e8a36c2f2854-hosts-file\") pod \"node-resolver-7f64n\" (UID: \"6f7323dd-d769-4f58-92d7-e8a36c2f2854\") " pod="openshift-dns/node-resolver-7f64n" Sep 17 10:10:47.933795 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.932330 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/896c7cf8-3f20-4809-8f6d-a6852a068ad2-run-ovn\") pod \"ovnkube-node-9zbjf\" (UID: \"896c7cf8-3f20-4809-8f6d-a6852a068ad2\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zbjf" Sep 17 10:10:47.933795 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.932349 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/896c7cf8-3f20-4809-8f6d-a6852a068ad2-env-overrides\") pod \"ovnkube-node-9zbjf\" (UID: \"896c7cf8-3f20-4809-8f6d-a6852a068ad2\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zbjf" Sep 17 10:10:47.933795 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.932375 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a80abea0-a53c-4d9e-8934-7a7f073d71e3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4w7ff\" (UID: \"a80abea0-a53c-4d9e-8934-7a7f073d71e3\") " pod="openshift-multus/multus-additional-cni-plugins-4w7ff" Sep 17 10:10:47.933795 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.932403 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rn9xb\" (UniqueName: \"kubernetes.io/projected/a80abea0-a53c-4d9e-8934-7a7f073d71e3-kube-api-access-rn9xb\") pod \"multus-additional-cni-plugins-4w7ff\" (UID: \"a80abea0-a53c-4d9e-8934-7a7f073d71e3\") " pod="openshift-multus/multus-additional-cni-plugins-4w7ff" Sep 17 10:10:47.933795 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.932428 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ae147838-ff96-4652-bf25-ca99bc25be02-host-var-lib-cni-multus\") pod \"multus-nxbjh\" (UID: \"ae147838-ff96-4652-bf25-ca99bc25be02\") " pod="openshift-multus/multus-nxbjh" Sep 17 10:10:47.933795 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.932442 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a80abea0-a53c-4d9e-8934-7a7f073d71e3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4w7ff\" (UID: \"a80abea0-a53c-4d9e-8934-7a7f073d71e3\") " pod="openshift-multus/multus-additional-cni-plugins-4w7ff" Sep 17 10:10:47.933795 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.932451 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ae147838-ff96-4652-bf25-ca99bc25be02-multus-cni-dir\") pod \"multus-nxbjh\" (UID: \"ae147838-ff96-4652-bf25-ca99bc25be02\") " pod="openshift-multus/multus-nxbjh" Sep 17 10:10:47.933795 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.932453 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ae147838-ff96-4652-bf25-ca99bc25be02-host-run-multus-certs\") pod \"multus-nxbjh\" (UID: \"ae147838-ff96-4652-bf25-ca99bc25be02\") " pod="openshift-multus/multus-nxbjh" Sep 17 10:10:47.933795 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.932487 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ae147838-ff96-4652-bf25-ca99bc25be02-host-run-multus-certs\") pod \"multus-nxbjh\" (UID: \"ae147838-ff96-4652-bf25-ca99bc25be02\") " pod="openshift-multus/multus-nxbjh" Sep 17 10:10:47.933795 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.932495 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njprp\" (UniqueName: \"kubernetes.io/projected/6f7323dd-d769-4f58-92d7-e8a36c2f2854-kube-api-access-njprp\") pod \"node-resolver-7f64n\" (UID: \"6f7323dd-d769-4f58-92d7-e8a36c2f2854\") " pod="openshift-dns/node-resolver-7f64n" Sep 17 10:10:47.933795 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.931736 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e4b73c92-0ac6-4d8f-90d8-249cfbcadf4b-sys\") pod \"tuned-kg2hz\" (UID: \"e4b73c92-0ac6-4d8f-90d8-249cfbcadf4b\") " pod="openshift-cluster-node-tuning-operator/tuned-kg2hz" Sep 17 10:10:47.933795 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.932568 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e4b73c92-0ac6-4d8f-90d8-249cfbcadf4b-etc-sysconfig\") pod \"tuned-kg2hz\" (UID: \"e4b73c92-0ac6-4d8f-90d8-249cfbcadf4b\") " pod="openshift-cluster-node-tuning-operator/tuned-kg2hz" Sep 17 10:10:47.933795 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:10:47.932586 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 17 10:10:47.933795 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:10:47.932636 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c304d2b0-b1ce-4a29-86b1-341d5db70d4f-original-pull-secret podName:c304d2b0-b1ce-4a29-86b1-341d5db70d4f nodeName:}" failed. No retries permitted until 2026-09-17 10:10:48.432622312 +0000 UTC m=+2.981179400 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c304d2b0-b1ce-4a29-86b1-341d5db70d4f-original-pull-secret") pod "global-pull-secret-syncer-v9xpw" (UID: "c304d2b0-b1ce-4a29-86b1-341d5db70d4f") : object "kube-system"/"original-pull-secret" not registered Sep 17 10:10:47.933795 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.932649 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e4b73c92-0ac6-4d8f-90d8-249cfbcadf4b-lib-modules\") pod \"tuned-kg2hz\" (UID: \"e4b73c92-0ac6-4d8f-90d8-249cfbcadf4b\") " pod="openshift-cluster-node-tuning-operator/tuned-kg2hz" Sep 17 10:10:47.933795 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.932697 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ae147838-ff96-4652-bf25-ca99bc25be02-host-var-lib-cni-multus\") pod \"multus-nxbjh\" (UID: \"ae147838-ff96-4652-bf25-ca99bc25be02\") " pod="openshift-multus/multus-nxbjh" Sep 17 10:10:47.933795 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.932699 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ae147838-ff96-4652-bf25-ca99bc25be02-cnibin\") pod \"multus-nxbjh\" (UID: \"ae147838-ff96-4652-bf25-ca99bc25be02\") " pod="openshift-multus/multus-nxbjh" Sep 17 10:10:47.934650 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.932650 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ae147838-ff96-4652-bf25-ca99bc25be02-system-cni-dir\") pod \"multus-nxbjh\" (UID: \"ae147838-ff96-4652-bf25-ca99bc25be02\") " pod="openshift-multus/multus-nxbjh" Sep 17 10:10:47.934650 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.932743 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e4b73c92-0ac6-4d8f-90d8-249cfbcadf4b-etc-modprobe-d\") pod \"tuned-kg2hz\" (UID: \"e4b73c92-0ac6-4d8f-90d8-249cfbcadf4b\") " pod="openshift-cluster-node-tuning-operator/tuned-kg2hz" Sep 17 10:10:47.934650 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.933005 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e4b73c92-0ac6-4d8f-90d8-249cfbcadf4b-var-lib-kubelet\") pod \"tuned-kg2hz\" (UID: \"e4b73c92-0ac6-4d8f-90d8-249cfbcadf4b\") " pod="openshift-cluster-node-tuning-operator/tuned-kg2hz" Sep 17 10:10:47.934650 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.933040 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/896c7cf8-3f20-4809-8f6d-a6852a068ad2-etc-openvswitch\") pod \"ovnkube-node-9zbjf\" (UID: \"896c7cf8-3f20-4809-8f6d-a6852a068ad2\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zbjf" Sep 17 10:10:47.934650 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.933137 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a80abea0-a53c-4d9e-8934-7a7f073d71e3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4w7ff\" (UID: \"a80abea0-a53c-4d9e-8934-7a7f073d71e3\") " pod="openshift-multus/multus-additional-cni-plugins-4w7ff" Sep 17 10:10:47.934650 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.933150 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/896c7cf8-3f20-4809-8f6d-a6852a068ad2-run-openvswitch\") pod \"ovnkube-node-9zbjf\" (UID: \"896c7cf8-3f20-4809-8f6d-a6852a068ad2\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zbjf" Sep 17 10:10:47.934650 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.933198 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e4b73c92-0ac6-4d8f-90d8-249cfbcadf4b-etc-sysctl-conf\") pod \"tuned-kg2hz\" (UID: \"e4b73c92-0ac6-4d8f-90d8-249cfbcadf4b\") " pod="openshift-cluster-node-tuning-operator/tuned-kg2hz" Sep 17 10:10:47.934650 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.933214 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ae147838-ff96-4652-bf25-ca99bc25be02-hostroot\") pod \"multus-nxbjh\" (UID: \"ae147838-ff96-4652-bf25-ca99bc25be02\") " pod="openshift-multus/multus-nxbjh" Sep 17 10:10:47.934650 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.933244 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ae147838-ff96-4652-bf25-ca99bc25be02-multus-conf-dir\") pod \"multus-nxbjh\" (UID: \"ae147838-ff96-4652-bf25-ca99bc25be02\") " pod="openshift-multus/multus-nxbjh" Sep 17 10:10:47.934650 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.933267 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/606f20f7-c795-451a-b146-03f3a4f92684-socket-dir\") pod \"aws-ebs-csi-driver-node-kcztl\" (UID: \"606f20f7-c795-451a-b146-03f3a4f92684\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kcztl" Sep 17 10:10:47.934650 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.933290 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/606f20f7-c795-451a-b146-03f3a4f92684-registration-dir\") pod \"aws-ebs-csi-driver-node-kcztl\" (UID: \"606f20f7-c795-451a-b146-03f3a4f92684\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kcztl" Sep 17 10:10:47.934650 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.933308 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e4b73c92-0ac6-4d8f-90d8-249cfbcadf4b-etc-sysctl-conf\") pod \"tuned-kg2hz\" (UID: \"e4b73c92-0ac6-4d8f-90d8-249cfbcadf4b\") " pod="openshift-cluster-node-tuning-operator/tuned-kg2hz" Sep 17 10:10:47.934650 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.933341 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ae147838-ff96-4652-bf25-ca99bc25be02-multus-conf-dir\") pod \"multus-nxbjh\" (UID: \"ae147838-ff96-4652-bf25-ca99bc25be02\") " pod="openshift-multus/multus-nxbjh" Sep 17 10:10:47.934650 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.933349 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/896c7cf8-3f20-4809-8f6d-a6852a068ad2-log-socket\") pod \"ovnkube-node-9zbjf\" (UID: \"896c7cf8-3f20-4809-8f6d-a6852a068ad2\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zbjf" Sep 17 10:10:47.934650 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.933345 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ae147838-ff96-4652-bf25-ca99bc25be02-hostroot\") pod \"multus-nxbjh\" (UID: \"ae147838-ff96-4652-bf25-ca99bc25be02\") " pod="openshift-multus/multus-nxbjh" Sep 17 10:10:47.934650 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.933379 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ae147838-ff96-4652-bf25-ca99bc25be02-host-run-k8s-cni-cncf-io\") pod \"multus-nxbjh\" (UID: \"ae147838-ff96-4652-bf25-ca99bc25be02\") " pod="openshift-multus/multus-nxbjh" Sep 17 10:10:47.934650 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.933403 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ae147838-ff96-4652-bf25-ca99bc25be02-host-run-k8s-cni-cncf-io\") pod \"multus-nxbjh\" (UID: \"ae147838-ff96-4652-bf25-ca99bc25be02\") " pod="openshift-multus/multus-nxbjh" Sep 17 10:10:47.935458 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.933414 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2c409c2a-64a4-4164-8a14-c2a87597dd93-serviceca\") pod \"node-ca-r2dcq\" (UID: \"2c409c2a-64a4-4164-8a14-c2a87597dd93\") " pod="openshift-image-registry/node-ca-r2dcq" Sep 17 10:10:47.935458 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.933448 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b506216d-c159-420e-9111-56acab7568df-iptables-alerter-script\") pod \"iptables-alerter-g8lzm\" (UID: \"b506216d-c159-420e-9111-56acab7568df\") " pod="openshift-network-operator/iptables-alerter-g8lzm" Sep 17 10:10:47.935458 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.933474 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e4b73c92-0ac6-4d8f-90d8-249cfbcadf4b-etc-systemd\") pod \"tuned-kg2hz\" (UID: \"e4b73c92-0ac6-4d8f-90d8-249cfbcadf4b\") " pod="openshift-cluster-node-tuning-operator/tuned-kg2hz" Sep 17 10:10:47.935458 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.933524 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e4b73c92-0ac6-4d8f-90d8-249cfbcadf4b-etc-systemd\") pod \"tuned-kg2hz\" (UID: \"e4b73c92-0ac6-4d8f-90d8-249cfbcadf4b\") " pod="openshift-cluster-node-tuning-operator/tuned-kg2hz" Sep 17 10:10:47.935458 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.933743 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2c409c2a-64a4-4164-8a14-c2a87597dd93-serviceca\") pod \"node-ca-r2dcq\" (UID: \"2c409c2a-64a4-4164-8a14-c2a87597dd93\") " pod="openshift-image-registry/node-ca-r2dcq" Sep 17 10:10:47.935458 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.934170 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e4b73c92-0ac6-4d8f-90d8-249cfbcadf4b-tmp\") pod \"tuned-kg2hz\" (UID: \"e4b73c92-0ac6-4d8f-90d8-249cfbcadf4b\") " pod="openshift-cluster-node-tuning-operator/tuned-kg2hz" Sep 17 10:10:47.935458 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.934532 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/de933a7f-7eae-4e7b-870f-aa9c56565481-agent-certs\") pod \"konnectivity-agent-jkx62\" (UID: \"de933a7f-7eae-4e7b-870f-aa9c56565481\") " pod="kube-system/konnectivity-agent-jkx62" Sep 17 10:10:47.935458 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.935076 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e4b73c92-0ac6-4d8f-90d8-249cfbcadf4b-etc-tuned\") pod \"tuned-kg2hz\" (UID: \"e4b73c92-0ac6-4d8f-90d8-249cfbcadf4b\") " pod="openshift-cluster-node-tuning-operator/tuned-kg2hz" Sep 17 10:10:47.940252 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.940141 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qchf4\" (UniqueName: \"kubernetes.io/projected/2c409c2a-64a4-4164-8a14-c2a87597dd93-kube-api-access-qchf4\") pod \"node-ca-r2dcq\" (UID: \"2c409c2a-64a4-4164-8a14-c2a87597dd93\") " pod="openshift-image-registry/node-ca-r2dcq" Sep 17 10:10:47.940252 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.940195 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxp6q\" (UniqueName: \"kubernetes.io/projected/e4b73c92-0ac6-4d8f-90d8-249cfbcadf4b-kube-api-access-gxp6q\") pod \"tuned-kg2hz\" (UID: \"e4b73c92-0ac6-4d8f-90d8-249cfbcadf4b\") " pod="openshift-cluster-node-tuning-operator/tuned-kg2hz" Sep 17 10:10:47.941313 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.941207 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgzjj\" (UniqueName: \"kubernetes.io/projected/ae147838-ff96-4652-bf25-ca99bc25be02-kube-api-access-mgzjj\") pod \"multus-nxbjh\" (UID: \"ae147838-ff96-4652-bf25-ca99bc25be02\") " pod="openshift-multus/multus-nxbjh" Sep 17 10:10:47.941688 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.941619 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzz8c\" (UniqueName: \"kubernetes.io/projected/c84a7003-1537-4470-8094-ea10a89a86a2-kube-api-access-kzz8c\") pod \"network-metrics-daemon-wfccr\" (UID: \"c84a7003-1537-4470-8094-ea10a89a86a2\") " pod="openshift-multus/network-metrics-daemon-wfccr" Sep 17 10:10:47.941741 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:47.941721 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rn9xb\" (UniqueName: \"kubernetes.io/projected/a80abea0-a53c-4d9e-8934-7a7f073d71e3-kube-api-access-rn9xb\") pod \"multus-additional-cni-plugins-4w7ff\" (UID: \"a80abea0-a53c-4d9e-8934-7a7f073d71e3\") " pod="openshift-multus/multus-additional-cni-plugins-4w7ff" Sep 17 10:10:48.033998 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:48.033912 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/606f20f7-c795-451a-b146-03f3a4f92684-etc-selinux\") pod \"aws-ebs-csi-driver-node-kcztl\" (UID: \"606f20f7-c795-451a-b146-03f3a4f92684\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kcztl" Sep 17 10:10:48.033998 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:48.033951 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b506216d-c159-420e-9111-56acab7568df-host-slash\") pod \"iptables-alerter-g8lzm\" (UID: \"b506216d-c159-420e-9111-56acab7568df\") " pod="openshift-network-operator/iptables-alerter-g8lzm" Sep 17 10:10:48.033998 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:48.033972 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cnjs\" (UniqueName: \"kubernetes.io/projected/b506216d-c159-420e-9111-56acab7568df-kube-api-access-6cnjs\") pod \"iptables-alerter-g8lzm\" (UID: \"b506216d-c159-420e-9111-56acab7568df\") " pod="openshift-network-operator/iptables-alerter-g8lzm" Sep 17 10:10:48.033998 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:48.033993 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/606f20f7-c795-451a-b146-03f3a4f92684-sys-fs\") pod \"aws-ebs-csi-driver-node-kcztl\" (UID: \"606f20f7-c795-451a-b146-03f3a4f92684\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kcztl" Sep 17 10:10:48.034293 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:48.034016 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f7323dd-d769-4f58-92d7-e8a36c2f2854-tmp-dir\") pod \"node-resolver-7f64n\" (UID: \"6f7323dd-d769-4f58-92d7-e8a36c2f2854\") " pod="openshift-dns/node-resolver-7f64n" Sep 17 10:10:48.034293 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:48.034039 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/896c7cf8-3f20-4809-8f6d-a6852a068ad2-host-cni-netd\") pod \"ovnkube-node-9zbjf\" (UID: \"896c7cf8-3f20-4809-8f6d-a6852a068ad2\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zbjf" Sep 17 10:10:48.034293 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:48.034045 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b506216d-c159-420e-9111-56acab7568df-host-slash\") pod \"iptables-alerter-g8lzm\" (UID: \"b506216d-c159-420e-9111-56acab7568df\") " pod="openshift-network-operator/iptables-alerter-g8lzm" Sep 17 10:10:48.034293 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:48.034062 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/896c7cf8-3f20-4809-8f6d-a6852a068ad2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9zbjf\" (UID: \"896c7cf8-3f20-4809-8f6d-a6852a068ad2\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zbjf" Sep 17 10:10:48.034293 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:48.034096 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/606f20f7-c795-451a-b146-03f3a4f92684-etc-selinux\") pod \"aws-ebs-csi-driver-node-kcztl\" (UID: \"606f20f7-c795-451a-b146-03f3a4f92684\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kcztl" Sep 17 10:10:48.034293 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:48.034168 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/896c7cf8-3f20-4809-8f6d-a6852a068ad2-node-log\") pod \"ovnkube-node-9zbjf\" (UID: \"896c7cf8-3f20-4809-8f6d-a6852a068ad2\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zbjf" Sep 17 10:10:48.034293 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:48.034198 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/606f20f7-c795-451a-b146-03f3a4f92684-sys-fs\") pod \"aws-ebs-csi-driver-node-kcztl\" (UID: \"606f20f7-c795-451a-b146-03f3a4f92684\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kcztl" Sep 17 10:10:48.034293 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:48.034206 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/896c7cf8-3f20-4809-8f6d-a6852a068ad2-host-cni-bin\") pod \"ovnkube-node-9zbjf\" (UID: \"896c7cf8-3f20-4809-8f6d-a6852a068ad2\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zbjf" Sep 17 10:10:48.034293 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:48.034233 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/896c7cf8-3f20-4809-8f6d-a6852a068ad2-ovnkube-config\") pod \"ovnkube-node-9zbjf\" (UID: \"896c7cf8-3f20-4809-8f6d-a6852a068ad2\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zbjf" Sep 17 10:10:48.034293 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:48.034234 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/896c7cf8-3f20-4809-8f6d-a6852a068ad2-node-log\") pod \"ovnkube-node-9zbjf\" (UID: \"896c7cf8-3f20-4809-8f6d-a6852a068ad2\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zbjf" Sep 17 10:10:48.034293 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:48.034232 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/896c7cf8-3f20-4809-8f6d-a6852a068ad2-host-cni-netd\") pod \"ovnkube-node-9zbjf\" (UID: \"896c7cf8-3f20-4809-8f6d-a6852a068ad2\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zbjf" Sep 17 10:10:48.034293 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:48.034274 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/896c7cf8-3f20-4809-8f6d-a6852a068ad2-host-cni-bin\") pod \"ovnkube-node-9zbjf\" (UID: \"896c7cf8-3f20-4809-8f6d-a6852a068ad2\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zbjf" Sep 17 10:10:48.034293 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:48.034291 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/896c7cf8-3f20-4809-8f6d-a6852a068ad2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9zbjf\" (UID: \"896c7cf8-3f20-4809-8f6d-a6852a068ad2\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zbjf" Sep 17 10:10:48.034293 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:48.034293 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wj8z\" (UniqueName: \"kubernetes.io/projected/896c7cf8-3f20-4809-8f6d-a6852a068ad2-kube-api-access-2wj8z\") pod \"ovnkube-node-9zbjf\" (UID: \"896c7cf8-3f20-4809-8f6d-a6852a068ad2\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zbjf" Sep 17 10:10:48.034964 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:48.034320 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/896c7cf8-3f20-4809-8f6d-a6852a068ad2-host-slash\") pod \"ovnkube-node-9zbjf\" (UID: \"896c7cf8-3f20-4809-8f6d-a6852a068ad2\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zbjf" Sep 17 10:10:48.034964 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:48.034335 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/896c7cf8-3f20-4809-8f6d-a6852a068ad2-run-systemd\") pod \"ovnkube-node-9zbjf\" (UID: \"896c7cf8-3f20-4809-8f6d-a6852a068ad2\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zbjf" Sep 17 10:10:48.034964 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:48.034350 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/896c7cf8-3f20-4809-8f6d-a6852a068ad2-var-lib-openvswitch\") pod \"ovnkube-node-9zbjf\" (UID: \"896c7cf8-3f20-4809-8f6d-a6852a068ad2\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zbjf" Sep 17 10:10:48.034964 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:48.034387 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/896c7cf8-3f20-4809-8f6d-a6852a068ad2-host-slash\") pod \"ovnkube-node-9zbjf\" (UID: \"896c7cf8-3f20-4809-8f6d-a6852a068ad2\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zbjf" Sep 17 10:10:48.034964 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:48.034394 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/896c7cf8-3f20-4809-8f6d-a6852a068ad2-host-run-netns\") pod \"ovnkube-node-9zbjf\" (UID: \"896c7cf8-3f20-4809-8f6d-a6852a068ad2\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zbjf" Sep 17 10:10:48.034964 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:48.034424 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/896c7cf8-3f20-4809-8f6d-a6852a068ad2-var-lib-openvswitch\") pod \"ovnkube-node-9zbjf\" (UID: \"896c7cf8-3f20-4809-8f6d-a6852a068ad2\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zbjf" Sep 17 10:10:48.034964 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:48.034452 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/896c7cf8-3f20-4809-8f6d-a6852a068ad2-host-run-netns\") pod \"ovnkube-node-9zbjf\" (UID: \"896c7cf8-3f20-4809-8f6d-a6852a068ad2\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zbjf" Sep 17 10:10:48.034964 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:48.034479 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxnn2\" (UniqueName: \"kubernetes.io/projected/f6509b04-a69d-49d5-a08c-aabdd8bf6c86-kube-api-access-nxnn2\") pod \"network-check-target-fppwm\" (UID: \"f6509b04-a69d-49d5-a08c-aabdd8bf6c86\") " pod="openshift-network-diagnostics/network-check-target-fppwm" Sep 17 10:10:48.034964 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:48.034512 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/896c7cf8-3f20-4809-8f6d-a6852a068ad2-host-kubelet\") pod \"ovnkube-node-9zbjf\" (UID: \"896c7cf8-3f20-4809-8f6d-a6852a068ad2\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zbjf" Sep 17 10:10:48.034964 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:48.034509 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f7323dd-d769-4f58-92d7-e8a36c2f2854-tmp-dir\") pod \"node-resolver-7f64n\" (UID: \"6f7323dd-d769-4f58-92d7-e8a36c2f2854\") " pod="openshift-dns/node-resolver-7f64n" Sep 17 10:10:48.034964 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:48.034542 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2x84p\" (UniqueName: \"kubernetes.io/projected/606f20f7-c795-451a-b146-03f3a4f92684-kube-api-access-2x84p\") pod \"aws-ebs-csi-driver-node-kcztl\" (UID: \"606f20f7-c795-451a-b146-03f3a4f92684\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kcztl" Sep 17 10:10:48.034964 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:48.034547 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/896c7cf8-3f20-4809-8f6d-a6852a068ad2-host-kubelet\") pod \"ovnkube-node-9zbjf\" (UID: \"896c7cf8-3f20-4809-8f6d-a6852a068ad2\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zbjf" Sep 17 10:10:48.034964 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:48.034487 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/896c7cf8-3f20-4809-8f6d-a6852a068ad2-run-systemd\") pod \"ovnkube-node-9zbjf\" (UID: \"896c7cf8-3f20-4809-8f6d-a6852a068ad2\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zbjf" Sep 17 10:10:48.034964 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:48.034568 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/896c7cf8-3f20-4809-8f6d-a6852a068ad2-systemd-units\") pod \"ovnkube-node-9zbjf\" (UID: \"896c7cf8-3f20-4809-8f6d-a6852a068ad2\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zbjf" Sep 17 10:10:48.034964 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:48.034592 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/606f20f7-c795-451a-b146-03f3a4f92684-device-dir\") pod \"aws-ebs-csi-driver-node-kcztl\" (UID: \"606f20f7-c795-451a-b146-03f3a4f92684\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kcztl" Sep 17 10:10:48.034964 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:48.034616 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/896c7cf8-3f20-4809-8f6d-a6852a068ad2-host-run-ovn-kubernetes\") pod \"ovnkube-node-9zbjf\" (UID: \"896c7cf8-3f20-4809-8f6d-a6852a068ad2\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zbjf" Sep 17 10:10:48.034964 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:48.034638 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/896c7cf8-3f20-4809-8f6d-a6852a068ad2-ovn-node-metrics-cert\") pod \"ovnkube-node-9zbjf\" (UID: \"896c7cf8-3f20-4809-8f6d-a6852a068ad2\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zbjf" Sep 17 10:10:48.035646 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:48.034648 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/896c7cf8-3f20-4809-8f6d-a6852a068ad2-systemd-units\") pod \"ovnkube-node-9zbjf\" (UID: \"896c7cf8-3f20-4809-8f6d-a6852a068ad2\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zbjf" Sep 17 10:10:48.035646 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:48.034665 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/896c7cf8-3f20-4809-8f6d-a6852a068ad2-ovnkube-script-lib\") pod \"ovnkube-node-9zbjf\" (UID: \"896c7cf8-3f20-4809-8f6d-a6852a068ad2\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zbjf" Sep 17 10:10:48.035646 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:48.034682 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/896c7cf8-3f20-4809-8f6d-a6852a068ad2-host-run-ovn-kubernetes\") pod \"ovnkube-node-9zbjf\" (UID: \"896c7cf8-3f20-4809-8f6d-a6852a068ad2\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zbjf" Sep 17 10:10:48.035646 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:48.034691 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/606f20f7-c795-451a-b146-03f3a4f92684-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kcztl\" (UID: \"606f20f7-c795-451a-b146-03f3a4f92684\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kcztl" Sep 17 10:10:48.035646 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:48.034686 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/606f20f7-c795-451a-b146-03f3a4f92684-device-dir\") pod \"aws-ebs-csi-driver-node-kcztl\" (UID: \"606f20f7-c795-451a-b146-03f3a4f92684\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kcztl" Sep 17 10:10:48.035646 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:48.034715 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6f7323dd-d769-4f58-92d7-e8a36c2f2854-hosts-file\") pod \"node-resolver-7f64n\" (UID: \"6f7323dd-d769-4f58-92d7-e8a36c2f2854\") " pod="openshift-dns/node-resolver-7f64n" Sep 17 10:10:48.035646 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:48.034749 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/896c7cf8-3f20-4809-8f6d-a6852a068ad2-ovnkube-config\") pod \"ovnkube-node-9zbjf\" (UID: \"896c7cf8-3f20-4809-8f6d-a6852a068ad2\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zbjf" Sep 17 10:10:48.035646 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:48.034754 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/896c7cf8-3f20-4809-8f6d-a6852a068ad2-run-ovn\") pod \"ovnkube-node-9zbjf\" (UID: \"896c7cf8-3f20-4809-8f6d-a6852a068ad2\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zbjf" Sep 17 10:10:48.035646 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:48.034760 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/606f20f7-c795-451a-b146-03f3a4f92684-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kcztl\" (UID: \"606f20f7-c795-451a-b146-03f3a4f92684\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kcztl" Sep 17 10:10:48.035646 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:48.034769 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6f7323dd-d769-4f58-92d7-e8a36c2f2854-hosts-file\") pod \"node-resolver-7f64n\" (UID: \"6f7323dd-d769-4f58-92d7-e8a36c2f2854\") " pod="openshift-dns/node-resolver-7f64n" Sep 17 10:10:48.035646 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:48.034795 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/896c7cf8-3f20-4809-8f6d-a6852a068ad2-env-overrides\") pod \"ovnkube-node-9zbjf\" (UID: \"896c7cf8-3f20-4809-8f6d-a6852a068ad2\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zbjf" Sep 17 10:10:48.035646 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:48.034813 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/896c7cf8-3f20-4809-8f6d-a6852a068ad2-run-ovn\") pod \"ovnkube-node-9zbjf\" (UID: \"896c7cf8-3f20-4809-8f6d-a6852a068ad2\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zbjf" Sep 17 10:10:48.035646 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:48.034826 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njprp\" (UniqueName: \"kubernetes.io/projected/6f7323dd-d769-4f58-92d7-e8a36c2f2854-kube-api-access-njprp\") pod \"node-resolver-7f64n\" (UID: \"6f7323dd-d769-4f58-92d7-e8a36c2f2854\") " pod="openshift-dns/node-resolver-7f64n" Sep 17 10:10:48.035646 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:48.034845 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/896c7cf8-3f20-4809-8f6d-a6852a068ad2-etc-openvswitch\") pod \"ovnkube-node-9zbjf\" (UID: \"896c7cf8-3f20-4809-8f6d-a6852a068ad2\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zbjf" Sep 17 10:10:48.035646 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:48.034863 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/896c7cf8-3f20-4809-8f6d-a6852a068ad2-run-openvswitch\") pod \"ovnkube-node-9zbjf\" (UID: \"896c7cf8-3f20-4809-8f6d-a6852a068ad2\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zbjf" Sep 17 10:10:48.035646 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:48.034887 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/606f20f7-c795-451a-b146-03f3a4f92684-socket-dir\") pod \"aws-ebs-csi-driver-node-kcztl\" (UID: \"606f20f7-c795-451a-b146-03f3a4f92684\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kcztl" Sep 17 10:10:48.035646 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:48.034898 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/896c7cf8-3f20-4809-8f6d-a6852a068ad2-etc-openvswitch\") pod \"ovnkube-node-9zbjf\" (UID: \"896c7cf8-3f20-4809-8f6d-a6852a068ad2\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zbjf" Sep 17 10:10:48.036275 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:48.034904 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/606f20f7-c795-451a-b146-03f3a4f92684-registration-dir\") pod \"aws-ebs-csi-driver-node-kcztl\" (UID: \"606f20f7-c795-451a-b146-03f3a4f92684\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kcztl" Sep 17 10:10:48.036275 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:48.034935 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/606f20f7-c795-451a-b146-03f3a4f92684-registration-dir\") pod \"aws-ebs-csi-driver-node-kcztl\" (UID: \"606f20f7-c795-451a-b146-03f3a4f92684\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kcztl" Sep 17 10:10:48.036275 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:48.034938 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/896c7cf8-3f20-4809-8f6d-a6852a068ad2-log-socket\") pod \"ovnkube-node-9zbjf\" (UID: \"896c7cf8-3f20-4809-8f6d-a6852a068ad2\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zbjf" Sep 17 10:10:48.036275 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:48.034966 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b506216d-c159-420e-9111-56acab7568df-iptables-alerter-script\") pod \"iptables-alerter-g8lzm\" (UID: \"b506216d-c159-420e-9111-56acab7568df\") " pod="openshift-network-operator/iptables-alerter-g8lzm" Sep 17 10:10:48.036275 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:48.034971 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/896c7cf8-3f20-4809-8f6d-a6852a068ad2-run-openvswitch\") pod \"ovnkube-node-9zbjf\" (UID: \"896c7cf8-3f20-4809-8f6d-a6852a068ad2\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zbjf" Sep 17 10:10:48.036275 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:48.035039 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/896c7cf8-3f20-4809-8f6d-a6852a068ad2-log-socket\") pod \"ovnkube-node-9zbjf\" (UID: \"896c7cf8-3f20-4809-8f6d-a6852a068ad2\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zbjf" Sep 17 10:10:48.036275 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:48.035058 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/606f20f7-c795-451a-b146-03f3a4f92684-socket-dir\") pod \"aws-ebs-csi-driver-node-kcztl\" (UID: \"606f20f7-c795-451a-b146-03f3a4f92684\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kcztl" Sep 17 10:10:48.036275 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:48.035162 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/896c7cf8-3f20-4809-8f6d-a6852a068ad2-env-overrides\") pod \"ovnkube-node-9zbjf\" (UID: \"896c7cf8-3f20-4809-8f6d-a6852a068ad2\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zbjf" Sep 17 10:10:48.036275 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:48.035240 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/896c7cf8-3f20-4809-8f6d-a6852a068ad2-ovnkube-script-lib\") pod \"ovnkube-node-9zbjf\" (UID: \"896c7cf8-3f20-4809-8f6d-a6852a068ad2\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zbjf" Sep 17 10:10:48.036275 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:48.035457 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b506216d-c159-420e-9111-56acab7568df-iptables-alerter-script\") pod \"iptables-alerter-g8lzm\" (UID: \"b506216d-c159-420e-9111-56acab7568df\") " pod="openshift-network-operator/iptables-alerter-g8lzm" Sep 17 10:10:48.037347 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:48.037323 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/896c7cf8-3f20-4809-8f6d-a6852a068ad2-ovn-node-metrics-cert\") pod \"ovnkube-node-9zbjf\" (UID: \"896c7cf8-3f20-4809-8f6d-a6852a068ad2\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zbjf" Sep 17 10:10:48.044521 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:10:48.044486 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 17 10:10:48.044521 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:10:48.044514 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 17 10:10:48.044712 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:10:48.044531 2581 projected.go:196] Error preparing data for projected volume kube-api-access-nxnn2 for pod openshift-network-diagnostics/network-check-target-fppwm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 10:10:48.044712 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:10:48.044599 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f6509b04-a69d-49d5-a08c-aabdd8bf6c86-kube-api-access-nxnn2 podName:f6509b04-a69d-49d5-a08c-aabdd8bf6c86 nodeName:}" failed. No retries permitted until 2026-09-17 10:10:48.544578667 +0000 UTC m=+3.093135773 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-nxnn2" (UniqueName: "kubernetes.io/projected/f6509b04-a69d-49d5-a08c-aabdd8bf6c86-kube-api-access-nxnn2") pod "network-check-target-fppwm" (UID: "f6509b04-a69d-49d5-a08c-aabdd8bf6c86") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 10:10:48.047048 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:48.047024 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2x84p\" (UniqueName: \"kubernetes.io/projected/606f20f7-c795-451a-b146-03f3a4f92684-kube-api-access-2x84p\") pod \"aws-ebs-csi-driver-node-kcztl\" (UID: \"606f20f7-c795-451a-b146-03f3a4f92684\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kcztl" Sep 17 10:10:48.047152 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:48.047072 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cnjs\" (UniqueName: \"kubernetes.io/projected/b506216d-c159-420e-9111-56acab7568df-kube-api-access-6cnjs\") pod \"iptables-alerter-g8lzm\" (UID: \"b506216d-c159-420e-9111-56acab7568df\") " pod="openshift-network-operator/iptables-alerter-g8lzm" Sep 17 10:10:48.047152 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:48.047131 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-njprp\" (UniqueName: \"kubernetes.io/projected/6f7323dd-d769-4f58-92d7-e8a36c2f2854-kube-api-access-njprp\") pod \"node-resolver-7f64n\" (UID: \"6f7323dd-d769-4f58-92d7-e8a36c2f2854\") " pod="openshift-dns/node-resolver-7f64n" Sep 17 10:10:48.047527 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:48.047506 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wj8z\" (UniqueName: \"kubernetes.io/projected/896c7cf8-3f20-4809-8f6d-a6852a068ad2-kube-api-access-2wj8z\") pod \"ovnkube-node-9zbjf\" (UID: \"896c7cf8-3f20-4809-8f6d-a6852a068ad2\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zbjf" Sep 17 10:10:48.115946 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:48.115911 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-jkx62" Sep 17 10:10:48.125445 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:48.125412 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-kg2hz" Sep 17 10:10:48.134049 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:48.134022 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-4w7ff" Sep 17 10:10:48.140758 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:48.140740 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-nxbjh" Sep 17 10:10:48.149515 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:48.149494 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-r2dcq" Sep 17 10:10:48.157266 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:48.157243 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kcztl" Sep 17 10:10:48.164924 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:48.164894 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-7f64n" Sep 17 10:10:48.173629 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:48.173596 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-g8lzm" Sep 17 10:10:48.179590 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:48.179563 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-9zbjf" Sep 17 10:10:48.437343 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:48.437300 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c84a7003-1537-4470-8094-ea10a89a86a2-metrics-certs\") pod \"network-metrics-daemon-wfccr\" (UID: \"c84a7003-1537-4470-8094-ea10a89a86a2\") " pod="openshift-multus/network-metrics-daemon-wfccr" Sep 17 10:10:48.437343 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:48.437355 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c304d2b0-b1ce-4a29-86b1-341d5db70d4f-original-pull-secret\") pod \"global-pull-secret-syncer-v9xpw\" (UID: \"c304d2b0-b1ce-4a29-86b1-341d5db70d4f\") " pod="kube-system/global-pull-secret-syncer-v9xpw" Sep 17 10:10:48.437602 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:10:48.437469 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 17 10:10:48.437602 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:10:48.437481 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 10:10:48.437602 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:10:48.437541 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c304d2b0-b1ce-4a29-86b1-341d5db70d4f-original-pull-secret podName:c304d2b0-b1ce-4a29-86b1-341d5db70d4f nodeName:}" failed. No retries permitted until 2026-09-17 10:10:49.437522731 +0000 UTC m=+3.986079825 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c304d2b0-b1ce-4a29-86b1-341d5db70d4f-original-pull-secret") pod "global-pull-secret-syncer-v9xpw" (UID: "c304d2b0-b1ce-4a29-86b1-341d5db70d4f") : object "kube-system"/"original-pull-secret" not registered Sep 17 10:10:48.437602 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:10:48.437560 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c84a7003-1537-4470-8094-ea10a89a86a2-metrics-certs podName:c84a7003-1537-4470-8094-ea10a89a86a2 nodeName:}" failed. No retries permitted until 2026-09-17 10:10:49.437551454 +0000 UTC m=+3.986108549 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c84a7003-1537-4470-8094-ea10a89a86a2-metrics-certs") pod "network-metrics-daemon-wfccr" (UID: "c84a7003-1537-4470-8094-ea10a89a86a2") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 10:10:48.638564 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:48.638470 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxnn2\" (UniqueName: \"kubernetes.io/projected/f6509b04-a69d-49d5-a08c-aabdd8bf6c86-kube-api-access-nxnn2\") pod \"network-check-target-fppwm\" (UID: \"f6509b04-a69d-49d5-a08c-aabdd8bf6c86\") " pod="openshift-network-diagnostics/network-check-target-fppwm" Sep 17 10:10:48.638724 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:10:48.638666 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 17 10:10:48.638724 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:10:48.638698 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 17 10:10:48.638724 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:10:48.638711 2581 projected.go:196] Error preparing data for projected volume kube-api-access-nxnn2 for pod openshift-network-diagnostics/network-check-target-fppwm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 10:10:48.638857 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:10:48.638773 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f6509b04-a69d-49d5-a08c-aabdd8bf6c86-kube-api-access-nxnn2 podName:f6509b04-a69d-49d5-a08c-aabdd8bf6c86 nodeName:}" failed. No retries permitted until 2026-09-17 10:10:49.638756757 +0000 UTC m=+4.187313848 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-nxnn2" (UniqueName: "kubernetes.io/projected/f6509b04-a69d-49d5-a08c-aabdd8bf6c86-kube-api-access-nxnn2") pod "network-check-target-fppwm" (UID: "f6509b04-a69d-49d5-a08c-aabdd8bf6c86") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 10:10:48.863624 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:48.863576 2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 10:05:48 +0000 UTC" deadline="2028-06-07 18:32:13.16190791 +0000 UTC" Sep 17 10:10:48.863624 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:48.863614 2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15104h21m24.298298711s" Sep 17 10:10:48.932370 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:48.932342 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda80abea0_a53c_4d9e_8934_7a7f073d71e3.slice/crio-0ea07e8a522a66b210f4c2f2244957a5c631569e067f8fda8fba07064f24ee2b WatchSource:0}: Error finding container 0ea07e8a522a66b210f4c2f2244957a5c631569e067f8fda8fba07064f24ee2b: Status 404 returned error can't find the container with id 0ea07e8a522a66b210f4c2f2244957a5c631569e067f8fda8fba07064f24ee2b Sep 17 10:10:48.933591 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:48.933552 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c409c2a_64a4_4164_8a14_c2a87597dd93.slice/crio-05c4a6b19253f61d9be6c83f99038b53df15dd75b33068116c685421aa72113c WatchSource:0}: Error finding container 05c4a6b19253f61d9be6c83f99038b53df15dd75b33068116c685421aa72113c: Status 404 returned error can't find the container with id 05c4a6b19253f61d9be6c83f99038b53df15dd75b33068116c685421aa72113c Sep 17 10:10:48.934904 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:48.934445 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb506216d_c159_420e_9111_56acab7568df.slice/crio-93882f7f2e12821f9a425c18d2c1a71bc27bb9007f2aab62d5a4730b5c4c0416 WatchSource:0}: Error finding container 93882f7f2e12821f9a425c18d2c1a71bc27bb9007f2aab62d5a4730b5c4c0416: Status 404 returned error can't find the container with id 93882f7f2e12821f9a425c18d2c1a71bc27bb9007f2aab62d5a4730b5c4c0416 Sep 17 10:10:48.938387 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:48.938259 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wfccr" Sep 17 10:10:48.938387 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:48.938367 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod896c7cf8_3f20_4809_8f6d_a6852a068ad2.slice/crio-67fe4932e7e09488406aa9fe2e1badc83986451d929910ca73c23ff41c9ba5fd WatchSource:0}: Error finding container 67fe4932e7e09488406aa9fe2e1badc83986451d929910ca73c23ff41c9ba5fd: Status 404 returned error can't find the container with id 67fe4932e7e09488406aa9fe2e1badc83986451d929910ca73c23ff41c9ba5fd Sep 17 10:10:48.938524 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:10:48.938441 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wfccr" podUID="c84a7003-1537-4470-8094-ea10a89a86a2" Sep 17 10:10:48.939290 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:48.939268 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae147838_ff96_4652_bf25_ca99bc25be02.slice/crio-b7a9aaa76956fcd882e1410e1abf190e049ae5e2675610e2ca926184c5f0c85f WatchSource:0}: Error finding container b7a9aaa76956fcd882e1410e1abf190e049ae5e2675610e2ca926184c5f0c85f: Status 404 returned error can't find the container with id b7a9aaa76956fcd882e1410e1abf190e049ae5e2675610e2ca926184c5f0c85f Sep 17 10:10:48.941288 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:48.941210 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod606f20f7_c795_451a_b146_03f3a4f92684.slice/crio-6852aa2dc59bae380babe19ec00fa2c44140e49087a5a22e5e71c713391f50f6 WatchSource:0}: Error finding container 6852aa2dc59bae380babe19ec00fa2c44140e49087a5a22e5e71c713391f50f6: Status 404 returned error can't find the container with id 6852aa2dc59bae380babe19ec00fa2c44140e49087a5a22e5e71c713391f50f6 Sep 17 10:10:48.942086 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:48.941859 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde933a7f_7eae_4e7b_870f_aa9c56565481.slice/crio-551177c7bb3e5b25b567768069a9fc32f16b09111ecfc8ce88abe105a556adfb WatchSource:0}: Error finding container 551177c7bb3e5b25b567768069a9fc32f16b09111ecfc8ce88abe105a556adfb: Status 404 returned error can't find the container with id 551177c7bb3e5b25b567768069a9fc32f16b09111ecfc8ce88abe105a556adfb Sep 17 10:10:48.944092 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:48.944008 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f7323dd_d769_4f58_92d7_e8a36c2f2854.slice/crio-8376b112d7e544c5daad107e10c431ebf1910279842abf1a5ca07fb792559234 WatchSource:0}: Error finding container 8376b112d7e544c5daad107e10c431ebf1910279842abf1a5ca07fb792559234: Status 404 returned error can't find the container with id 8376b112d7e544c5daad107e10c431ebf1910279842abf1a5ca07fb792559234 Sep 17 10:10:48.944814 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:10:48.944731 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4b73c92_0ac6_4d8f_90d8_249cfbcadf4b.slice/crio-ad01ddcf306f852eda19e8f935dcd3018243ae651a8267c561adf454ca7e48de WatchSource:0}: Error finding container ad01ddcf306f852eda19e8f935dcd3018243ae651a8267c561adf454ca7e48de: Status 404 returned error can't find the container with id ad01ddcf306f852eda19e8f935dcd3018243ae651a8267c561adf454ca7e48de Sep 17 10:10:48.947441 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:48.947417 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-7f64n" event={"ID":"6f7323dd-d769-4f58-92d7-e8a36c2f2854","Type":"ContainerStarted","Data":"8376b112d7e544c5daad107e10c431ebf1910279842abf1a5ca07fb792559234"} Sep 17 10:10:48.949044 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:48.949018 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-jkx62" event={"ID":"de933a7f-7eae-4e7b-870f-aa9c56565481","Type":"ContainerStarted","Data":"551177c7bb3e5b25b567768069a9fc32f16b09111ecfc8ce88abe105a556adfb"} Sep 17 10:10:48.950706 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:48.950684 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kcztl" event={"ID":"606f20f7-c795-451a-b146-03f3a4f92684","Type":"ContainerStarted","Data":"6852aa2dc59bae380babe19ec00fa2c44140e49087a5a22e5e71c713391f50f6"} Sep 17 10:10:48.952093 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:48.952065 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-g8lzm" event={"ID":"b506216d-c159-420e-9111-56acab7568df","Type":"ContainerStarted","Data":"93882f7f2e12821f9a425c18d2c1a71bc27bb9007f2aab62d5a4730b5c4c0416"} Sep 17 10:10:48.953277 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:48.953257 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-kg2hz" event={"ID":"e4b73c92-0ac6-4d8f-90d8-249cfbcadf4b","Type":"ContainerStarted","Data":"ad01ddcf306f852eda19e8f935dcd3018243ae651a8267c561adf454ca7e48de"} Sep 17 10:10:48.954617 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:48.954592 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-nxbjh" event={"ID":"ae147838-ff96-4652-bf25-ca99bc25be02","Type":"ContainerStarted","Data":"b7a9aaa76956fcd882e1410e1abf190e049ae5e2675610e2ca926184c5f0c85f"} Sep 17 10:10:48.955916 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:48.955885 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9zbjf" event={"ID":"896c7cf8-3f20-4809-8f6d-a6852a068ad2","Type":"ContainerStarted","Data":"67fe4932e7e09488406aa9fe2e1badc83986451d929910ca73c23ff41c9ba5fd"} Sep 17 10:10:48.961488 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:48.961402 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-r2dcq" event={"ID":"2c409c2a-64a4-4164-8a14-c2a87597dd93","Type":"ContainerStarted","Data":"05c4a6b19253f61d9be6c83f99038b53df15dd75b33068116c685421aa72113c"} Sep 17 10:10:48.964794 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:48.964752 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4w7ff" event={"ID":"a80abea0-a53c-4d9e-8934-7a7f073d71e3","Type":"ContainerStarted","Data":"0ea07e8a522a66b210f4c2f2244957a5c631569e067f8fda8fba07064f24ee2b"} Sep 17 10:10:49.444644 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:49.444266 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c84a7003-1537-4470-8094-ea10a89a86a2-metrics-certs\") pod \"network-metrics-daemon-wfccr\" (UID: \"c84a7003-1537-4470-8094-ea10a89a86a2\") " pod="openshift-multus/network-metrics-daemon-wfccr" Sep 17 10:10:49.444644 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:49.444592 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c304d2b0-b1ce-4a29-86b1-341d5db70d4f-original-pull-secret\") pod \"global-pull-secret-syncer-v9xpw\" (UID: \"c304d2b0-b1ce-4a29-86b1-341d5db70d4f\") " pod="kube-system/global-pull-secret-syncer-v9xpw" Sep 17 10:10:49.444644 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:10:49.444429 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 10:10:49.444938 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:10:49.444705 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c84a7003-1537-4470-8094-ea10a89a86a2-metrics-certs podName:c84a7003-1537-4470-8094-ea10a89a86a2 nodeName:}" failed. No retries permitted until 2026-09-17 10:10:51.444683284 +0000 UTC m=+5.993240398 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c84a7003-1537-4470-8094-ea10a89a86a2-metrics-certs") pod "network-metrics-daemon-wfccr" (UID: "c84a7003-1537-4470-8094-ea10a89a86a2") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 10:10:49.444938 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:10:49.444758 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 17 10:10:49.444938 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:10:49.444816 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c304d2b0-b1ce-4a29-86b1-341d5db70d4f-original-pull-secret podName:c304d2b0-b1ce-4a29-86b1-341d5db70d4f nodeName:}" failed. No retries permitted until 2026-09-17 10:10:51.444801506 +0000 UTC m=+5.993358599 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c304d2b0-b1ce-4a29-86b1-341d5db70d4f-original-pull-secret") pod "global-pull-secret-syncer-v9xpw" (UID: "c304d2b0-b1ce-4a29-86b1-341d5db70d4f") : object "kube-system"/"original-pull-secret" not registered Sep 17 10:10:49.480586 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:49.480553 2581 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 17 10:10:49.646884 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:49.646788 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxnn2\" (UniqueName: \"kubernetes.io/projected/f6509b04-a69d-49d5-a08c-aabdd8bf6c86-kube-api-access-nxnn2\") pod \"network-check-target-fppwm\" (UID: \"f6509b04-a69d-49d5-a08c-aabdd8bf6c86\") " pod="openshift-network-diagnostics/network-check-target-fppwm" Sep 17 10:10:49.647074 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:10:49.647035 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 17 10:10:49.647074 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:10:49.647062 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 17 10:10:49.647074 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:10:49.647076 2581 projected.go:196] Error preparing data for projected volume kube-api-access-nxnn2 for pod openshift-network-diagnostics/network-check-target-fppwm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 10:10:49.647241 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:10:49.647142 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f6509b04-a69d-49d5-a08c-aabdd8bf6c86-kube-api-access-nxnn2 podName:f6509b04-a69d-49d5-a08c-aabdd8bf6c86 nodeName:}" failed. No retries permitted until 2026-09-17 10:10:51.647122212 +0000 UTC m=+6.195679306 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-nxnn2" (UniqueName: "kubernetes.io/projected/f6509b04-a69d-49d5-a08c-aabdd8bf6c86-kube-api-access-nxnn2") pod "network-check-target-fppwm" (UID: "f6509b04-a69d-49d5-a08c-aabdd8bf6c86") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 10:10:49.864701 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:49.864558 2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 10:05:48 +0000 UTC" deadline="2028-04-11 03:56:42.114309915 +0000 UTC" Sep 17 10:10:49.864701 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:49.864650 2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13721h45m52.249665102s" Sep 17 10:10:49.938652 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:49.938612 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v9xpw" Sep 17 10:10:49.938861 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:10:49.938763 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v9xpw" podUID="c304d2b0-b1ce-4a29-86b1-341d5db70d4f" Sep 17 10:10:49.939255 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:49.939231 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fppwm" Sep 17 10:10:49.939361 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:10:49.939341 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fppwm" podUID="f6509b04-a69d-49d5-a08c-aabdd8bf6c86" Sep 17 10:10:49.985033 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:49.984993 2581 generic.go:356] "Generic (PLEG): container finished" podID="c075fc41f60411f931a6e82e74eff54a" containerID="0c6cc5a08cab3f1823275918216f4a4f51abf165ff1fda83c26c265030d06226" exitCode=0 Sep 17 10:10:49.985228 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:49.985099 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-203.ec2.internal" event={"ID":"c075fc41f60411f931a6e82e74eff54a","Type":"ContainerDied","Data":"0c6cc5a08cab3f1823275918216f4a4f51abf165ff1fda83c26c265030d06226"} Sep 17 10:10:49.999971 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:49.999931 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-203.ec2.internal" event={"ID":"fcac5a3f4a7d9d4b6710d0af7b8c7711","Type":"ContainerStarted","Data":"f9f471b4e87764eda4b6984f38fe046f87d37811d8eca3228838abfaa9d32f24"} Sep 17 10:10:50.020772 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:50.020707 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-203.ec2.internal" podStartSLOduration=3.020674299 podStartE2EDuration="3.020674299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 10:10:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 10:10:50.02033236 +0000 UTC m=+4.568889489" watchObservedRunningTime="2026-09-17 10:10:50.020674299 +0000 UTC m=+4.569231453" Sep 17 10:10:50.939088 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:50.938999 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wfccr" Sep 17 10:10:50.939670 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:10:50.939162 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wfccr" podUID="c84a7003-1537-4470-8094-ea10a89a86a2" Sep 17 10:10:51.044293 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:51.044237 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-203.ec2.internal" event={"ID":"c075fc41f60411f931a6e82e74eff54a","Type":"ContainerStarted","Data":"363d8d8018a1e22665f172f6badaec96877ebbf277fd87117c4597bbdd12b97e"} Sep 17 10:10:51.462162 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:51.462095 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c304d2b0-b1ce-4a29-86b1-341d5db70d4f-original-pull-secret\") pod \"global-pull-secret-syncer-v9xpw\" (UID: \"c304d2b0-b1ce-4a29-86b1-341d5db70d4f\") " pod="kube-system/global-pull-secret-syncer-v9xpw" Sep 17 10:10:51.462392 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:51.462212 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c84a7003-1537-4470-8094-ea10a89a86a2-metrics-certs\") pod \"network-metrics-daemon-wfccr\" (UID: \"c84a7003-1537-4470-8094-ea10a89a86a2\") " pod="openshift-multus/network-metrics-daemon-wfccr" Sep 17 10:10:51.462392 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:10:51.462355 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 10:10:51.462523 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:10:51.462424 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c84a7003-1537-4470-8094-ea10a89a86a2-metrics-certs podName:c84a7003-1537-4470-8094-ea10a89a86a2 nodeName:}" failed. No retries permitted until 2026-09-17 10:10:55.462403871 +0000 UTC m=+10.010960966 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c84a7003-1537-4470-8094-ea10a89a86a2-metrics-certs") pod "network-metrics-daemon-wfccr" (UID: "c84a7003-1537-4470-8094-ea10a89a86a2") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 10:10:51.462892 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:10:51.462871 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 17 10:10:51.462990 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:10:51.462926 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c304d2b0-b1ce-4a29-86b1-341d5db70d4f-original-pull-secret podName:c304d2b0-b1ce-4a29-86b1-341d5db70d4f nodeName:}" failed. No retries permitted until 2026-09-17 10:10:55.462910948 +0000 UTC m=+10.011468043 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c304d2b0-b1ce-4a29-86b1-341d5db70d4f-original-pull-secret") pod "global-pull-secret-syncer-v9xpw" (UID: "c304d2b0-b1ce-4a29-86b1-341d5db70d4f") : object "kube-system"/"original-pull-secret" not registered Sep 17 10:10:51.664348 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:51.664303 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxnn2\" (UniqueName: \"kubernetes.io/projected/f6509b04-a69d-49d5-a08c-aabdd8bf6c86-kube-api-access-nxnn2\") pod \"network-check-target-fppwm\" (UID: \"f6509b04-a69d-49d5-a08c-aabdd8bf6c86\") " pod="openshift-network-diagnostics/network-check-target-fppwm" Sep 17 10:10:51.664572 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:10:51.664553 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 17 10:10:51.664649 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:10:51.664581 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 17 10:10:51.664649 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:10:51.664595 2581 projected.go:196] Error preparing data for projected volume kube-api-access-nxnn2 for pod openshift-network-diagnostics/network-check-target-fppwm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 10:10:51.664761 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:10:51.664663 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f6509b04-a69d-49d5-a08c-aabdd8bf6c86-kube-api-access-nxnn2 podName:f6509b04-a69d-49d5-a08c-aabdd8bf6c86 nodeName:}" failed. No retries permitted until 2026-09-17 10:10:55.664642054 +0000 UTC m=+10.213199170 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-nxnn2" (UniqueName: "kubernetes.io/projected/f6509b04-a69d-49d5-a08c-aabdd8bf6c86-kube-api-access-nxnn2") pod "network-check-target-fppwm" (UID: "f6509b04-a69d-49d5-a08c-aabdd8bf6c86") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 10:10:51.938774 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:51.938671 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v9xpw" Sep 17 10:10:51.938974 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:10:51.938853 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v9xpw" podUID="c304d2b0-b1ce-4a29-86b1-341d5db70d4f" Sep 17 10:10:51.939316 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:51.939270 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fppwm" Sep 17 10:10:51.939725 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:10:51.939368 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fppwm" podUID="f6509b04-a69d-49d5-a08c-aabdd8bf6c86" Sep 17 10:10:52.938858 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:52.938587 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wfccr" Sep 17 10:10:52.938858 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:10:52.938761 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wfccr" podUID="c84a7003-1537-4470-8094-ea10a89a86a2" Sep 17 10:10:53.938736 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:53.938696 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fppwm" Sep 17 10:10:53.939247 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:10:53.938867 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fppwm" podUID="f6509b04-a69d-49d5-a08c-aabdd8bf6c86" Sep 17 10:10:53.939247 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:53.938964 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v9xpw" Sep 17 10:10:53.939247 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:10:53.939081 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v9xpw" podUID="c304d2b0-b1ce-4a29-86b1-341d5db70d4f" Sep 17 10:10:54.938518 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:54.938437 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wfccr" Sep 17 10:10:54.938665 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:10:54.938591 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wfccr" podUID="c84a7003-1537-4470-8094-ea10a89a86a2" Sep 17 10:10:55.501230 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:55.501159 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c84a7003-1537-4470-8094-ea10a89a86a2-metrics-certs\") pod \"network-metrics-daemon-wfccr\" (UID: \"c84a7003-1537-4470-8094-ea10a89a86a2\") " pod="openshift-multus/network-metrics-daemon-wfccr" Sep 17 10:10:55.501230 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:55.501212 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c304d2b0-b1ce-4a29-86b1-341d5db70d4f-original-pull-secret\") pod \"global-pull-secret-syncer-v9xpw\" (UID: \"c304d2b0-b1ce-4a29-86b1-341d5db70d4f\") " pod="kube-system/global-pull-secret-syncer-v9xpw" Sep 17 10:10:55.501726 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:10:55.501376 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 17 10:10:55.501726 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:10:55.501437 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c304d2b0-b1ce-4a29-86b1-341d5db70d4f-original-pull-secret podName:c304d2b0-b1ce-4a29-86b1-341d5db70d4f nodeName:}" failed. No retries permitted until 2026-09-17 10:11:03.501419461 +0000 UTC m=+18.049976557 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c304d2b0-b1ce-4a29-86b1-341d5db70d4f-original-pull-secret") pod "global-pull-secret-syncer-v9xpw" (UID: "c304d2b0-b1ce-4a29-86b1-341d5db70d4f") : object "kube-system"/"original-pull-secret" not registered Sep 17 10:10:55.501905 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:10:55.501886 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 10:10:55.501968 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:10:55.501949 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c84a7003-1537-4470-8094-ea10a89a86a2-metrics-certs podName:c84a7003-1537-4470-8094-ea10a89a86a2 nodeName:}" failed. No retries permitted until 2026-09-17 10:11:03.501933773 +0000 UTC m=+18.050490880 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c84a7003-1537-4470-8094-ea10a89a86a2-metrics-certs") pod "network-metrics-daemon-wfccr" (UID: "c84a7003-1537-4470-8094-ea10a89a86a2") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 10:10:55.703936 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:55.703332 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxnn2\" (UniqueName: \"kubernetes.io/projected/f6509b04-a69d-49d5-a08c-aabdd8bf6c86-kube-api-access-nxnn2\") pod \"network-check-target-fppwm\" (UID: \"f6509b04-a69d-49d5-a08c-aabdd8bf6c86\") " pod="openshift-network-diagnostics/network-check-target-fppwm" Sep 17 10:10:55.703936 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:10:55.703542 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 17 10:10:55.703936 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:10:55.703561 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 17 10:10:55.703936 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:10:55.703574 2581 projected.go:196] Error preparing data for projected volume kube-api-access-nxnn2 for pod openshift-network-diagnostics/network-check-target-fppwm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 10:10:55.703936 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:10:55.703649 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f6509b04-a69d-49d5-a08c-aabdd8bf6c86-kube-api-access-nxnn2 podName:f6509b04-a69d-49d5-a08c-aabdd8bf6c86 nodeName:}" failed. No retries permitted until 2026-09-17 10:11:03.703630732 +0000 UTC m=+18.252187825 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-nxnn2" (UniqueName: "kubernetes.io/projected/f6509b04-a69d-49d5-a08c-aabdd8bf6c86-kube-api-access-nxnn2") pod "network-check-target-fppwm" (UID: "f6509b04-a69d-49d5-a08c-aabdd8bf6c86") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 10:10:55.942168 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:55.942032 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v9xpw" Sep 17 10:10:55.942168 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:10:55.942146 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v9xpw" podUID="c304d2b0-b1ce-4a29-86b1-341d5db70d4f" Sep 17 10:10:55.942576 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:55.942038 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fppwm" Sep 17 10:10:55.942576 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:10:55.942539 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fppwm" podUID="f6509b04-a69d-49d5-a08c-aabdd8bf6c86" Sep 17 10:10:56.939682 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:56.939128 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wfccr" Sep 17 10:10:56.939682 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:10:56.939275 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wfccr" podUID="c84a7003-1537-4470-8094-ea10a89a86a2" Sep 17 10:10:57.938816 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:57.938402 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fppwm" Sep 17 10:10:57.938816 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:10:57.938530 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fppwm" podUID="f6509b04-a69d-49d5-a08c-aabdd8bf6c86" Sep 17 10:10:57.938816 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:57.938812 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v9xpw" Sep 17 10:10:57.939107 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:10:57.938907 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v9xpw" podUID="c304d2b0-b1ce-4a29-86b1-341d5db70d4f" Sep 17 10:10:58.939148 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:58.939109 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wfccr" Sep 17 10:10:58.939566 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:10:58.939262 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wfccr" podUID="c84a7003-1537-4470-8094-ea10a89a86a2" Sep 17 10:10:59.938823 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:59.938716 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v9xpw" Sep 17 10:10:59.938823 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:10:59.938727 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fppwm" Sep 17 10:10:59.939047 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:10:59.938872 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v9xpw" podUID="c304d2b0-b1ce-4a29-86b1-341d5db70d4f" Sep 17 10:10:59.939047 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:10:59.938929 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fppwm" podUID="f6509b04-a69d-49d5-a08c-aabdd8bf6c86" Sep 17 10:11:00.939274 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:00.939234 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wfccr" Sep 17 10:11:00.939743 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:11:00.939387 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wfccr" podUID="c84a7003-1537-4470-8094-ea10a89a86a2" Sep 17 10:11:01.942730 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:01.942699 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fppwm" Sep 17 10:11:01.943180 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:01.942699 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v9xpw" Sep 17 10:11:01.943180 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:11:01.942831 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fppwm" podUID="f6509b04-a69d-49d5-a08c-aabdd8bf6c86" Sep 17 10:11:01.943180 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:11:01.942901 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v9xpw" podUID="c304d2b0-b1ce-4a29-86b1-341d5db70d4f" Sep 17 10:11:02.939144 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:02.939109 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wfccr" Sep 17 10:11:02.939324 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:11:02.939217 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wfccr" podUID="c84a7003-1537-4470-8094-ea10a89a86a2" Sep 17 10:11:03.558302 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:03.558264 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c84a7003-1537-4470-8094-ea10a89a86a2-metrics-certs\") pod \"network-metrics-daemon-wfccr\" (UID: \"c84a7003-1537-4470-8094-ea10a89a86a2\") " pod="openshift-multus/network-metrics-daemon-wfccr" Sep 17 10:11:03.558711 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:03.558316 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c304d2b0-b1ce-4a29-86b1-341d5db70d4f-original-pull-secret\") pod \"global-pull-secret-syncer-v9xpw\" (UID: \"c304d2b0-b1ce-4a29-86b1-341d5db70d4f\") " pod="kube-system/global-pull-secret-syncer-v9xpw" Sep 17 10:11:03.558711 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:11:03.558423 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 10:11:03.558711 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:11:03.558494 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c84a7003-1537-4470-8094-ea10a89a86a2-metrics-certs podName:c84a7003-1537-4470-8094-ea10a89a86a2 nodeName:}" failed. No retries permitted until 2026-09-17 10:11:19.558476116 +0000 UTC m=+34.107033224 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c84a7003-1537-4470-8094-ea10a89a86a2-metrics-certs") pod "network-metrics-daemon-wfccr" (UID: "c84a7003-1537-4470-8094-ea10a89a86a2") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 10:11:03.558711 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:11:03.558426 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 17 10:11:03.558711 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:11:03.558570 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c304d2b0-b1ce-4a29-86b1-341d5db70d4f-original-pull-secret podName:c304d2b0-b1ce-4a29-86b1-341d5db70d4f nodeName:}" failed. No retries permitted until 2026-09-17 10:11:19.558556825 +0000 UTC m=+34.107113915 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c304d2b0-b1ce-4a29-86b1-341d5db70d4f-original-pull-secret") pod "global-pull-secret-syncer-v9xpw" (UID: "c304d2b0-b1ce-4a29-86b1-341d5db70d4f") : object "kube-system"/"original-pull-secret" not registered Sep 17 10:11:03.760574 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:03.760536 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxnn2\" (UniqueName: \"kubernetes.io/projected/f6509b04-a69d-49d5-a08c-aabdd8bf6c86-kube-api-access-nxnn2\") pod \"network-check-target-fppwm\" (UID: \"f6509b04-a69d-49d5-a08c-aabdd8bf6c86\") " pod="openshift-network-diagnostics/network-check-target-fppwm" Sep 17 10:11:03.760737 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:11:03.760714 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 17 10:11:03.760808 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:11:03.760743 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 17 10:11:03.760808 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:11:03.760755 2581 projected.go:196] Error preparing data for projected volume kube-api-access-nxnn2 for pod openshift-network-diagnostics/network-check-target-fppwm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 10:11:03.760886 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:11:03.760834 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f6509b04-a69d-49d5-a08c-aabdd8bf6c86-kube-api-access-nxnn2 podName:f6509b04-a69d-49d5-a08c-aabdd8bf6c86 nodeName:}" failed. No retries permitted until 2026-09-17 10:11:19.760817718 +0000 UTC m=+34.309374826 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-nxnn2" (UniqueName: "kubernetes.io/projected/f6509b04-a69d-49d5-a08c-aabdd8bf6c86-kube-api-access-nxnn2") pod "network-check-target-fppwm" (UID: "f6509b04-a69d-49d5-a08c-aabdd8bf6c86") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 10:11:03.944332 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:03.944249 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v9xpw" Sep 17 10:11:03.944480 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:03.944249 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fppwm" Sep 17 10:11:03.944480 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:11:03.944378 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v9xpw" podUID="c304d2b0-b1ce-4a29-86b1-341d5db70d4f" Sep 17 10:11:03.944592 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:11:03.944506 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fppwm" podUID="f6509b04-a69d-49d5-a08c-aabdd8bf6c86" Sep 17 10:11:04.938610 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:04.938573 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wfccr" Sep 17 10:11:04.939072 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:11:04.938700 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wfccr" podUID="c84a7003-1537-4470-8094-ea10a89a86a2" Sep 17 10:11:05.939849 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:05.939379 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fppwm" Sep 17 10:11:05.939849 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:11:05.939499 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fppwm" podUID="f6509b04-a69d-49d5-a08c-aabdd8bf6c86" Sep 17 10:11:05.939849 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:05.939590 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v9xpw" Sep 17 10:11:05.939849 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:11:05.939691 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v9xpw" podUID="c304d2b0-b1ce-4a29-86b1-341d5db70d4f" Sep 17 10:11:06.939058 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:06.939035 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wfccr" Sep 17 10:11:06.939157 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:11:06.939133 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wfccr" podUID="c84a7003-1537-4470-8094-ea10a89a86a2" Sep 17 10:11:07.099895 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:07.099162 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-jkx62" event={"ID":"de933a7f-7eae-4e7b-870f-aa9c56565481","Type":"ContainerStarted","Data":"831c8ecf43be6c797385ff6956f88ace3f95318c05e302901fa67fe2f27ce1ba"} Sep 17 10:11:07.115946 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:07.114914 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-203.ec2.internal" podStartSLOduration=20.114893378 podStartE2EDuration="20.114893378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 10:10:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 10:10:51.063780459 +0000 UTC m=+5.612337574" watchObservedRunningTime="2026-09-17 10:11:07.114893378 +0000 UTC m=+21.663450490" Sep 17 10:11:07.938595 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:07.938300 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fppwm" Sep 17 10:11:07.938799 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:07.938348 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v9xpw" Sep 17 10:11:07.938799 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:11:07.938661 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fppwm" podUID="f6509b04-a69d-49d5-a08c-aabdd8bf6c86" Sep 17 10:11:07.938799 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:11:07.938722 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v9xpw" podUID="c304d2b0-b1ce-4a29-86b1-341d5db70d4f" Sep 17 10:11:08.102614 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:08.102581 2581 generic.go:356] "Generic (PLEG): container finished" podID="a80abea0-a53c-4d9e-8934-7a7f073d71e3" containerID="d6eab8c1c038af2e341c3effe41fcaa671f6dba5804c800975ef6a8409489bde" exitCode=0 Sep 17 10:11:08.103499 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:08.102664 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4w7ff" event={"ID":"a80abea0-a53c-4d9e-8934-7a7f073d71e3","Type":"ContainerDied","Data":"d6eab8c1c038af2e341c3effe41fcaa671f6dba5804c800975ef6a8409489bde"} Sep 17 10:11:08.104359 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:08.104192 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-7f64n" event={"ID":"6f7323dd-d769-4f58-92d7-e8a36c2f2854","Type":"ContainerStarted","Data":"9c70cdfe58ba91d19cc7c557fa928c73bb3b3aa82c47582778790db8a4a5cdc4"} Sep 17 10:11:08.105793 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:08.105759 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kcztl" event={"ID":"606f20f7-c795-451a-b146-03f3a4f92684","Type":"ContainerStarted","Data":"54fe209a48114ca0bcebd7367d49e96e59727aa01bbdc1729330972b7c84cb90"} Sep 17 10:11:08.107170 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:08.107146 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-kg2hz" event={"ID":"e4b73c92-0ac6-4d8f-90d8-249cfbcadf4b","Type":"ContainerStarted","Data":"ef6c1bd4f3bdf1bfae0635367c8b707cb3b375d57bd62b22f162b0a8f7db2019"} Sep 17 10:11:08.108680 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:08.108660 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-nxbjh" event={"ID":"ae147838-ff96-4652-bf25-ca99bc25be02","Type":"ContainerStarted","Data":"19504ce3df6a12a331e36264f7561197225a658121bc057ea50f09271dda9748"} Sep 17 10:11:08.111618 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:08.111598 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9zbjf" event={"ID":"896c7cf8-3f20-4809-8f6d-a6852a068ad2","Type":"ContainerStarted","Data":"35195b61028e7be98eb6710102084a27dd433e1b759d4af7fc08f2c683e7431c"} Sep 17 10:11:08.111714 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:08.111667 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9zbjf" event={"ID":"896c7cf8-3f20-4809-8f6d-a6852a068ad2","Type":"ContainerStarted","Data":"de04195cbac3716079b73545c5740ec49cda5f1b9f22cc8fe709e6eaba52d14f"} Sep 17 10:11:08.111798 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:08.111720 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9zbjf" event={"ID":"896c7cf8-3f20-4809-8f6d-a6852a068ad2","Type":"ContainerStarted","Data":"9d6a12784f0c848a28bccd567fd49932c4ba196dceaa43a79accd15ca948055c"} Sep 17 10:11:08.111866 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:08.111799 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9zbjf" event={"ID":"896c7cf8-3f20-4809-8f6d-a6852a068ad2","Type":"ContainerStarted","Data":"1b4d4cb34160043ff3ce25acf1d52f743c177cafd9357afcc873893758be899c"} Sep 17 10:11:08.111866 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:08.111849 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9zbjf" event={"ID":"896c7cf8-3f20-4809-8f6d-a6852a068ad2","Type":"ContainerStarted","Data":"963ad579c77090bdb78c787841ec18b84b99fa61954dd58dfa90eec541797ced"} Sep 17 10:11:08.111961 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:08.111899 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9zbjf" event={"ID":"896c7cf8-3f20-4809-8f6d-a6852a068ad2","Type":"ContainerStarted","Data":"c953d9d178f3c756432e2260caaca56dd66b029b61fdb387cee594e19b169b58"} Sep 17 10:11:08.113062 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:08.113023 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-r2dcq" event={"ID":"2c409c2a-64a4-4164-8a14-c2a87597dd93","Type":"ContainerStarted","Data":"5c00eb955623eca915bcc6a03d13ae61cbb1ae1e08607cd8ec39f2bdf861be04"} Sep 17 10:11:08.128441 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:08.128397 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-jkx62" podStartSLOduration=11.631062382 podStartE2EDuration="22.12838162s" totalImagesPullingTime="10.497319238s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 10:10:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 10:11:07.115931043 +0000 UTC m=+21.664488159" watchObservedRunningTime="2026-09-17 10:11:08.12838162 +0000 UTC m=+22.676938730" Sep 17 10:11:08.141954 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:08.141926 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-jkx62" Sep 17 10:11:08.142854 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:08.142836 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-jkx62" Sep 17 10:11:08.151360 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:08.149689 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-7f64n" podStartSLOduration=4.202189937 podStartE2EDuration="22.149670707s" totalImagesPullingTime="17.94748077s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 10:10:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 10:11:08.14888905 +0000 UTC m=+22.697446162" watchObservedRunningTime="2026-09-17 10:11:08.149670707 +0000 UTC m=+22.698227822" Sep 17 10:11:08.169501 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:08.169454 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-nxbjh" podStartSLOduration=4.016012082 podStartE2EDuration="22.169439235s" totalImagesPullingTime="18.153427153s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 10:10:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 10:11:08.169292192 +0000 UTC m=+22.717849302" watchObservedRunningTime="2026-09-17 10:11:08.169439235 +0000 UTC m=+22.717996343" Sep 17 10:11:08.200447 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:08.200372 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-r2dcq" podStartSLOduration=4.220193789 podStartE2EDuration="22.200357202s" totalImagesPullingTime="17.980163413s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 10:10:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 10:11:08.200090164 +0000 UTC m=+22.748647274" watchObservedRunningTime="2026-09-17 10:11:08.200357202 +0000 UTC m=+22.748914312" Sep 17 10:11:08.221336 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:08.221283 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-kg2hz" podStartSLOduration=4.272859687 podStartE2EDuration="22.221267016s" totalImagesPullingTime="17.948407329s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 10:10:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 10:11:08.220723487 +0000 UTC m=+22.769280612" watchObservedRunningTime="2026-09-17 10:11:08.221267016 +0000 UTC m=+22.769824169" Sep 17 10:11:08.505958 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:08.505932 2581 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 17 10:11:08.886013 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:08.885912 2581 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-17T10:11:08.505950509Z","UUID":"7e8c9d14-5288-46a1-b7d7-a7e623aa13e8","Handler":null,"Name":"","Endpoint":""} Sep 17 10:11:08.887472 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:08.887449 2581 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 17 10:11:08.887472 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:08.887478 2581 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 17 10:11:08.939066 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:08.939029 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wfccr" Sep 17 10:11:08.939246 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:11:08.939179 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wfccr" podUID="c84a7003-1537-4470-8094-ea10a89a86a2" Sep 17 10:11:09.118745 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:09.118706 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kcztl" event={"ID":"606f20f7-c795-451a-b146-03f3a4f92684","Type":"ContainerStarted","Data":"7378e5103095006f61fccf7e5b452bae45b5202a46c5bb90f5956901983ca5d7"} Sep 17 10:11:09.120528 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:09.120500 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-g8lzm" event={"ID":"b506216d-c159-420e-9111-56acab7568df","Type":"ContainerStarted","Data":"6f24563804424d84006de49b72d4f5c5d11f1a4a06a669620291f99de02d94a0"} Sep 17 10:11:09.121853 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:09.121820 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-jkx62" Sep 17 10:11:09.122332 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:09.122314 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-jkx62" Sep 17 10:11:09.136615 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:09.136513 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-g8lzm" podStartSLOduration=5.156748212 podStartE2EDuration="23.136493957s" totalImagesPullingTime="17.979745745s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 10:10:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 10:11:09.136291623 +0000 UTC m=+23.684848735" watchObservedRunningTime="2026-09-17 10:11:09.136493957 +0000 UTC m=+23.685051068" Sep 17 10:11:09.938817 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:09.938719 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v9xpw" Sep 17 10:11:09.938817 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:09.938770 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fppwm" Sep 17 10:11:09.939010 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:11:09.938870 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v9xpw" podUID="c304d2b0-b1ce-4a29-86b1-341d5db70d4f" Sep 17 10:11:09.939066 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:11:09.939006 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fppwm" podUID="f6509b04-a69d-49d5-a08c-aabdd8bf6c86" Sep 17 10:11:10.126402 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:10.126362 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kcztl" event={"ID":"606f20f7-c795-451a-b146-03f3a4f92684","Type":"ContainerStarted","Data":"841f667ef66821d5923926f2bc8f9781ae4855ad85efcc268daa05b12e87e13c"} Sep 17 10:11:10.130867 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:10.130835 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9zbjf" event={"ID":"896c7cf8-3f20-4809-8f6d-a6852a068ad2","Type":"ContainerStarted","Data":"a74efa214243910ff0d257131326dd2b617ef86a82956ca6849c26c67df3c71b"} Sep 17 10:11:10.145530 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:10.145472 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kcztl" podStartSLOduration=3.889257117 podStartE2EDuration="24.14545483s" totalImagesPullingTime="20.256197713s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 10:10:46 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 10:11:10.145228374 +0000 UTC m=+24.693785487" watchObservedRunningTime="2026-09-17 10:11:10.14545483 +0000 UTC m=+24.694011940" Sep 17 10:11:10.939200 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:10.939164 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wfccr" Sep 17 10:11:10.939393 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:11:10.939302 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wfccr" podUID="c84a7003-1537-4470-8094-ea10a89a86a2" Sep 17 10:11:11.939274 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:11.939022 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fppwm" Sep 17 10:11:11.939710 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:11.939080 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v9xpw" Sep 17 10:11:11.939710 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:11:11.939382 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fppwm" podUID="f6509b04-a69d-49d5-a08c-aabdd8bf6c86" Sep 17 10:11:11.939710 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:11:11.939510 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v9xpw" podUID="c304d2b0-b1ce-4a29-86b1-341d5db70d4f" Sep 17 10:11:12.939212 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:12.939124 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wfccr" Sep 17 10:11:12.939358 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:11:12.939239 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wfccr" podUID="c84a7003-1537-4470-8094-ea10a89a86a2" Sep 17 10:11:13.141299 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:13.141269 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9zbjf" event={"ID":"896c7cf8-3f20-4809-8f6d-a6852a068ad2","Type":"ContainerStarted","Data":"f786260f6c368f9197b32a355bc96bee67b1789934adf7df7e78e60b4b9e79a3"} Sep 17 10:11:13.141601 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:13.141572 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9zbjf" Sep 17 10:11:13.143219 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:13.143192 2581 generic.go:356] "Generic (PLEG): container finished" podID="a80abea0-a53c-4d9e-8934-7a7f073d71e3" containerID="42f01b858b6186515337e64eb5f2982f22536ebcfe717a8f82290289d5a72f85" exitCode=0 Sep 17 10:11:13.143328 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:13.143245 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4w7ff" event={"ID":"a80abea0-a53c-4d9e-8934-7a7f073d71e3","Type":"ContainerDied","Data":"42f01b858b6186515337e64eb5f2982f22536ebcfe717a8f82290289d5a72f85"} Sep 17 10:11:13.157694 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:13.157669 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9zbjf" Sep 17 10:11:13.175755 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:13.174433 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-9zbjf" podStartSLOduration=9.181562672 podStartE2EDuration="27.174412672s" totalImagesPullingTime="17.99285s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 10:10:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 10:11:13.173393668 +0000 UTC m=+27.721950776" watchObservedRunningTime="2026-09-17 10:11:13.174412672 +0000 UTC m=+27.722969788" Sep 17 10:11:13.938907 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:13.938703 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fppwm" Sep 17 10:11:13.939085 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:13.938754 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v9xpw" Sep 17 10:11:13.939085 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:11:13.939010 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fppwm" podUID="f6509b04-a69d-49d5-a08c-aabdd8bf6c86" Sep 17 10:11:13.939085 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:11:13.939064 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v9xpw" podUID="c304d2b0-b1ce-4a29-86b1-341d5db70d4f" Sep 17 10:11:14.147025 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:14.146984 2581 generic.go:356] "Generic (PLEG): container finished" podID="a80abea0-a53c-4d9e-8934-7a7f073d71e3" containerID="ce9540bfad5c578d00af7fe866e7e48800fe9ad20871d1a4bcb294405794c667" exitCode=0 Sep 17 10:11:14.147437 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:14.147067 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4w7ff" event={"ID":"a80abea0-a53c-4d9e-8934-7a7f073d71e3","Type":"ContainerDied","Data":"ce9540bfad5c578d00af7fe866e7e48800fe9ad20871d1a4bcb294405794c667"} Sep 17 10:11:14.147437 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:14.147174 2581 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Sep 17 10:11:14.147672 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:14.147643 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9zbjf" Sep 17 10:11:14.163236 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:14.163212 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9zbjf" Sep 17 10:11:14.274282 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:14.274254 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-v9xpw"] Sep 17 10:11:14.274438 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:14.274365 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v9xpw" Sep 17 10:11:14.274495 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:11:14.274478 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v9xpw" podUID="c304d2b0-b1ce-4a29-86b1-341d5db70d4f" Sep 17 10:11:14.274897 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:14.274869 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-fppwm"] Sep 17 10:11:14.275031 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:14.274960 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fppwm" Sep 17 10:11:14.275113 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:11:14.275075 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fppwm" podUID="f6509b04-a69d-49d5-a08c-aabdd8bf6c86" Sep 17 10:11:14.275848 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:14.275825 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-wfccr"] Sep 17 10:11:14.275947 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:14.275933 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wfccr" Sep 17 10:11:14.276049 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:11:14.276030 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wfccr" podUID="c84a7003-1537-4470-8094-ea10a89a86a2" Sep 17 10:11:15.152238 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:15.152157 2581 generic.go:356] "Generic (PLEG): container finished" podID="a80abea0-a53c-4d9e-8934-7a7f073d71e3" containerID="d10e5c4b7a5ac3578cd88c88c78c2d6e4fe0e99ea04e0042222671a369f05b79" exitCode=0 Sep 17 10:11:15.152569 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:15.152247 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4w7ff" event={"ID":"a80abea0-a53c-4d9e-8934-7a7f073d71e3","Type":"ContainerDied","Data":"d10e5c4b7a5ac3578cd88c88c78c2d6e4fe0e99ea04e0042222671a369f05b79"} Sep 17 10:11:15.152569 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:15.152320 2581 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Sep 17 10:11:15.940031 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:15.939840 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fppwm" Sep 17 10:11:15.940150 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:15.939840 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wfccr" Sep 17 10:11:15.940150 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:11:15.940129 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fppwm" podUID="f6509b04-a69d-49d5-a08c-aabdd8bf6c86" Sep 17 10:11:15.940378 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:11:15.940232 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wfccr" podUID="c84a7003-1537-4470-8094-ea10a89a86a2" Sep 17 10:11:15.940378 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:15.939869 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v9xpw" Sep 17 10:11:15.940378 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:11:15.940348 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v9xpw" podUID="c304d2b0-b1ce-4a29-86b1-341d5db70d4f" Sep 17 10:11:16.155040 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:16.155008 2581 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Sep 17 10:11:16.699117 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:16.699040 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9zbjf" Sep 17 10:11:17.172895 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:17.172842 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-9zbjf" podUID="896c7cf8-3f20-4809-8f6d-a6852a068ad2" containerName="ovnkube-controller" probeResult="failure" output="" Sep 17 10:11:17.938363 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:17.938337 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v9xpw" Sep 17 10:11:17.938509 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:17.938338 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wfccr" Sep 17 10:11:17.938509 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:11:17.938458 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v9xpw" podUID="c304d2b0-b1ce-4a29-86b1-341d5db70d4f" Sep 17 10:11:17.938628 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:11:17.938548 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wfccr" podUID="c84a7003-1537-4470-8094-ea10a89a86a2" Sep 17 10:11:17.938628 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:17.938338 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fppwm" Sep 17 10:11:17.938731 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:11:17.938633 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fppwm" podUID="f6509b04-a69d-49d5-a08c-aabdd8bf6c86" Sep 17 10:11:19.599142 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:19.599115 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c304d2b0-b1ce-4a29-86b1-341d5db70d4f-original-pull-secret\") pod \"global-pull-secret-syncer-v9xpw\" (UID: \"c304d2b0-b1ce-4a29-86b1-341d5db70d4f\") " pod="kube-system/global-pull-secret-syncer-v9xpw" Sep 17 10:11:19.599661 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:19.599202 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c84a7003-1537-4470-8094-ea10a89a86a2-metrics-certs\") pod \"network-metrics-daemon-wfccr\" (UID: \"c84a7003-1537-4470-8094-ea10a89a86a2\") " pod="openshift-multus/network-metrics-daemon-wfccr" Sep 17 10:11:19.599661 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:11:19.599264 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 17 10:11:19.599661 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:11:19.599294 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 10:11:19.599661 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:11:19.599334 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c304d2b0-b1ce-4a29-86b1-341d5db70d4f-original-pull-secret podName:c304d2b0-b1ce-4a29-86b1-341d5db70d4f nodeName:}" failed. No retries permitted until 2026-09-17 10:11:51.599317666 +0000 UTC m=+66.147874755 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c304d2b0-b1ce-4a29-86b1-341d5db70d4f-original-pull-secret") pod "global-pull-secret-syncer-v9xpw" (UID: "c304d2b0-b1ce-4a29-86b1-341d5db70d4f") : object "kube-system"/"original-pull-secret" not registered Sep 17 10:11:19.599661 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:11:19.599349 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c84a7003-1537-4470-8094-ea10a89a86a2-metrics-certs podName:c84a7003-1537-4470-8094-ea10a89a86a2 nodeName:}" failed. No retries permitted until 2026-09-17 10:11:51.599342118 +0000 UTC m=+66.147899207 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c84a7003-1537-4470-8094-ea10a89a86a2-metrics-certs") pod "network-metrics-daemon-wfccr" (UID: "c84a7003-1537-4470-8094-ea10a89a86a2") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 10:11:19.734495 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:19.734469 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-203.ec2.internal" event="NodeReady" Sep 17 10:11:19.734664 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:19.734607 2581 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 17 10:11:19.783319 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:19.783278 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-lpfcm"] Sep 17 10:11:19.796708 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:19.796561 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-ds6sz"] Sep 17 10:11:19.796868 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:19.796822 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-lpfcm" Sep 17 10:11:19.799716 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:19.799692 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 17 10:11:19.800169 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:19.799967 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 17 10:11:19.800169 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:19.799981 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 17 10:11:19.800314 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:19.800234 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-bg5f5\"" Sep 17 10:11:19.801056 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:19.801036 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxnn2\" (UniqueName: \"kubernetes.io/projected/f6509b04-a69d-49d5-a08c-aabdd8bf6c86-kube-api-access-nxnn2\") pod \"network-check-target-fppwm\" (UID: \"f6509b04-a69d-49d5-a08c-aabdd8bf6c86\") " pod="openshift-network-diagnostics/network-check-target-fppwm" Sep 17 10:11:19.801167 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:11:19.801144 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 17 10:11:19.801167 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:11:19.801159 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 17 10:11:19.801167 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:11:19.801167 2581 projected.go:196] Error preparing data for projected volume kube-api-access-nxnn2 for pod openshift-network-diagnostics/network-check-target-fppwm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 10:11:19.801367 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:11:19.801222 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f6509b04-a69d-49d5-a08c-aabdd8bf6c86-kube-api-access-nxnn2 podName:f6509b04-a69d-49d5-a08c-aabdd8bf6c86 nodeName:}" failed. No retries permitted until 2026-09-17 10:11:51.801204517 +0000 UTC m=+66.349761606 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-nxnn2" (UniqueName: "kubernetes.io/projected/f6509b04-a69d-49d5-a08c-aabdd8bf6c86-kube-api-access-nxnn2") pod "network-check-target-fppwm" (UID: "f6509b04-a69d-49d5-a08c-aabdd8bf6c86") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 10:11:19.811468 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:19.811452 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-lpfcm"] Sep 17 10:11:19.811547 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:19.811477 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-ds6sz"] Sep 17 10:11:19.811588 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:19.811554 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ds6sz" Sep 17 10:11:19.814316 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:19.814295 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 17 10:11:19.814409 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:19.814300 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 17 10:11:19.814409 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:19.814343 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-sglxr\"" Sep 17 10:11:19.901576 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:19.901548 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mx4ss\" (UniqueName: \"kubernetes.io/projected/026a85ed-57e7-474d-9f05-c8824906de72-kube-api-access-mx4ss\") pod \"dns-default-ds6sz\" (UID: \"026a85ed-57e7-474d-9f05-c8824906de72\") " pod="openshift-dns/dns-default-ds6sz" Sep 17 10:11:19.901801 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:19.901592 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/026a85ed-57e7-474d-9f05-c8824906de72-tmp-dir\") pod \"dns-default-ds6sz\" (UID: \"026a85ed-57e7-474d-9f05-c8824906de72\") " pod="openshift-dns/dns-default-ds6sz" Sep 17 10:11:19.901801 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:19.901685 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhl58\" (UniqueName: \"kubernetes.io/projected/2769e90f-7bdb-4fce-bfa6-ebf02a0963da-kube-api-access-qhl58\") pod \"ingress-canary-lpfcm\" (UID: \"2769e90f-7bdb-4fce-bfa6-ebf02a0963da\") " pod="openshift-ingress-canary/ingress-canary-lpfcm" Sep 17 10:11:19.901801 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:19.901738 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/026a85ed-57e7-474d-9f05-c8824906de72-config-volume\") pod \"dns-default-ds6sz\" (UID: \"026a85ed-57e7-474d-9f05-c8824906de72\") " pod="openshift-dns/dns-default-ds6sz" Sep 17 10:11:19.901948 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:19.901811 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/026a85ed-57e7-474d-9f05-c8824906de72-metrics-tls\") pod \"dns-default-ds6sz\" (UID: \"026a85ed-57e7-474d-9f05-c8824906de72\") " pod="openshift-dns/dns-default-ds6sz" Sep 17 10:11:19.901948 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:19.901838 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2769e90f-7bdb-4fce-bfa6-ebf02a0963da-cert\") pod \"ingress-canary-lpfcm\" (UID: \"2769e90f-7bdb-4fce-bfa6-ebf02a0963da\") " pod="openshift-ingress-canary/ingress-canary-lpfcm" Sep 17 10:11:19.938895 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:19.938865 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v9xpw" Sep 17 10:11:19.939024 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:19.938866 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wfccr" Sep 17 10:11:19.939024 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:19.938962 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fppwm" Sep 17 10:11:19.942322 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:19.942297 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-thcs5\"" Sep 17 10:11:19.942444 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:19.942335 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 17 10:11:19.942444 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:19.942366 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 17 10:11:19.942547 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:19.942473 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-bxncx\"" Sep 17 10:11:19.942547 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:19.942543 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 17 10:11:19.942642 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:19.942623 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 17 10:11:20.002770 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:20.002704 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhl58\" (UniqueName: \"kubernetes.io/projected/2769e90f-7bdb-4fce-bfa6-ebf02a0963da-kube-api-access-qhl58\") pod \"ingress-canary-lpfcm\" (UID: \"2769e90f-7bdb-4fce-bfa6-ebf02a0963da\") " pod="openshift-ingress-canary/ingress-canary-lpfcm" Sep 17 10:11:20.002770 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:20.002738 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/026a85ed-57e7-474d-9f05-c8824906de72-config-volume\") pod \"dns-default-ds6sz\" (UID: \"026a85ed-57e7-474d-9f05-c8824906de72\") " pod="openshift-dns/dns-default-ds6sz" Sep 17 10:11:20.002770 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:20.002762 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/026a85ed-57e7-474d-9f05-c8824906de72-metrics-tls\") pod \"dns-default-ds6sz\" (UID: \"026a85ed-57e7-474d-9f05-c8824906de72\") " pod="openshift-dns/dns-default-ds6sz" Sep 17 10:11:20.003081 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:20.002792 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2769e90f-7bdb-4fce-bfa6-ebf02a0963da-cert\") pod \"ingress-canary-lpfcm\" (UID: \"2769e90f-7bdb-4fce-bfa6-ebf02a0963da\") " pod="openshift-ingress-canary/ingress-canary-lpfcm" Sep 17 10:11:20.003081 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:20.002821 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mx4ss\" (UniqueName: \"kubernetes.io/projected/026a85ed-57e7-474d-9f05-c8824906de72-kube-api-access-mx4ss\") pod \"dns-default-ds6sz\" (UID: \"026a85ed-57e7-474d-9f05-c8824906de72\") " pod="openshift-dns/dns-default-ds6sz" Sep 17 10:11:20.003081 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:20.002862 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/026a85ed-57e7-474d-9f05-c8824906de72-tmp-dir\") pod \"dns-default-ds6sz\" (UID: \"026a85ed-57e7-474d-9f05-c8824906de72\") " pod="openshift-dns/dns-default-ds6sz" Sep 17 10:11:20.003081 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:11:20.002893 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 17 10:11:20.003081 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:11:20.002965 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/026a85ed-57e7-474d-9f05-c8824906de72-metrics-tls podName:026a85ed-57e7-474d-9f05-c8824906de72 nodeName:}" failed. No retries permitted until 2026-09-17 10:11:20.502945261 +0000 UTC m=+35.051502350 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/026a85ed-57e7-474d-9f05-c8824906de72-metrics-tls") pod "dns-default-ds6sz" (UID: "026a85ed-57e7-474d-9f05-c8824906de72") : secret "dns-default-metrics-tls" not found Sep 17 10:11:20.003298 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:20.003155 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/026a85ed-57e7-474d-9f05-c8824906de72-tmp-dir\") pod \"dns-default-ds6sz\" (UID: \"026a85ed-57e7-474d-9f05-c8824906de72\") " pod="openshift-dns/dns-default-ds6sz" Sep 17 10:11:20.003350 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:20.003332 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/026a85ed-57e7-474d-9f05-c8824906de72-config-volume\") pod \"dns-default-ds6sz\" (UID: \"026a85ed-57e7-474d-9f05-c8824906de72\") " pod="openshift-dns/dns-default-ds6sz" Sep 17 10:11:20.006774 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:20.006756 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2769e90f-7bdb-4fce-bfa6-ebf02a0963da-cert\") pod \"ingress-canary-lpfcm\" (UID: \"2769e90f-7bdb-4fce-bfa6-ebf02a0963da\") " pod="openshift-ingress-canary/ingress-canary-lpfcm" Sep 17 10:11:20.012359 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:20.012330 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mx4ss\" (UniqueName: \"kubernetes.io/projected/026a85ed-57e7-474d-9f05-c8824906de72-kube-api-access-mx4ss\") pod \"dns-default-ds6sz\" (UID: \"026a85ed-57e7-474d-9f05-c8824906de72\") " pod="openshift-dns/dns-default-ds6sz" Sep 17 10:11:20.012444 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:20.012407 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhl58\" (UniqueName: \"kubernetes.io/projected/2769e90f-7bdb-4fce-bfa6-ebf02a0963da-kube-api-access-qhl58\") pod \"ingress-canary-lpfcm\" (UID: \"2769e90f-7bdb-4fce-bfa6-ebf02a0963da\") " pod="openshift-ingress-canary/ingress-canary-lpfcm" Sep 17 10:11:20.106048 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:20.106021 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-lpfcm" Sep 17 10:11:20.167142 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:20.167110 2581 generic.go:356] "Generic (PLEG): container finished" podID="a80abea0-a53c-4d9e-8934-7a7f073d71e3" containerID="d605b9bd7d2705811c58d967939f6da8c7517b5b2b87b729028833818558b2f3" exitCode=0 Sep 17 10:11:20.167259 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:20.167164 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4w7ff" event={"ID":"a80abea0-a53c-4d9e-8934-7a7f073d71e3","Type":"ContainerDied","Data":"d605b9bd7d2705811c58d967939f6da8c7517b5b2b87b729028833818558b2f3"} Sep 17 10:11:20.430164 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:20.429993 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-lpfcm"] Sep 17 10:11:20.433297 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:11:20.433271 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2769e90f_7bdb_4fce_bfa6_ebf02a0963da.slice/crio-1de05f6f5af1657141a501ec7eccd00ea6105f1f6c1b6a449aef6fbec59f3d88 WatchSource:0}: Error finding container 1de05f6f5af1657141a501ec7eccd00ea6105f1f6c1b6a449aef6fbec59f3d88: Status 404 returned error can't find the container with id 1de05f6f5af1657141a501ec7eccd00ea6105f1f6c1b6a449aef6fbec59f3d88 Sep 17 10:11:20.507961 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:20.507930 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/026a85ed-57e7-474d-9f05-c8824906de72-metrics-tls\") pod \"dns-default-ds6sz\" (UID: \"026a85ed-57e7-474d-9f05-c8824906de72\") " pod="openshift-dns/dns-default-ds6sz" Sep 17 10:11:20.508059 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:11:20.508041 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 17 10:11:20.508123 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:11:20.508113 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/026a85ed-57e7-474d-9f05-c8824906de72-metrics-tls podName:026a85ed-57e7-474d-9f05-c8824906de72 nodeName:}" failed. No retries permitted until 2026-09-17 10:11:21.508095122 +0000 UTC m=+36.056652212 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/026a85ed-57e7-474d-9f05-c8824906de72-metrics-tls") pod "dns-default-ds6sz" (UID: "026a85ed-57e7-474d-9f05-c8824906de72") : secret "dns-default-metrics-tls" not found Sep 17 10:11:21.173946 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:21.173913 2581 generic.go:356] "Generic (PLEG): container finished" podID="a80abea0-a53c-4d9e-8934-7a7f073d71e3" containerID="f135e474da33c7040433b25ce3b3941b69dcf5e228e3c91b35d7844c2fae6653" exitCode=0 Sep 17 10:11:21.174652 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:21.173995 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4w7ff" event={"ID":"a80abea0-a53c-4d9e-8934-7a7f073d71e3","Type":"ContainerDied","Data":"f135e474da33c7040433b25ce3b3941b69dcf5e228e3c91b35d7844c2fae6653"} Sep 17 10:11:21.175302 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:21.175278 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-lpfcm" event={"ID":"2769e90f-7bdb-4fce-bfa6-ebf02a0963da","Type":"ContainerStarted","Data":"1de05f6f5af1657141a501ec7eccd00ea6105f1f6c1b6a449aef6fbec59f3d88"} Sep 17 10:11:21.514744 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:21.514704 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/026a85ed-57e7-474d-9f05-c8824906de72-metrics-tls\") pod \"dns-default-ds6sz\" (UID: \"026a85ed-57e7-474d-9f05-c8824906de72\") " pod="openshift-dns/dns-default-ds6sz" Sep 17 10:11:21.514935 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:11:21.514905 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 17 10:11:21.515018 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:11:21.514979 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/026a85ed-57e7-474d-9f05-c8824906de72-metrics-tls podName:026a85ed-57e7-474d-9f05-c8824906de72 nodeName:}" failed. No retries permitted until 2026-09-17 10:11:23.514959136 +0000 UTC m=+38.063516227 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/026a85ed-57e7-474d-9f05-c8824906de72-metrics-tls") pod "dns-default-ds6sz" (UID: "026a85ed-57e7-474d-9f05-c8824906de72") : secret "dns-default-metrics-tls" not found Sep 17 10:11:22.181596 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:22.181558 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4w7ff" event={"ID":"a80abea0-a53c-4d9e-8934-7a7f073d71e3","Type":"ContainerStarted","Data":"43451d9f762c2bd6a68265796196c4c5174b74b1b7cced82aee8a84f88be973f"} Sep 17 10:11:22.206600 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:22.206548 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-4w7ff" podStartSLOduration=8.482617185 podStartE2EDuration="36.206532524s" totalImagesPullingTime="27.723915339s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 10:10:46 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 10:11:22.204440765 +0000 UTC m=+36.752997877" watchObservedRunningTime="2026-09-17 10:11:22.206532524 +0000 UTC m=+36.755089634" Sep 17 10:11:23.186405 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:23.186192 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-lpfcm" event={"ID":"2769e90f-7bdb-4fce-bfa6-ebf02a0963da","Type":"ContainerStarted","Data":"cf2168271e09b917392c4b985c3305b41cc11ad136de26c7785d3057df1f6b46"} Sep 17 10:11:23.202790 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:23.202730 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-lpfcm" podStartSLOduration=1.671245165 podStartE2EDuration="4.202709039s" totalImagesPullingTime="2.531463874s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 10:11:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 10:11:23.201982537 +0000 UTC m=+37.750539648" watchObservedRunningTime="2026-09-17 10:11:23.202709039 +0000 UTC m=+37.751266150" Sep 17 10:11:23.531275 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:23.531240 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/026a85ed-57e7-474d-9f05-c8824906de72-metrics-tls\") pod \"dns-default-ds6sz\" (UID: \"026a85ed-57e7-474d-9f05-c8824906de72\") " pod="openshift-dns/dns-default-ds6sz" Sep 17 10:11:23.531462 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:11:23.531401 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 17 10:11:23.531528 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:11:23.531473 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/026a85ed-57e7-474d-9f05-c8824906de72-metrics-tls podName:026a85ed-57e7-474d-9f05-c8824906de72 nodeName:}" failed. No retries permitted until 2026-09-17 10:11:27.531453778 +0000 UTC m=+42.080010872 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/026a85ed-57e7-474d-9f05-c8824906de72-metrics-tls") pod "dns-default-ds6sz" (UID: "026a85ed-57e7-474d-9f05-c8824906de72") : secret "dns-default-metrics-tls" not found Sep 17 10:11:27.560249 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:27.560199 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/026a85ed-57e7-474d-9f05-c8824906de72-metrics-tls\") pod \"dns-default-ds6sz\" (UID: \"026a85ed-57e7-474d-9f05-c8824906de72\") " pod="openshift-dns/dns-default-ds6sz" Sep 17 10:11:27.560648 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:11:27.560350 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 17 10:11:27.560648 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:11:27.560415 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/026a85ed-57e7-474d-9f05-c8824906de72-metrics-tls podName:026a85ed-57e7-474d-9f05-c8824906de72 nodeName:}" failed. No retries permitted until 2026-09-17 10:11:35.560396293 +0000 UTC m=+50.108953391 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/026a85ed-57e7-474d-9f05-c8824906de72-metrics-tls") pod "dns-default-ds6sz" (UID: "026a85ed-57e7-474d-9f05-c8824906de72") : secret "dns-default-metrics-tls" not found Sep 17 10:11:35.614819 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:35.614758 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/026a85ed-57e7-474d-9f05-c8824906de72-metrics-tls\") pod \"dns-default-ds6sz\" (UID: \"026a85ed-57e7-474d-9f05-c8824906de72\") " pod="openshift-dns/dns-default-ds6sz" Sep 17 10:11:35.615246 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:11:35.614906 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 17 10:11:35.615246 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:11:35.614985 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/026a85ed-57e7-474d-9f05-c8824906de72-metrics-tls podName:026a85ed-57e7-474d-9f05-c8824906de72 nodeName:}" failed. No retries permitted until 2026-09-17 10:11:51.614969254 +0000 UTC m=+66.163526373 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/026a85ed-57e7-474d-9f05-c8824906de72-metrics-tls") pod "dns-default-ds6sz" (UID: "026a85ed-57e7-474d-9f05-c8824906de72") : secret "dns-default-metrics-tls" not found Sep 17 10:11:46.464715 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:46.464677 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5844b856db-fdc72"] Sep 17 10:11:46.467479 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:46.467456 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-fdc72" Sep 17 10:11:46.470985 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:46.470650 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 17 10:11:46.470985 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:46.470701 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Sep 17 10:11:46.471830 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:46.471796 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 17 10:11:46.471830 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:46.471796 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-mtmxj\"" Sep 17 10:11:46.472004 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:46.471885 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Sep 17 10:11:46.477152 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:46.477129 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Sep 17 10:11:46.480305 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:46.480284 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-fdc72"] Sep 17 10:11:46.490877 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:46.490851 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a13ce06b-1f1b-4614-a6f1-7e95dbd6f1d4-trusted-ca-bundle\") pod \"insights-operator-5844b856db-fdc72\" (UID: \"a13ce06b-1f1b-4614-a6f1-7e95dbd6f1d4\") " pod="openshift-insights/insights-operator-5844b856db-fdc72" Sep 17 10:11:46.491006 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:46.490885 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5crz\" (UniqueName: \"kubernetes.io/projected/a13ce06b-1f1b-4614-a6f1-7e95dbd6f1d4-kube-api-access-c5crz\") pod \"insights-operator-5844b856db-fdc72\" (UID: \"a13ce06b-1f1b-4614-a6f1-7e95dbd6f1d4\") " pod="openshift-insights/insights-operator-5844b856db-fdc72" Sep 17 10:11:46.491006 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:46.490929 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a13ce06b-1f1b-4614-a6f1-7e95dbd6f1d4-tmp\") pod \"insights-operator-5844b856db-fdc72\" (UID: \"a13ce06b-1f1b-4614-a6f1-7e95dbd6f1d4\") " pod="openshift-insights/insights-operator-5844b856db-fdc72" Sep 17 10:11:46.491105 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:46.490995 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/a13ce06b-1f1b-4614-a6f1-7e95dbd6f1d4-snapshots\") pod \"insights-operator-5844b856db-fdc72\" (UID: \"a13ce06b-1f1b-4614-a6f1-7e95dbd6f1d4\") " pod="openshift-insights/insights-operator-5844b856db-fdc72" Sep 17 10:11:46.491105 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:46.491065 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a13ce06b-1f1b-4614-a6f1-7e95dbd6f1d4-service-ca-bundle\") pod \"insights-operator-5844b856db-fdc72\" (UID: \"a13ce06b-1f1b-4614-a6f1-7e95dbd6f1d4\") " pod="openshift-insights/insights-operator-5844b856db-fdc72" Sep 17 10:11:46.491105 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:46.491085 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a13ce06b-1f1b-4614-a6f1-7e95dbd6f1d4-serving-cert\") pod \"insights-operator-5844b856db-fdc72\" (UID: \"a13ce06b-1f1b-4614-a6f1-7e95dbd6f1d4\") " pod="openshift-insights/insights-operator-5844b856db-fdc72" Sep 17 10:11:46.566610 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:46.566580 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-d6hwv"] Sep 17 10:11:46.568306 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:46.568286 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-d6hwv" Sep 17 10:11:46.569714 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:46.569693 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-b9wnf"] Sep 17 10:11:46.571196 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:46.571176 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-hs69z\"" Sep 17 10:11:46.571282 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:46.571197 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Sep 17 10:11:46.571282 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:46.571217 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Sep 17 10:11:46.571374 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:46.571299 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-b9wnf" Sep 17 10:11:46.574338 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:46.574317 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Sep 17 10:11:46.574526 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:46.574488 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Sep 17 10:11:46.574741 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:46.574721 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Sep 17 10:11:46.574873 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:46.574751 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-5z585\"" Sep 17 10:11:46.574926 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:46.574880 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Sep 17 10:11:46.575808 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:46.575789 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-85865fbbd8-bkwfh"] Sep 17 10:11:46.577373 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:46.577356 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-85865fbbd8-bkwfh" Sep 17 10:11:46.580122 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:46.580093 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Sep 17 10:11:46.580275 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:46.580234 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 17 10:11:46.580359 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:46.580313 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Sep 17 10:11:46.580359 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:46.580355 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Sep 17 10:11:46.580509 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:46.580362 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Sep 17 10:11:46.580509 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:46.580315 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-42fjm\"" Sep 17 10:11:46.580612 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:46.580599 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 17 10:11:46.580612 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:46.580604 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-d6hwv"] Sep 17 10:11:46.580673 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:46.580611 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Sep 17 10:11:46.591076 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:46.591048 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-85865fbbd8-bkwfh"] Sep 17 10:11:46.591359 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:46.591340 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a13ce06b-1f1b-4614-a6f1-7e95dbd6f1d4-service-ca-bundle\") pod \"insights-operator-5844b856db-fdc72\" (UID: \"a13ce06b-1f1b-4614-a6f1-7e95dbd6f1d4\") " pod="openshift-insights/insights-operator-5844b856db-fdc72" Sep 17 10:11:46.591445 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:46.591376 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a13ce06b-1f1b-4614-a6f1-7e95dbd6f1d4-serving-cert\") pod \"insights-operator-5844b856db-fdc72\" (UID: \"a13ce06b-1f1b-4614-a6f1-7e95dbd6f1d4\") " pod="openshift-insights/insights-operator-5844b856db-fdc72" Sep 17 10:11:46.591445 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:46.591403 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e18215ee-6cb8-49d5-bb04-aaa79b65f012-service-ca-bundle\") pod \"router-default-85865fbbd8-bkwfh\" (UID: \"e18215ee-6cb8-49d5-bb04-aaa79b65f012\") " pod="openshift-ingress/router-default-85865fbbd8-bkwfh" Sep 17 10:11:46.591558 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:46.591462 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85q22\" (UniqueName: \"kubernetes.io/projected/1ddcc945-9657-4614-bc59-f0cfcf613c1e-kube-api-access-85q22\") pod \"volume-data-source-validator-86695b7cc7-d6hwv\" (UID: \"1ddcc945-9657-4614-bc59-f0cfcf613c1e\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-d6hwv" Sep 17 10:11:46.591636 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:46.591613 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/762d3e10-7b7d-4ce9-9773-90a4fe41b474-serving-cert\") pod \"console-operator-696884fb9c-b9wnf\" (UID: \"762d3e10-7b7d-4ce9-9773-90a4fe41b474\") " pod="openshift-console-operator/console-operator-696884fb9c-b9wnf" Sep 17 10:11:46.591694 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:46.591671 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/762d3e10-7b7d-4ce9-9773-90a4fe41b474-config\") pod \"console-operator-696884fb9c-b9wnf\" (UID: \"762d3e10-7b7d-4ce9-9773-90a4fe41b474\") " pod="openshift-console-operator/console-operator-696884fb9c-b9wnf" Sep 17 10:11:46.591752 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:46.591696 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/e18215ee-6cb8-49d5-bb04-aaa79b65f012-default-certificate\") pod \"router-default-85865fbbd8-bkwfh\" (UID: \"e18215ee-6cb8-49d5-bb04-aaa79b65f012\") " pod="openshift-ingress/router-default-85865fbbd8-bkwfh" Sep 17 10:11:46.591752 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:46.591738 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a13ce06b-1f1b-4614-a6f1-7e95dbd6f1d4-trusted-ca-bundle\") pod \"insights-operator-5844b856db-fdc72\" (UID: \"a13ce06b-1f1b-4614-a6f1-7e95dbd6f1d4\") " pod="openshift-insights/insights-operator-5844b856db-fdc72" Sep 17 10:11:46.591882 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:46.591768 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgkc7\" (UniqueName: \"kubernetes.io/projected/e18215ee-6cb8-49d5-bb04-aaa79b65f012-kube-api-access-mgkc7\") pod \"router-default-85865fbbd8-bkwfh\" (UID: \"e18215ee-6cb8-49d5-bb04-aaa79b65f012\") " pod="openshift-ingress/router-default-85865fbbd8-bkwfh" Sep 17 10:11:46.591882 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:46.591818 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5crz\" (UniqueName: \"kubernetes.io/projected/a13ce06b-1f1b-4614-a6f1-7e95dbd6f1d4-kube-api-access-c5crz\") pod \"insights-operator-5844b856db-fdc72\" (UID: \"a13ce06b-1f1b-4614-a6f1-7e95dbd6f1d4\") " pod="openshift-insights/insights-operator-5844b856db-fdc72" Sep 17 10:11:46.591882 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:46.591844 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grlrh\" (UniqueName: \"kubernetes.io/projected/762d3e10-7b7d-4ce9-9773-90a4fe41b474-kube-api-access-grlrh\") pod \"console-operator-696884fb9c-b9wnf\" (UID: \"762d3e10-7b7d-4ce9-9773-90a4fe41b474\") " pod="openshift-console-operator/console-operator-696884fb9c-b9wnf" Sep 17 10:11:46.591882 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:46.591877 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/e18215ee-6cb8-49d5-bb04-aaa79b65f012-stats-auth\") pod \"router-default-85865fbbd8-bkwfh\" (UID: \"e18215ee-6cb8-49d5-bb04-aaa79b65f012\") " pod="openshift-ingress/router-default-85865fbbd8-bkwfh" Sep 17 10:11:46.592083 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:46.592030 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/762d3e10-7b7d-4ce9-9773-90a4fe41b474-trusted-ca\") pod \"console-operator-696884fb9c-b9wnf\" (UID: \"762d3e10-7b7d-4ce9-9773-90a4fe41b474\") " pod="openshift-console-operator/console-operator-696884fb9c-b9wnf" Sep 17 10:11:46.592083 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:46.592053 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a13ce06b-1f1b-4614-a6f1-7e95dbd6f1d4-tmp\") pod \"insights-operator-5844b856db-fdc72\" (UID: \"a13ce06b-1f1b-4614-a6f1-7e95dbd6f1d4\") " pod="openshift-insights/insights-operator-5844b856db-fdc72" Sep 17 10:11:46.592083 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:46.592073 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/a13ce06b-1f1b-4614-a6f1-7e95dbd6f1d4-snapshots\") pod \"insights-operator-5844b856db-fdc72\" (UID: \"a13ce06b-1f1b-4614-a6f1-7e95dbd6f1d4\") " pod="openshift-insights/insights-operator-5844b856db-fdc72" Sep 17 10:11:46.592224 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:46.592091 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e18215ee-6cb8-49d5-bb04-aaa79b65f012-metrics-certs\") pod \"router-default-85865fbbd8-bkwfh\" (UID: \"e18215ee-6cb8-49d5-bb04-aaa79b65f012\") " pod="openshift-ingress/router-default-85865fbbd8-bkwfh" Sep 17 10:11:46.592224 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:46.592124 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a13ce06b-1f1b-4614-a6f1-7e95dbd6f1d4-service-ca-bundle\") pod \"insights-operator-5844b856db-fdc72\" (UID: \"a13ce06b-1f1b-4614-a6f1-7e95dbd6f1d4\") " pod="openshift-insights/insights-operator-5844b856db-fdc72" Sep 17 10:11:46.592584 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:46.592567 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a13ce06b-1f1b-4614-a6f1-7e95dbd6f1d4-tmp\") pod \"insights-operator-5844b856db-fdc72\" (UID: \"a13ce06b-1f1b-4614-a6f1-7e95dbd6f1d4\") " pod="openshift-insights/insights-operator-5844b856db-fdc72" Sep 17 10:11:46.592584 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:46.592578 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/a13ce06b-1f1b-4614-a6f1-7e95dbd6f1d4-snapshots\") pod \"insights-operator-5844b856db-fdc72\" (UID: \"a13ce06b-1f1b-4614-a6f1-7e95dbd6f1d4\") " pod="openshift-insights/insights-operator-5844b856db-fdc72" Sep 17 10:11:46.593028 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:46.593009 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a13ce06b-1f1b-4614-a6f1-7e95dbd6f1d4-trusted-ca-bundle\") pod \"insights-operator-5844b856db-fdc72\" (UID: \"a13ce06b-1f1b-4614-a6f1-7e95dbd6f1d4\") " pod="openshift-insights/insights-operator-5844b856db-fdc72" Sep 17 10:11:46.595493 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:46.595473 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a13ce06b-1f1b-4614-a6f1-7e95dbd6f1d4-serving-cert\") pod \"insights-operator-5844b856db-fdc72\" (UID: \"a13ce06b-1f1b-4614-a6f1-7e95dbd6f1d4\") " pod="openshift-insights/insights-operator-5844b856db-fdc72" Sep 17 10:11:46.596800 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:46.596756 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-b9wnf"] Sep 17 10:11:46.608917 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:46.608890 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5crz\" (UniqueName: \"kubernetes.io/projected/a13ce06b-1f1b-4614-a6f1-7e95dbd6f1d4-kube-api-access-c5crz\") pod \"insights-operator-5844b856db-fdc72\" (UID: \"a13ce06b-1f1b-4614-a6f1-7e95dbd6f1d4\") " pod="openshift-insights/insights-operator-5844b856db-fdc72" Sep 17 10:11:46.693166 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:46.693130 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgkc7\" (UniqueName: \"kubernetes.io/projected/e18215ee-6cb8-49d5-bb04-aaa79b65f012-kube-api-access-mgkc7\") pod \"router-default-85865fbbd8-bkwfh\" (UID: \"e18215ee-6cb8-49d5-bb04-aaa79b65f012\") " pod="openshift-ingress/router-default-85865fbbd8-bkwfh" Sep 17 10:11:46.693166 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:46.693168 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grlrh\" (UniqueName: \"kubernetes.io/projected/762d3e10-7b7d-4ce9-9773-90a4fe41b474-kube-api-access-grlrh\") pod \"console-operator-696884fb9c-b9wnf\" (UID: \"762d3e10-7b7d-4ce9-9773-90a4fe41b474\") " pod="openshift-console-operator/console-operator-696884fb9c-b9wnf" Sep 17 10:11:46.693398 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:46.693201 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/e18215ee-6cb8-49d5-bb04-aaa79b65f012-stats-auth\") pod \"router-default-85865fbbd8-bkwfh\" (UID: \"e18215ee-6cb8-49d5-bb04-aaa79b65f012\") " pod="openshift-ingress/router-default-85865fbbd8-bkwfh" Sep 17 10:11:46.693398 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:46.693264 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/762d3e10-7b7d-4ce9-9773-90a4fe41b474-trusted-ca\") pod \"console-operator-696884fb9c-b9wnf\" (UID: \"762d3e10-7b7d-4ce9-9773-90a4fe41b474\") " pod="openshift-console-operator/console-operator-696884fb9c-b9wnf" Sep 17 10:11:46.693398 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:46.693297 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e18215ee-6cb8-49d5-bb04-aaa79b65f012-metrics-certs\") pod \"router-default-85865fbbd8-bkwfh\" (UID: \"e18215ee-6cb8-49d5-bb04-aaa79b65f012\") " pod="openshift-ingress/router-default-85865fbbd8-bkwfh" Sep 17 10:11:46.693398 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:46.693340 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e18215ee-6cb8-49d5-bb04-aaa79b65f012-service-ca-bundle\") pod \"router-default-85865fbbd8-bkwfh\" (UID: \"e18215ee-6cb8-49d5-bb04-aaa79b65f012\") " pod="openshift-ingress/router-default-85865fbbd8-bkwfh" Sep 17 10:11:46.693398 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:46.693365 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85q22\" (UniqueName: \"kubernetes.io/projected/1ddcc945-9657-4614-bc59-f0cfcf613c1e-kube-api-access-85q22\") pod \"volume-data-source-validator-86695b7cc7-d6hwv\" (UID: \"1ddcc945-9657-4614-bc59-f0cfcf613c1e\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-d6hwv" Sep 17 10:11:46.693655 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:46.693407 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/762d3e10-7b7d-4ce9-9773-90a4fe41b474-serving-cert\") pod \"console-operator-696884fb9c-b9wnf\" (UID: \"762d3e10-7b7d-4ce9-9773-90a4fe41b474\") " pod="openshift-console-operator/console-operator-696884fb9c-b9wnf" Sep 17 10:11:46.693655 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:46.693435 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/762d3e10-7b7d-4ce9-9773-90a4fe41b474-config\") pod \"console-operator-696884fb9c-b9wnf\" (UID: \"762d3e10-7b7d-4ce9-9773-90a4fe41b474\") " pod="openshift-console-operator/console-operator-696884fb9c-b9wnf" Sep 17 10:11:46.693655 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:46.693458 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/e18215ee-6cb8-49d5-bb04-aaa79b65f012-default-certificate\") pod \"router-default-85865fbbd8-bkwfh\" (UID: \"e18215ee-6cb8-49d5-bb04-aaa79b65f012\") " pod="openshift-ingress/router-default-85865fbbd8-bkwfh" Sep 17 10:11:46.693655 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:11:46.693512 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e18215ee-6cb8-49d5-bb04-aaa79b65f012-service-ca-bundle podName:e18215ee-6cb8-49d5-bb04-aaa79b65f012 nodeName:}" failed. No retries permitted until 2026-09-17 10:11:47.193490405 +0000 UTC m=+61.742047517 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e18215ee-6cb8-49d5-bb04-aaa79b65f012-service-ca-bundle") pod "router-default-85865fbbd8-bkwfh" (UID: "e18215ee-6cb8-49d5-bb04-aaa79b65f012") : configmap references non-existent config key: service-ca.crt Sep 17 10:11:46.693655 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:11:46.693563 2581 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 17 10:11:46.693655 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:11:46.693630 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e18215ee-6cb8-49d5-bb04-aaa79b65f012-metrics-certs podName:e18215ee-6cb8-49d5-bb04-aaa79b65f012 nodeName:}" failed. No retries permitted until 2026-09-17 10:11:47.193603477 +0000 UTC m=+61.742160579 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e18215ee-6cb8-49d5-bb04-aaa79b65f012-metrics-certs") pod "router-default-85865fbbd8-bkwfh" (UID: "e18215ee-6cb8-49d5-bb04-aaa79b65f012") : secret "router-metrics-certs-default" not found Sep 17 10:11:46.694199 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:46.694173 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/762d3e10-7b7d-4ce9-9773-90a4fe41b474-config\") pod \"console-operator-696884fb9c-b9wnf\" (UID: \"762d3e10-7b7d-4ce9-9773-90a4fe41b474\") " pod="openshift-console-operator/console-operator-696884fb9c-b9wnf" Sep 17 10:11:46.694323 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:46.694303 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/762d3e10-7b7d-4ce9-9773-90a4fe41b474-trusted-ca\") pod \"console-operator-696884fb9c-b9wnf\" (UID: \"762d3e10-7b7d-4ce9-9773-90a4fe41b474\") " pod="openshift-console-operator/console-operator-696884fb9c-b9wnf" Sep 17 10:11:46.695705 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:46.695677 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/e18215ee-6cb8-49d5-bb04-aaa79b65f012-stats-auth\") pod \"router-default-85865fbbd8-bkwfh\" (UID: \"e18215ee-6cb8-49d5-bb04-aaa79b65f012\") " pod="openshift-ingress/router-default-85865fbbd8-bkwfh" Sep 17 10:11:46.695837 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:46.695796 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/e18215ee-6cb8-49d5-bb04-aaa79b65f012-default-certificate\") pod \"router-default-85865fbbd8-bkwfh\" (UID: \"e18215ee-6cb8-49d5-bb04-aaa79b65f012\") " pod="openshift-ingress/router-default-85865fbbd8-bkwfh" Sep 17 10:11:46.695947 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:46.695926 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/762d3e10-7b7d-4ce9-9773-90a4fe41b474-serving-cert\") pod \"console-operator-696884fb9c-b9wnf\" (UID: \"762d3e10-7b7d-4ce9-9773-90a4fe41b474\") " pod="openshift-console-operator/console-operator-696884fb9c-b9wnf" Sep 17 10:11:46.707198 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:46.707166 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgkc7\" (UniqueName: \"kubernetes.io/projected/e18215ee-6cb8-49d5-bb04-aaa79b65f012-kube-api-access-mgkc7\") pod \"router-default-85865fbbd8-bkwfh\" (UID: \"e18215ee-6cb8-49d5-bb04-aaa79b65f012\") " pod="openshift-ingress/router-default-85865fbbd8-bkwfh" Sep 17 10:11:46.707985 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:46.707963 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-grlrh\" (UniqueName: \"kubernetes.io/projected/762d3e10-7b7d-4ce9-9773-90a4fe41b474-kube-api-access-grlrh\") pod \"console-operator-696884fb9c-b9wnf\" (UID: \"762d3e10-7b7d-4ce9-9773-90a4fe41b474\") " pod="openshift-console-operator/console-operator-696884fb9c-b9wnf" Sep 17 10:11:46.707985 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:46.707969 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-85q22\" (UniqueName: \"kubernetes.io/projected/1ddcc945-9657-4614-bc59-f0cfcf613c1e-kube-api-access-85q22\") pod \"volume-data-source-validator-86695b7cc7-d6hwv\" (UID: \"1ddcc945-9657-4614-bc59-f0cfcf613c1e\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-d6hwv" Sep 17 10:11:46.770957 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:46.770923 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-sv877"] Sep 17 10:11:46.773867 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:46.773847 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-mjspg"] Sep 17 10:11:46.774008 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:46.773967 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-sv877" Sep 17 10:11:46.775648 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:46.775630 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-mjspg" Sep 17 10:11:46.776542 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:46.776524 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-gj6vg\"" Sep 17 10:11:46.776634 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:46.776549 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Sep 17 10:11:46.776634 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:46.776597 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Sep 17 10:11:46.776749 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:46.776726 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Sep 17 10:11:46.778199 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:46.778175 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Sep 17 10:11:46.778294 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:46.778228 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Sep 17 10:11:46.778294 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:46.778277 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-blpvl\"" Sep 17 10:11:46.778403 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:46.778294 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-fdc72" Sep 17 10:11:46.778463 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:46.778435 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Sep 17 10:11:46.778573 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:46.778558 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Sep 17 10:11:46.783773 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:46.783753 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-sv877"] Sep 17 10:11:46.786485 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:46.786465 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-mjspg"] Sep 17 10:11:46.794693 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:46.794671 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/da9abd04-2db6-44cc-b6bb-c5d2999ec8c4-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-mjspg\" (UID: \"da9abd04-2db6-44cc-b6bb-c5d2999ec8c4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-mjspg" Sep 17 10:11:46.794831 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:46.794731 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hc24l\" (UniqueName: \"kubernetes.io/projected/c802c004-f487-44df-b73a-fb50df751388-kube-api-access-hc24l\") pod \"cluster-samples-operator-86499c89c4-sv877\" (UID: \"c802c004-f487-44df-b73a-fb50df751388\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-sv877" Sep 17 10:11:46.794831 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:46.794758 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c802c004-f487-44df-b73a-fb50df751388-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-sv877\" (UID: \"c802c004-f487-44df-b73a-fb50df751388\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-sv877" Sep 17 10:11:46.794949 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:46.794846 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/da9abd04-2db6-44cc-b6bb-c5d2999ec8c4-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-mjspg\" (UID: \"da9abd04-2db6-44cc-b6bb-c5d2999ec8c4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-mjspg" Sep 17 10:11:46.794949 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:46.794935 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vq7qh\" (UniqueName: \"kubernetes.io/projected/da9abd04-2db6-44cc-b6bb-c5d2999ec8c4-kube-api-access-vq7qh\") pod \"kube-storage-version-migrator-operator-6c8f84f897-mjspg\" (UID: \"da9abd04-2db6-44cc-b6bb-c5d2999ec8c4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-mjspg" Sep 17 10:11:46.879093 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:46.879055 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-d6hwv" Sep 17 10:11:46.887371 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:46.887352 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-b9wnf" Sep 17 10:11:46.896890 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:46.896859 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hc24l\" (UniqueName: \"kubernetes.io/projected/c802c004-f487-44df-b73a-fb50df751388-kube-api-access-hc24l\") pod \"cluster-samples-operator-86499c89c4-sv877\" (UID: \"c802c004-f487-44df-b73a-fb50df751388\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-sv877" Sep 17 10:11:46.896994 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:46.896916 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c802c004-f487-44df-b73a-fb50df751388-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-sv877\" (UID: \"c802c004-f487-44df-b73a-fb50df751388\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-sv877" Sep 17 10:11:46.896994 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:46.896953 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/da9abd04-2db6-44cc-b6bb-c5d2999ec8c4-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-mjspg\" (UID: \"da9abd04-2db6-44cc-b6bb-c5d2999ec8c4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-mjspg" Sep 17 10:11:46.897101 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:46.897035 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vq7qh\" (UniqueName: \"kubernetes.io/projected/da9abd04-2db6-44cc-b6bb-c5d2999ec8c4-kube-api-access-vq7qh\") pod \"kube-storage-version-migrator-operator-6c8f84f897-mjspg\" (UID: \"da9abd04-2db6-44cc-b6bb-c5d2999ec8c4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-mjspg" Sep 17 10:11:46.897101 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:46.897092 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/da9abd04-2db6-44cc-b6bb-c5d2999ec8c4-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-mjspg\" (UID: \"da9abd04-2db6-44cc-b6bb-c5d2999ec8c4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-mjspg" Sep 17 10:11:46.897201 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:11:46.897148 2581 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 17 10:11:46.897250 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:11:46.897224 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c802c004-f487-44df-b73a-fb50df751388-samples-operator-tls podName:c802c004-f487-44df-b73a-fb50df751388 nodeName:}" failed. No retries permitted until 2026-09-17 10:11:47.397201989 +0000 UTC m=+61.945759080 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c802c004-f487-44df-b73a-fb50df751388-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-sv877" (UID: "c802c004-f487-44df-b73a-fb50df751388") : secret "samples-operator-tls" not found Sep 17 10:11:46.897768 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:46.897732 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/da9abd04-2db6-44cc-b6bb-c5d2999ec8c4-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-mjspg\" (UID: \"da9abd04-2db6-44cc-b6bb-c5d2999ec8c4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-mjspg" Sep 17 10:11:46.899737 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:46.899716 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/da9abd04-2db6-44cc-b6bb-c5d2999ec8c4-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-mjspg\" (UID: \"da9abd04-2db6-44cc-b6bb-c5d2999ec8c4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-mjspg" Sep 17 10:11:46.907969 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:46.907916 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vq7qh\" (UniqueName: \"kubernetes.io/projected/da9abd04-2db6-44cc-b6bb-c5d2999ec8c4-kube-api-access-vq7qh\") pod \"kube-storage-version-migrator-operator-6c8f84f897-mjspg\" (UID: \"da9abd04-2db6-44cc-b6bb-c5d2999ec8c4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-mjspg" Sep 17 10:11:46.909527 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:46.909504 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hc24l\" (UniqueName: \"kubernetes.io/projected/c802c004-f487-44df-b73a-fb50df751388-kube-api-access-hc24l\") pod \"cluster-samples-operator-86499c89c4-sv877\" (UID: \"c802c004-f487-44df-b73a-fb50df751388\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-sv877" Sep 17 10:11:47.048616 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:47.048456 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-fdc72"] Sep 17 10:11:47.051019 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:11:47.050991 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda13ce06b_1f1b_4614_a6f1_7e95dbd6f1d4.slice/crio-573b43df7a4a8b476f39d0b10500662f61ae7d0df8d137071d20c99341ebfe02 WatchSource:0}: Error finding container 573b43df7a4a8b476f39d0b10500662f61ae7d0df8d137071d20c99341ebfe02: Status 404 returned error can't find the container with id 573b43df7a4a8b476f39d0b10500662f61ae7d0df8d137071d20c99341ebfe02 Sep 17 10:11:47.091229 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:47.091197 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-mjspg" Sep 17 10:11:47.153467 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:47.153389 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-d6hwv"] Sep 17 10:11:47.157190 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:11:47.157165 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ddcc945_9657_4614_bc59_f0cfcf613c1e.slice/crio-e9681e99d5bfd025cb65574a1083d2bc6cf9b0789858db2b67a0473cd76f4938 WatchSource:0}: Error finding container e9681e99d5bfd025cb65574a1083d2bc6cf9b0789858db2b67a0473cd76f4938: Status 404 returned error can't find the container with id e9681e99d5bfd025cb65574a1083d2bc6cf9b0789858db2b67a0473cd76f4938 Sep 17 10:11:47.167597 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:47.167570 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9zbjf" Sep 17 10:11:47.171732 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:47.171707 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-b9wnf"] Sep 17 10:11:47.174676 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:11:47.174652 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod762d3e10_7b7d_4ce9_9773_90a4fe41b474.slice/crio-8cdfca093d2956bf7b131617e8600ac4d65f9d8a430f9167a319010cea318058 WatchSource:0}: Error finding container 8cdfca093d2956bf7b131617e8600ac4d65f9d8a430f9167a319010cea318058: Status 404 returned error can't find the container with id 8cdfca093d2956bf7b131617e8600ac4d65f9d8a430f9167a319010cea318058 Sep 17 10:11:47.199748 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:47.199720 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e18215ee-6cb8-49d5-bb04-aaa79b65f012-metrics-certs\") pod \"router-default-85865fbbd8-bkwfh\" (UID: \"e18215ee-6cb8-49d5-bb04-aaa79b65f012\") " pod="openshift-ingress/router-default-85865fbbd8-bkwfh" Sep 17 10:11:47.199902 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:11:47.199885 2581 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 17 10:11:47.199940 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:47.199915 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e18215ee-6cb8-49d5-bb04-aaa79b65f012-service-ca-bundle\") pod \"router-default-85865fbbd8-bkwfh\" (UID: \"e18215ee-6cb8-49d5-bb04-aaa79b65f012\") " pod="openshift-ingress/router-default-85865fbbd8-bkwfh" Sep 17 10:11:47.199974 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:11:47.199947 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e18215ee-6cb8-49d5-bb04-aaa79b65f012-metrics-certs podName:e18215ee-6cb8-49d5-bb04-aaa79b65f012 nodeName:}" failed. No retries permitted until 2026-09-17 10:11:48.199928927 +0000 UTC m=+62.748486036 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e18215ee-6cb8-49d5-bb04-aaa79b65f012-metrics-certs") pod "router-default-85865fbbd8-bkwfh" (UID: "e18215ee-6cb8-49d5-bb04-aaa79b65f012") : secret "router-metrics-certs-default" not found Sep 17 10:11:47.200027 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:11:47.200018 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e18215ee-6cb8-49d5-bb04-aaa79b65f012-service-ca-bundle podName:e18215ee-6cb8-49d5-bb04-aaa79b65f012 nodeName:}" failed. No retries permitted until 2026-09-17 10:11:48.200003004 +0000 UTC m=+62.748560113 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e18215ee-6cb8-49d5-bb04-aaa79b65f012-service-ca-bundle") pod "router-default-85865fbbd8-bkwfh" (UID: "e18215ee-6cb8-49d5-bb04-aaa79b65f012") : configmap references non-existent config key: service-ca.crt Sep 17 10:11:47.243410 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:47.243375 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-d6hwv" event={"ID":"1ddcc945-9657-4614-bc59-f0cfcf613c1e","Type":"ContainerStarted","Data":"e9681e99d5bfd025cb65574a1083d2bc6cf9b0789858db2b67a0473cd76f4938"} Sep 17 10:11:47.244524 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:47.244503 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-b9wnf" event={"ID":"762d3e10-7b7d-4ce9-9773-90a4fe41b474","Type":"ContainerStarted","Data":"8cdfca093d2956bf7b131617e8600ac4d65f9d8a430f9167a319010cea318058"} Sep 17 10:11:47.245559 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:47.245539 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-fdc72" event={"ID":"a13ce06b-1f1b-4614-a6f1-7e95dbd6f1d4","Type":"ContainerStarted","Data":"573b43df7a4a8b476f39d0b10500662f61ae7d0df8d137071d20c99341ebfe02"} Sep 17 10:11:47.357848 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:47.357755 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-mjspg"] Sep 17 10:11:47.360387 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:11:47.360363 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda9abd04_2db6_44cc_b6bb_c5d2999ec8c4.slice/crio-fa5cafb7d3da0c8e6ef567d474964eef311e346943adb11712a76fe3545e26a6 WatchSource:0}: Error finding container fa5cafb7d3da0c8e6ef567d474964eef311e346943adb11712a76fe3545e26a6: Status 404 returned error can't find the container with id fa5cafb7d3da0c8e6ef567d474964eef311e346943adb11712a76fe3545e26a6 Sep 17 10:11:47.402010 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:47.401979 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c802c004-f487-44df-b73a-fb50df751388-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-sv877\" (UID: \"c802c004-f487-44df-b73a-fb50df751388\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-sv877" Sep 17 10:11:47.402228 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:11:47.402087 2581 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 17 10:11:47.402228 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:11:47.402144 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c802c004-f487-44df-b73a-fb50df751388-samples-operator-tls podName:c802c004-f487-44df-b73a-fb50df751388 nodeName:}" failed. No retries permitted until 2026-09-17 10:11:48.402127162 +0000 UTC m=+62.950684257 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c802c004-f487-44df-b73a-fb50df751388-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-sv877" (UID: "c802c004-f487-44df-b73a-fb50df751388") : secret "samples-operator-tls" not found Sep 17 10:11:48.209444 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:48.209403 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e18215ee-6cb8-49d5-bb04-aaa79b65f012-metrics-certs\") pod \"router-default-85865fbbd8-bkwfh\" (UID: \"e18215ee-6cb8-49d5-bb04-aaa79b65f012\") " pod="openshift-ingress/router-default-85865fbbd8-bkwfh" Sep 17 10:11:48.209969 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:48.209482 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e18215ee-6cb8-49d5-bb04-aaa79b65f012-service-ca-bundle\") pod \"router-default-85865fbbd8-bkwfh\" (UID: \"e18215ee-6cb8-49d5-bb04-aaa79b65f012\") " pod="openshift-ingress/router-default-85865fbbd8-bkwfh" Sep 17 10:11:48.209969 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:11:48.209564 2581 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 17 10:11:48.209969 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:11:48.209649 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e18215ee-6cb8-49d5-bb04-aaa79b65f012-metrics-certs podName:e18215ee-6cb8-49d5-bb04-aaa79b65f012 nodeName:}" failed. No retries permitted until 2026-09-17 10:11:50.209627059 +0000 UTC m=+64.758184153 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e18215ee-6cb8-49d5-bb04-aaa79b65f012-metrics-certs") pod "router-default-85865fbbd8-bkwfh" (UID: "e18215ee-6cb8-49d5-bb04-aaa79b65f012") : secret "router-metrics-certs-default" not found Sep 17 10:11:48.209969 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:11:48.209681 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e18215ee-6cb8-49d5-bb04-aaa79b65f012-service-ca-bundle podName:e18215ee-6cb8-49d5-bb04-aaa79b65f012 nodeName:}" failed. No retries permitted until 2026-09-17 10:11:50.209664765 +0000 UTC m=+64.758221858 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e18215ee-6cb8-49d5-bb04-aaa79b65f012-service-ca-bundle") pod "router-default-85865fbbd8-bkwfh" (UID: "e18215ee-6cb8-49d5-bb04-aaa79b65f012") : configmap references non-existent config key: service-ca.crt Sep 17 10:11:48.250471 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:48.250364 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-mjspg" event={"ID":"da9abd04-2db6-44cc-b6bb-c5d2999ec8c4","Type":"ContainerStarted","Data":"fa5cafb7d3da0c8e6ef567d474964eef311e346943adb11712a76fe3545e26a6"} Sep 17 10:11:48.410918 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:48.410883 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c802c004-f487-44df-b73a-fb50df751388-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-sv877\" (UID: \"c802c004-f487-44df-b73a-fb50df751388\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-sv877" Sep 17 10:11:48.411106 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:11:48.411084 2581 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 17 10:11:48.411183 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:11:48.411159 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c802c004-f487-44df-b73a-fb50df751388-samples-operator-tls podName:c802c004-f487-44df-b73a-fb50df751388 nodeName:}" failed. No retries permitted until 2026-09-17 10:11:50.411140346 +0000 UTC m=+64.959697458 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c802c004-f487-44df-b73a-fb50df751388-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-sv877" (UID: "c802c004-f487-44df-b73a-fb50df751388") : secret "samples-operator-tls" not found Sep 17 10:11:49.254656 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:49.254616 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-d6hwv" event={"ID":"1ddcc945-9657-4614-bc59-f0cfcf613c1e","Type":"ContainerStarted","Data":"5a84d3946a7afeb46fe6de3962f89bb67b07eb89c97be623da0cd08bf34dff63"} Sep 17 10:11:49.271821 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:49.271752 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-d6hwv" podStartSLOduration=2.203275609 podStartE2EDuration="3.271735027s" totalImagesPullingTime="1.068459418s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 10:11:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 10:11:49.271256409 +0000 UTC m=+63.819813522" watchObservedRunningTime="2026-09-17 10:11:49.271735027 +0000 UTC m=+63.820292154" Sep 17 10:11:50.227461 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:50.227422 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e18215ee-6cb8-49d5-bb04-aaa79b65f012-metrics-certs\") pod \"router-default-85865fbbd8-bkwfh\" (UID: \"e18215ee-6cb8-49d5-bb04-aaa79b65f012\") " pod="openshift-ingress/router-default-85865fbbd8-bkwfh" Sep 17 10:11:50.227678 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:50.227505 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e18215ee-6cb8-49d5-bb04-aaa79b65f012-service-ca-bundle\") pod \"router-default-85865fbbd8-bkwfh\" (UID: \"e18215ee-6cb8-49d5-bb04-aaa79b65f012\") " pod="openshift-ingress/router-default-85865fbbd8-bkwfh" Sep 17 10:11:50.227678 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:11:50.227601 2581 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 17 10:11:50.227678 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:11:50.227675 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e18215ee-6cb8-49d5-bb04-aaa79b65f012-metrics-certs podName:e18215ee-6cb8-49d5-bb04-aaa79b65f012 nodeName:}" failed. No retries permitted until 2026-09-17 10:11:54.227655349 +0000 UTC m=+68.776212439 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e18215ee-6cb8-49d5-bb04-aaa79b65f012-metrics-certs") pod "router-default-85865fbbd8-bkwfh" (UID: "e18215ee-6cb8-49d5-bb04-aaa79b65f012") : secret "router-metrics-certs-default" not found Sep 17 10:11:50.227678 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:11:50.227705 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e18215ee-6cb8-49d5-bb04-aaa79b65f012-service-ca-bundle podName:e18215ee-6cb8-49d5-bb04-aaa79b65f012 nodeName:}" failed. No retries permitted until 2026-09-17 10:11:54.227689457 +0000 UTC m=+68.776246560 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e18215ee-6cb8-49d5-bb04-aaa79b65f012-service-ca-bundle") pod "router-default-85865fbbd8-bkwfh" (UID: "e18215ee-6cb8-49d5-bb04-aaa79b65f012") : configmap references non-existent config key: service-ca.crt Sep 17 10:11:50.259290 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:50.259254 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-mjspg" event={"ID":"da9abd04-2db6-44cc-b6bb-c5d2999ec8c4","Type":"ContainerStarted","Data":"89582ce03c7bf2530b70eb6a12035ec9057ee99d65658b4be38079f84d8e310c"} Sep 17 10:11:50.261357 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:50.261331 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-b9wnf_762d3e10-7b7d-4ce9-9773-90a4fe41b474/console-operator/0.log" Sep 17 10:11:50.261482 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:50.261378 2581 generic.go:356] "Generic (PLEG): container finished" podID="762d3e10-7b7d-4ce9-9773-90a4fe41b474" containerID="32bd194893d92f15c43885c624f3d052aaf24261ee63b6e5b6c27f819352bc52" exitCode=255 Sep 17 10:11:50.261482 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:50.261460 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-b9wnf" event={"ID":"762d3e10-7b7d-4ce9-9773-90a4fe41b474","Type":"ContainerDied","Data":"32bd194893d92f15c43885c624f3d052aaf24261ee63b6e5b6c27f819352bc52"} Sep 17 10:11:50.261628 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:50.261614 2581 scope.go:117] "RemoveContainer" containerID="32bd194893d92f15c43885c624f3d052aaf24261ee63b6e5b6c27f819352bc52" Sep 17 10:11:50.277767 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:50.277731 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-mjspg" podStartSLOduration=2.088897627 podStartE2EDuration="4.277715938s" totalImagesPullingTime="2.188818311s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 10:11:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 10:11:50.27672216 +0000 UTC m=+64.825279271" watchObservedRunningTime="2026-09-17 10:11:50.277715938 +0000 UTC m=+64.826273049" Sep 17 10:11:50.429434 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:50.429403 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c802c004-f487-44df-b73a-fb50df751388-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-sv877\" (UID: \"c802c004-f487-44df-b73a-fb50df751388\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-sv877" Sep 17 10:11:50.429662 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:11:50.429593 2581 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 17 10:11:50.429756 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:11:50.429673 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c802c004-f487-44df-b73a-fb50df751388-samples-operator-tls podName:c802c004-f487-44df-b73a-fb50df751388 nodeName:}" failed. No retries permitted until 2026-09-17 10:11:54.429652526 +0000 UTC m=+68.978209629 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c802c004-f487-44df-b73a-fb50df751388-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-sv877" (UID: "c802c004-f487-44df-b73a-fb50df751388") : secret "samples-operator-tls" not found Sep 17 10:11:51.265900 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:51.265872 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-b9wnf_762d3e10-7b7d-4ce9-9773-90a4fe41b474/console-operator/1.log" Sep 17 10:11:51.266267 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:51.266256 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-b9wnf_762d3e10-7b7d-4ce9-9773-90a4fe41b474/console-operator/0.log" Sep 17 10:11:51.266325 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:51.266288 2581 generic.go:356] "Generic (PLEG): container finished" podID="762d3e10-7b7d-4ce9-9773-90a4fe41b474" containerID="4ce48daa3f4f6488190f4a4717a8a6b0b6637089749dbcfdad5d78b78e4b6b43" exitCode=255 Sep 17 10:11:51.266358 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:51.266321 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-b9wnf" event={"ID":"762d3e10-7b7d-4ce9-9773-90a4fe41b474","Type":"ContainerDied","Data":"4ce48daa3f4f6488190f4a4717a8a6b0b6637089749dbcfdad5d78b78e4b6b43"} Sep 17 10:11:51.266408 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:51.266396 2581 scope.go:117] "RemoveContainer" containerID="32bd194893d92f15c43885c624f3d052aaf24261ee63b6e5b6c27f819352bc52" Sep 17 10:11:51.266611 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:51.266594 2581 scope.go:117] "RemoveContainer" containerID="4ce48daa3f4f6488190f4a4717a8a6b0b6637089749dbcfdad5d78b78e4b6b43" Sep 17 10:11:51.266836 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:11:51.266811 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-b9wnf_openshift-console-operator(762d3e10-7b7d-4ce9-9773-90a4fe41b474)\"" pod="openshift-console-operator/console-operator-696884fb9c-b9wnf" podUID="762d3e10-7b7d-4ce9-9773-90a4fe41b474" Sep 17 10:11:51.640027 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:51.639936 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c84a7003-1537-4470-8094-ea10a89a86a2-metrics-certs\") pod \"network-metrics-daemon-wfccr\" (UID: \"c84a7003-1537-4470-8094-ea10a89a86a2\") " pod="openshift-multus/network-metrics-daemon-wfccr" Sep 17 10:11:51.640027 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:51.639984 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c304d2b0-b1ce-4a29-86b1-341d5db70d4f-original-pull-secret\") pod \"global-pull-secret-syncer-v9xpw\" (UID: \"c304d2b0-b1ce-4a29-86b1-341d5db70d4f\") " pod="kube-system/global-pull-secret-syncer-v9xpw" Sep 17 10:11:51.640291 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:51.640090 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/026a85ed-57e7-474d-9f05-c8824906de72-metrics-tls\") pod \"dns-default-ds6sz\" (UID: \"026a85ed-57e7-474d-9f05-c8824906de72\") " pod="openshift-dns/dns-default-ds6sz" Sep 17 10:11:51.640291 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:11:51.640210 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 17 10:11:51.640291 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:11:51.640269 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/026a85ed-57e7-474d-9f05-c8824906de72-metrics-tls podName:026a85ed-57e7-474d-9f05-c8824906de72 nodeName:}" failed. No retries permitted until 2026-09-17 10:12:23.640251189 +0000 UTC m=+98.188808284 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/026a85ed-57e7-474d-9f05-c8824906de72-metrics-tls") pod "dns-default-ds6sz" (UID: "026a85ed-57e7-474d-9f05-c8824906de72") : secret "dns-default-metrics-tls" not found Sep 17 10:11:51.643006 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:51.642988 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 17 10:11:51.643071 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:51.642997 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 17 10:11:51.651016 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:11:51.650998 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 17 10:11:51.651077 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:11:51.651056 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c84a7003-1537-4470-8094-ea10a89a86a2-metrics-certs podName:c84a7003-1537-4470-8094-ea10a89a86a2 nodeName:}" failed. No retries permitted until 2026-09-17 10:12:55.651040306 +0000 UTC m=+130.199597398 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c84a7003-1537-4470-8094-ea10a89a86a2-metrics-certs") pod "network-metrics-daemon-wfccr" (UID: "c84a7003-1537-4470-8094-ea10a89a86a2") : secret "metrics-daemon-secret" not found Sep 17 10:11:51.653289 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:51.653270 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c304d2b0-b1ce-4a29-86b1-341d5db70d4f-original-pull-secret\") pod \"global-pull-secret-syncer-v9xpw\" (UID: \"c304d2b0-b1ce-4a29-86b1-341d5db70d4f\") " pod="kube-system/global-pull-secret-syncer-v9xpw" Sep 17 10:11:51.747774 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:51.747735 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v9xpw" Sep 17 10:11:51.841437 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:51.841380 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxnn2\" (UniqueName: \"kubernetes.io/projected/f6509b04-a69d-49d5-a08c-aabdd8bf6c86-kube-api-access-nxnn2\") pod \"network-check-target-fppwm\" (UID: \"f6509b04-a69d-49d5-a08c-aabdd8bf6c86\") " pod="openshift-network-diagnostics/network-check-target-fppwm" Sep 17 10:11:51.844401 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:51.844384 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 17 10:11:51.854734 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:51.854716 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 17 10:11:51.864840 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:51.864822 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxnn2\" (UniqueName: \"kubernetes.io/projected/f6509b04-a69d-49d5-a08c-aabdd8bf6c86-kube-api-access-nxnn2\") pod \"network-check-target-fppwm\" (UID: \"f6509b04-a69d-49d5-a08c-aabdd8bf6c86\") " pod="openshift-network-diagnostics/network-check-target-fppwm" Sep 17 10:11:52.014403 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:52.014373 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-v9xpw"] Sep 17 10:11:52.017457 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:11:52.017429 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc304d2b0_b1ce_4a29_86b1_341d5db70d4f.slice/crio-4988af8413adbeae8a1dd65c61e5c26353fbae920b2cfe0e50b528b8565fa9fe WatchSource:0}: Error finding container 4988af8413adbeae8a1dd65c61e5c26353fbae920b2cfe0e50b528b8565fa9fe: Status 404 returned error can't find the container with id 4988af8413adbeae8a1dd65c61e5c26353fbae920b2cfe0e50b528b8565fa9fe Sep 17 10:11:52.060874 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:52.060846 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-bxncx\"" Sep 17 10:11:52.068053 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:52.068032 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fppwm" Sep 17 10:11:52.270941 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:52.270870 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-b9wnf_762d3e10-7b7d-4ce9-9773-90a4fe41b474/console-operator/1.log" Sep 17 10:11:52.271355 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:52.271267 2581 scope.go:117] "RemoveContainer" containerID="4ce48daa3f4f6488190f4a4717a8a6b0b6637089749dbcfdad5d78b78e4b6b43" Sep 17 10:11:52.271465 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:11:52.271440 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-b9wnf_openshift-console-operator(762d3e10-7b7d-4ce9-9773-90a4fe41b474)\"" pod="openshift-console-operator/console-operator-696884fb9c-b9wnf" podUID="762d3e10-7b7d-4ce9-9773-90a4fe41b474" Sep 17 10:11:52.272575 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:52.272554 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-fdc72" event={"ID":"a13ce06b-1f1b-4614-a6f1-7e95dbd6f1d4","Type":"ContainerStarted","Data":"20f73970eac769a69eb7deb515b62c4a3c5ae6be2fa9d6421e76b577e0892988"} Sep 17 10:11:52.273660 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:52.273641 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-v9xpw" event={"ID":"c304d2b0-b1ce-4a29-86b1-341d5db70d4f","Type":"ContainerStarted","Data":"4988af8413adbeae8a1dd65c61e5c26353fbae920b2cfe0e50b528b8565fa9fe"} Sep 17 10:11:52.310176 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:52.310134 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5844b856db-fdc72" podStartSLOduration=2.011086401 podStartE2EDuration="6.31012092s" totalImagesPullingTime="4.299034519s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 10:11:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 10:11:52.309644453 +0000 UTC m=+66.858201564" watchObservedRunningTime="2026-09-17 10:11:52.31012092 +0000 UTC m=+66.858678061" Sep 17 10:11:52.334712 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:52.334398 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-fppwm"] Sep 17 10:11:52.337398 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:11:52.337373 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6509b04_a69d_49d5_a08c_aabdd8bf6c86.slice/crio-d29e76814854785cfc9d8a6a7e36bb9df7d5d6f467e2b375645c76d67629e581 WatchSource:0}: Error finding container d29e76814854785cfc9d8a6a7e36bb9df7d5d6f467e2b375645c76d67629e581: Status 404 returned error can't find the container with id d29e76814854785cfc9d8a6a7e36bb9df7d5d6f467e2b375645c76d67629e581 Sep 17 10:11:53.279172 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:53.279135 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-fppwm" event={"ID":"f6509b04-a69d-49d5-a08c-aabdd8bf6c86","Type":"ContainerStarted","Data":"d29e76814854785cfc9d8a6a7e36bb9df7d5d6f467e2b375645c76d67629e581"} Sep 17 10:11:53.457062 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:53.457027 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-2bn9v"] Sep 17 10:11:53.459562 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:53.459536 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2bn9v" Sep 17 10:11:53.462295 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:53.462269 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 17 10:11:53.462295 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:53.462291 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-4gbxx\"" Sep 17 10:11:53.462473 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:53.462279 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 17 10:11:53.474577 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:53.474514 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2bn9v"] Sep 17 10:11:53.555610 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:53.555522 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c65e3cab-f0ee-47b2-95f3-e059983bf8f7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2bn9v\" (UID: \"c65e3cab-f0ee-47b2-95f3-e059983bf8f7\") " pod="openshift-insights/insights-runtime-extractor-2bn9v" Sep 17 10:11:53.555764 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:53.555651 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c65e3cab-f0ee-47b2-95f3-e059983bf8f7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2bn9v\" (UID: \"c65e3cab-f0ee-47b2-95f3-e059983bf8f7\") " pod="openshift-insights/insights-runtime-extractor-2bn9v" Sep 17 10:11:53.555764 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:53.555692 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c65e3cab-f0ee-47b2-95f3-e059983bf8f7-data-volume\") pod \"insights-runtime-extractor-2bn9v\" (UID: \"c65e3cab-f0ee-47b2-95f3-e059983bf8f7\") " pod="openshift-insights/insights-runtime-extractor-2bn9v" Sep 17 10:11:53.555927 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:53.555848 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c65e3cab-f0ee-47b2-95f3-e059983bf8f7-crio-socket\") pod \"insights-runtime-extractor-2bn9v\" (UID: \"c65e3cab-f0ee-47b2-95f3-e059983bf8f7\") " pod="openshift-insights/insights-runtime-extractor-2bn9v" Sep 17 10:11:53.555982 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:53.555923 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nq66\" (UniqueName: \"kubernetes.io/projected/c65e3cab-f0ee-47b2-95f3-e059983bf8f7-kube-api-access-8nq66\") pod \"insights-runtime-extractor-2bn9v\" (UID: \"c65e3cab-f0ee-47b2-95f3-e059983bf8f7\") " pod="openshift-insights/insights-runtime-extractor-2bn9v" Sep 17 10:11:53.656538 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:53.656404 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c65e3cab-f0ee-47b2-95f3-e059983bf8f7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2bn9v\" (UID: \"c65e3cab-f0ee-47b2-95f3-e059983bf8f7\") " pod="openshift-insights/insights-runtime-extractor-2bn9v" Sep 17 10:11:53.656538 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:53.656451 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c65e3cab-f0ee-47b2-95f3-e059983bf8f7-data-volume\") pod \"insights-runtime-extractor-2bn9v\" (UID: \"c65e3cab-f0ee-47b2-95f3-e059983bf8f7\") " pod="openshift-insights/insights-runtime-extractor-2bn9v" Sep 17 10:11:53.656760 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:11:53.656590 2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 17 10:11:53.656760 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:53.656606 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c65e3cab-f0ee-47b2-95f3-e059983bf8f7-crio-socket\") pod \"insights-runtime-extractor-2bn9v\" (UID: \"c65e3cab-f0ee-47b2-95f3-e059983bf8f7\") " pod="openshift-insights/insights-runtime-extractor-2bn9v" Sep 17 10:11:53.656760 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:11:53.656667 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c65e3cab-f0ee-47b2-95f3-e059983bf8f7-insights-runtime-extractor-tls podName:c65e3cab-f0ee-47b2-95f3-e059983bf8f7 nodeName:}" failed. No retries permitted until 2026-09-17 10:11:54.156645073 +0000 UTC m=+68.705202180 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c65e3cab-f0ee-47b2-95f3-e059983bf8f7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2bn9v" (UID: "c65e3cab-f0ee-47b2-95f3-e059983bf8f7") : secret "insights-runtime-extractor-tls" not found Sep 17 10:11:53.656760 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:53.656700 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c65e3cab-f0ee-47b2-95f3-e059983bf8f7-crio-socket\") pod \"insights-runtime-extractor-2bn9v\" (UID: \"c65e3cab-f0ee-47b2-95f3-e059983bf8f7\") " pod="openshift-insights/insights-runtime-extractor-2bn9v" Sep 17 10:11:53.656760 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:53.656716 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nq66\" (UniqueName: \"kubernetes.io/projected/c65e3cab-f0ee-47b2-95f3-e059983bf8f7-kube-api-access-8nq66\") pod \"insights-runtime-extractor-2bn9v\" (UID: \"c65e3cab-f0ee-47b2-95f3-e059983bf8f7\") " pod="openshift-insights/insights-runtime-extractor-2bn9v" Sep 17 10:11:53.657055 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:53.656774 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c65e3cab-f0ee-47b2-95f3-e059983bf8f7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2bn9v\" (UID: \"c65e3cab-f0ee-47b2-95f3-e059983bf8f7\") " pod="openshift-insights/insights-runtime-extractor-2bn9v" Sep 17 10:11:53.657055 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:11:53.656881 2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 17 10:11:53.657055 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:53.656887 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c65e3cab-f0ee-47b2-95f3-e059983bf8f7-data-volume\") pod \"insights-runtime-extractor-2bn9v\" (UID: \"c65e3cab-f0ee-47b2-95f3-e059983bf8f7\") " pod="openshift-insights/insights-runtime-extractor-2bn9v" Sep 17 10:11:53.657055 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:11:53.656921 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c65e3cab-f0ee-47b2-95f3-e059983bf8f7-kube-rbac-proxy-cm podName:c65e3cab-f0ee-47b2-95f3-e059983bf8f7 nodeName:}" failed. No retries permitted until 2026-09-17 10:11:54.15690704 +0000 UTC m=+68.705464144 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c65e3cab-f0ee-47b2-95f3-e059983bf8f7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2bn9v" (UID: "c65e3cab-f0ee-47b2-95f3-e059983bf8f7") : configmap "kube-rbac-proxy" not found Sep 17 10:11:53.670298 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:53.670240 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nq66\" (UniqueName: \"kubernetes.io/projected/c65e3cab-f0ee-47b2-95f3-e059983bf8f7-kube-api-access-8nq66\") pod \"insights-runtime-extractor-2bn9v\" (UID: \"c65e3cab-f0ee-47b2-95f3-e059983bf8f7\") " pod="openshift-insights/insights-runtime-extractor-2bn9v" Sep 17 10:11:54.162799 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:54.162333 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c65e3cab-f0ee-47b2-95f3-e059983bf8f7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2bn9v\" (UID: \"c65e3cab-f0ee-47b2-95f3-e059983bf8f7\") " pod="openshift-insights/insights-runtime-extractor-2bn9v" Sep 17 10:11:54.162799 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:54.162445 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c65e3cab-f0ee-47b2-95f3-e059983bf8f7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2bn9v\" (UID: \"c65e3cab-f0ee-47b2-95f3-e059983bf8f7\") " pod="openshift-insights/insights-runtime-extractor-2bn9v" Sep 17 10:11:54.162799 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:11:54.162490 2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 17 10:11:54.162799 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:11:54.162613 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c65e3cab-f0ee-47b2-95f3-e059983bf8f7-kube-rbac-proxy-cm podName:c65e3cab-f0ee-47b2-95f3-e059983bf8f7 nodeName:}" failed. No retries permitted until 2026-09-17 10:11:55.162591789 +0000 UTC m=+69.711148884 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c65e3cab-f0ee-47b2-95f3-e059983bf8f7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2bn9v" (UID: "c65e3cab-f0ee-47b2-95f3-e059983bf8f7") : configmap "kube-rbac-proxy" not found Sep 17 10:11:54.162799 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:11:54.162614 2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 17 10:11:54.162799 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:11:54.162678 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c65e3cab-f0ee-47b2-95f3-e059983bf8f7-insights-runtime-extractor-tls podName:c65e3cab-f0ee-47b2-95f3-e059983bf8f7 nodeName:}" failed. No retries permitted until 2026-09-17 10:11:55.162661102 +0000 UTC m=+69.711218211 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c65e3cab-f0ee-47b2-95f3-e059983bf8f7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2bn9v" (UID: "c65e3cab-f0ee-47b2-95f3-e059983bf8f7") : secret "insights-runtime-extractor-tls" not found Sep 17 10:11:54.263168 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:54.263125 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e18215ee-6cb8-49d5-bb04-aaa79b65f012-service-ca-bundle\") pod \"router-default-85865fbbd8-bkwfh\" (UID: \"e18215ee-6cb8-49d5-bb04-aaa79b65f012\") " pod="openshift-ingress/router-default-85865fbbd8-bkwfh" Sep 17 10:11:54.263337 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:54.263214 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e18215ee-6cb8-49d5-bb04-aaa79b65f012-metrics-certs\") pod \"router-default-85865fbbd8-bkwfh\" (UID: \"e18215ee-6cb8-49d5-bb04-aaa79b65f012\") " pod="openshift-ingress/router-default-85865fbbd8-bkwfh" Sep 17 10:11:54.263410 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:11:54.263345 2581 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 17 10:11:54.263455 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:11:54.263409 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e18215ee-6cb8-49d5-bb04-aaa79b65f012-metrics-certs podName:e18215ee-6cb8-49d5-bb04-aaa79b65f012 nodeName:}" failed. No retries permitted until 2026-09-17 10:12:02.26338985 +0000 UTC m=+76.811946942 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e18215ee-6cb8-49d5-bb04-aaa79b65f012-metrics-certs") pod "router-default-85865fbbd8-bkwfh" (UID: "e18215ee-6cb8-49d5-bb04-aaa79b65f012") : secret "router-metrics-certs-default" not found Sep 17 10:11:54.263455 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:11:54.263450 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e18215ee-6cb8-49d5-bb04-aaa79b65f012-service-ca-bundle podName:e18215ee-6cb8-49d5-bb04-aaa79b65f012 nodeName:}" failed. No retries permitted until 2026-09-17 10:12:02.263439773 +0000 UTC m=+76.811996875 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e18215ee-6cb8-49d5-bb04-aaa79b65f012-service-ca-bundle") pod "router-default-85865fbbd8-bkwfh" (UID: "e18215ee-6cb8-49d5-bb04-aaa79b65f012") : configmap references non-existent config key: service-ca.crt Sep 17 10:11:54.465460 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:54.465321 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c802c004-f487-44df-b73a-fb50df751388-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-sv877\" (UID: \"c802c004-f487-44df-b73a-fb50df751388\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-sv877" Sep 17 10:11:54.465966 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:11:54.465481 2581 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 17 10:11:54.465966 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:11:54.465559 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c802c004-f487-44df-b73a-fb50df751388-samples-operator-tls podName:c802c004-f487-44df-b73a-fb50df751388 nodeName:}" failed. No retries permitted until 2026-09-17 10:12:02.465537829 +0000 UTC m=+77.014094925 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c802c004-f487-44df-b73a-fb50df751388-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-sv877" (UID: "c802c004-f487-44df-b73a-fb50df751388") : secret "samples-operator-tls" not found Sep 17 10:11:55.172656 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:55.172619 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c65e3cab-f0ee-47b2-95f3-e059983bf8f7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2bn9v\" (UID: \"c65e3cab-f0ee-47b2-95f3-e059983bf8f7\") " pod="openshift-insights/insights-runtime-extractor-2bn9v" Sep 17 10:11:55.172880 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:55.172681 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c65e3cab-f0ee-47b2-95f3-e059983bf8f7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2bn9v\" (UID: \"c65e3cab-f0ee-47b2-95f3-e059983bf8f7\") " pod="openshift-insights/insights-runtime-extractor-2bn9v" Sep 17 10:11:55.172880 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:11:55.172769 2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 17 10:11:55.172880 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:11:55.172827 2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 17 10:11:55.172880 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:11:55.172867 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c65e3cab-f0ee-47b2-95f3-e059983bf8f7-kube-rbac-proxy-cm podName:c65e3cab-f0ee-47b2-95f3-e059983bf8f7 nodeName:}" failed. No retries permitted until 2026-09-17 10:11:57.172844725 +0000 UTC m=+71.721401867 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c65e3cab-f0ee-47b2-95f3-e059983bf8f7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2bn9v" (UID: "c65e3cab-f0ee-47b2-95f3-e059983bf8f7") : configmap "kube-rbac-proxy" not found Sep 17 10:11:55.172880 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:11:55.172886 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c65e3cab-f0ee-47b2-95f3-e059983bf8f7-insights-runtime-extractor-tls podName:c65e3cab-f0ee-47b2-95f3-e059983bf8f7 nodeName:}" failed. No retries permitted until 2026-09-17 10:11:57.172879968 +0000 UTC m=+71.721437058 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c65e3cab-f0ee-47b2-95f3-e059983bf8f7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2bn9v" (UID: "c65e3cab-f0ee-47b2-95f3-e059983bf8f7") : secret "insights-runtime-extractor-tls" not found Sep 17 10:11:56.197732 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:56.197706 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-7f64n_6f7323dd-d769-4f58-92d7-e8a36c2f2854/dns-node-resolver/0.log" Sep 17 10:11:56.887933 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:56.887914 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-b9wnf" Sep 17 10:11:56.888039 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:56.887966 2581 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-b9wnf" Sep 17 10:11:56.888254 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:56.888243 2581 scope.go:117] "RemoveContainer" containerID="4ce48daa3f4f6488190f4a4717a8a6b0b6637089749dbcfdad5d78b78e4b6b43" Sep 17 10:11:56.888408 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:11:56.888387 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-b9wnf_openshift-console-operator(762d3e10-7b7d-4ce9-9773-90a4fe41b474)\"" pod="openshift-console-operator/console-operator-696884fb9c-b9wnf" podUID="762d3e10-7b7d-4ce9-9773-90a4fe41b474" Sep 17 10:11:57.189242 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:57.189161 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c65e3cab-f0ee-47b2-95f3-e059983bf8f7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2bn9v\" (UID: \"c65e3cab-f0ee-47b2-95f3-e059983bf8f7\") " pod="openshift-insights/insights-runtime-extractor-2bn9v" Sep 17 10:11:57.189242 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:57.189217 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c65e3cab-f0ee-47b2-95f3-e059983bf8f7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2bn9v\" (UID: \"c65e3cab-f0ee-47b2-95f3-e059983bf8f7\") " pod="openshift-insights/insights-runtime-extractor-2bn9v" Sep 17 10:11:57.189408 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:11:57.189289 2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 17 10:11:57.189408 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:11:57.189334 2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 17 10:11:57.189408 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:11:57.189355 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c65e3cab-f0ee-47b2-95f3-e059983bf8f7-kube-rbac-proxy-cm podName:c65e3cab-f0ee-47b2-95f3-e059983bf8f7 nodeName:}" failed. No retries permitted until 2026-09-17 10:12:01.189339833 +0000 UTC m=+75.737896934 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c65e3cab-f0ee-47b2-95f3-e059983bf8f7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2bn9v" (UID: "c65e3cab-f0ee-47b2-95f3-e059983bf8f7") : configmap "kube-rbac-proxy" not found Sep 17 10:11:57.189408 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:11:57.189407 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c65e3cab-f0ee-47b2-95f3-e059983bf8f7-insights-runtime-extractor-tls podName:c65e3cab-f0ee-47b2-95f3-e059983bf8f7 nodeName:}" failed. No retries permitted until 2026-09-17 10:12:01.189395798 +0000 UTC m=+75.737952905 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c65e3cab-f0ee-47b2-95f3-e059983bf8f7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2bn9v" (UID: "c65e3cab-f0ee-47b2-95f3-e059983bf8f7") : secret "insights-runtime-extractor-tls" not found Sep 17 10:11:57.291248 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:57.291214 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-fppwm" event={"ID":"f6509b04-a69d-49d5-a08c-aabdd8bf6c86","Type":"ContainerStarted","Data":"6474a4a63cef1288adcec9de9729755513c2b3048f5c0c1847bb0cd9ff8c1462"} Sep 17 10:11:57.291661 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:57.291279 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-fppwm" Sep 17 10:11:57.292811 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:57.292762 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-v9xpw" event={"ID":"c304d2b0-b1ce-4a29-86b1-341d5db70d4f","Type":"ContainerStarted","Data":"f51ae372cfbe3f671bdaf025af8e17428c82f0d44a55a58d81225dda3e02f299"} Sep 17 10:11:57.308011 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:57.307971 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-fppwm" podStartSLOduration=66.793057174 podStartE2EDuration="1m11.307952993s" totalImagesPullingTime="4.514895819s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 10:10:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 10:11:57.307255643 +0000 UTC m=+71.855812753" watchObservedRunningTime="2026-09-17 10:11:57.307952993 +0000 UTC m=+71.856510098" Sep 17 10:11:57.322949 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:57.322911 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-v9xpw" podStartSLOduration=66.477106624 podStartE2EDuration="1m11.322897007s" totalImagesPullingTime="4.845790383s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 10:10:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 10:11:57.322008259 +0000 UTC m=+71.870565370" watchObservedRunningTime="2026-09-17 10:11:57.322897007 +0000 UTC m=+71.871454118" Sep 17 10:11:57.402599 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:57.402580 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-r2dcq_2c409c2a-64a4-4164-8a14-c2a87597dd93/node-ca/0.log" Sep 17 10:11:57.999474 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:57.999445 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-lpfcm_2769e90f-7bdb-4fce-bfa6-ebf02a0963da/serve-healthcheck-canary/0.log" Sep 17 10:11:58.799549 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:11:58.799517 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-mjspg_da9abd04-2db6-44cc-b6bb-c5d2999ec8c4/kube-storage-version-migrator-operator/0.log" Sep 17 10:12:01.222937 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:01.222904 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c65e3cab-f0ee-47b2-95f3-e059983bf8f7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2bn9v\" (UID: \"c65e3cab-f0ee-47b2-95f3-e059983bf8f7\") " pod="openshift-insights/insights-runtime-extractor-2bn9v" Sep 17 10:12:01.223338 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:01.222957 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c65e3cab-f0ee-47b2-95f3-e059983bf8f7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2bn9v\" (UID: \"c65e3cab-f0ee-47b2-95f3-e059983bf8f7\") " pod="openshift-insights/insights-runtime-extractor-2bn9v" Sep 17 10:12:01.223338 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:12:01.223039 2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 17 10:12:01.223338 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:12:01.223089 2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 17 10:12:01.223338 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:12:01.223112 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c65e3cab-f0ee-47b2-95f3-e059983bf8f7-kube-rbac-proxy-cm podName:c65e3cab-f0ee-47b2-95f3-e059983bf8f7 nodeName:}" failed. No retries permitted until 2026-09-17 10:12:09.223095497 +0000 UTC m=+83.771652603 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c65e3cab-f0ee-47b2-95f3-e059983bf8f7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2bn9v" (UID: "c65e3cab-f0ee-47b2-95f3-e059983bf8f7") : configmap "kube-rbac-proxy" not found Sep 17 10:12:01.223338 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:12:01.223129 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c65e3cab-f0ee-47b2-95f3-e059983bf8f7-insights-runtime-extractor-tls podName:c65e3cab-f0ee-47b2-95f3-e059983bf8f7 nodeName:}" failed. No retries permitted until 2026-09-17 10:12:09.223118355 +0000 UTC m=+83.771675475 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c65e3cab-f0ee-47b2-95f3-e059983bf8f7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2bn9v" (UID: "c65e3cab-f0ee-47b2-95f3-e059983bf8f7") : secret "insights-runtime-extractor-tls" not found Sep 17 10:12:02.331279 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:02.331241 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e18215ee-6cb8-49d5-bb04-aaa79b65f012-metrics-certs\") pod \"router-default-85865fbbd8-bkwfh\" (UID: \"e18215ee-6cb8-49d5-bb04-aaa79b65f012\") " pod="openshift-ingress/router-default-85865fbbd8-bkwfh" Sep 17 10:12:02.331679 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:02.331329 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e18215ee-6cb8-49d5-bb04-aaa79b65f012-service-ca-bundle\") pod \"router-default-85865fbbd8-bkwfh\" (UID: \"e18215ee-6cb8-49d5-bb04-aaa79b65f012\") " pod="openshift-ingress/router-default-85865fbbd8-bkwfh" Sep 17 10:12:02.331679 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:12:02.331400 2581 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 17 10:12:02.331679 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:12:02.331444 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e18215ee-6cb8-49d5-bb04-aaa79b65f012-service-ca-bundle podName:e18215ee-6cb8-49d5-bb04-aaa79b65f012 nodeName:}" failed. No retries permitted until 2026-09-17 10:12:18.33143159 +0000 UTC m=+92.879988691 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e18215ee-6cb8-49d5-bb04-aaa79b65f012-service-ca-bundle") pod "router-default-85865fbbd8-bkwfh" (UID: "e18215ee-6cb8-49d5-bb04-aaa79b65f012") : configmap references non-existent config key: service-ca.crt Sep 17 10:12:02.331679 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:12:02.331459 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e18215ee-6cb8-49d5-bb04-aaa79b65f012-metrics-certs podName:e18215ee-6cb8-49d5-bb04-aaa79b65f012 nodeName:}" failed. No retries permitted until 2026-09-17 10:12:18.331453032 +0000 UTC m=+92.880010121 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e18215ee-6cb8-49d5-bb04-aaa79b65f012-metrics-certs") pod "router-default-85865fbbd8-bkwfh" (UID: "e18215ee-6cb8-49d5-bb04-aaa79b65f012") : secret "router-metrics-certs-default" not found Sep 17 10:12:02.533641 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:02.533600 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c802c004-f487-44df-b73a-fb50df751388-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-sv877\" (UID: \"c802c004-f487-44df-b73a-fb50df751388\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-sv877" Sep 17 10:12:02.535955 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:02.535937 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c802c004-f487-44df-b73a-fb50df751388-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-sv877\" (UID: \"c802c004-f487-44df-b73a-fb50df751388\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-sv877" Sep 17 10:12:02.685155 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:02.685055 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-sv877" Sep 17 10:12:02.953524 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:02.953471 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-sv877"] Sep 17 10:12:03.312854 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:03.312816 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-sv877" event={"ID":"c802c004-f487-44df-b73a-fb50df751388","Type":"ContainerStarted","Data":"1b4d7fd17e384aff453a636e418ffe8a774a36ccb5d136aac7b3135be9c6a295"} Sep 17 10:12:05.323352 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:05.323319 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-sv877" event={"ID":"c802c004-f487-44df-b73a-fb50df751388","Type":"ContainerStarted","Data":"edc5fdebfc6e8f6d2ac6937ad82d331c2a6333d873df395f8afd68f635c60d05"} Sep 17 10:12:05.323751 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:05.323410 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-sv877" event={"ID":"c802c004-f487-44df-b73a-fb50df751388","Type":"ContainerStarted","Data":"458448e320278b55912b3da351a96c1f7f05aac6df45da818a707f15e55871a1"} Sep 17 10:12:05.341029 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:05.340978 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-sv877" podStartSLOduration=17.365603296 podStartE2EDuration="19.340963479s" totalImagesPullingTime="1.975360183s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 10:11:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 10:12:05.340123443 +0000 UTC m=+79.888680556" watchObservedRunningTime="2026-09-17 10:12:05.340963479 +0000 UTC m=+79.889520590" Sep 17 10:12:09.285964 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:09.285853 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c65e3cab-f0ee-47b2-95f3-e059983bf8f7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2bn9v\" (UID: \"c65e3cab-f0ee-47b2-95f3-e059983bf8f7\") " pod="openshift-insights/insights-runtime-extractor-2bn9v" Sep 17 10:12:09.285964 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:09.285904 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c65e3cab-f0ee-47b2-95f3-e059983bf8f7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2bn9v\" (UID: \"c65e3cab-f0ee-47b2-95f3-e059983bf8f7\") " pod="openshift-insights/insights-runtime-extractor-2bn9v" Sep 17 10:12:09.286382 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:09.286350 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c65e3cab-f0ee-47b2-95f3-e059983bf8f7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2bn9v\" (UID: \"c65e3cab-f0ee-47b2-95f3-e059983bf8f7\") " pod="openshift-insights/insights-runtime-extractor-2bn9v" Sep 17 10:12:09.288128 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:09.288108 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c65e3cab-f0ee-47b2-95f3-e059983bf8f7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2bn9v\" (UID: \"c65e3cab-f0ee-47b2-95f3-e059983bf8f7\") " pod="openshift-insights/insights-runtime-extractor-2bn9v" Sep 17 10:12:09.372460 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:09.372422 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2bn9v" Sep 17 10:12:09.643433 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:09.643411 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2bn9v"] Sep 17 10:12:09.645751 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:12:09.645724 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc65e3cab_f0ee_47b2_95f3_e059983bf8f7.slice/crio-c5b0253aa235e3ca2e68122092354045454e2ebf91463601405f25303805a57b WatchSource:0}: Error finding container c5b0253aa235e3ca2e68122092354045454e2ebf91463601405f25303805a57b: Status 404 returned error can't find the container with id c5b0253aa235e3ca2e68122092354045454e2ebf91463601405f25303805a57b Sep 17 10:12:09.939370 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:09.939297 2581 scope.go:117] "RemoveContainer" containerID="4ce48daa3f4f6488190f4a4717a8a6b0b6637089749dbcfdad5d78b78e4b6b43" Sep 17 10:12:10.339556 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:10.339522 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2bn9v" event={"ID":"c65e3cab-f0ee-47b2-95f3-e059983bf8f7","Type":"ContainerStarted","Data":"72a90326ddcd2b37ce3ac8a89443e5bc4ca1102c07fe26adb122324521071688"} Sep 17 10:12:10.339870 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:10.339583 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2bn9v" event={"ID":"c65e3cab-f0ee-47b2-95f3-e059983bf8f7","Type":"ContainerStarted","Data":"abe0e7aa5286816bde55b1feb26e186d2b32b020b63778c4ac7fee96826e4833"} Sep 17 10:12:10.339870 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:10.339605 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2bn9v" event={"ID":"c65e3cab-f0ee-47b2-95f3-e059983bf8f7","Type":"ContainerStarted","Data":"c5b0253aa235e3ca2e68122092354045454e2ebf91463601405f25303805a57b"} Sep 17 10:12:10.341106 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:10.341089 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-b9wnf_762d3e10-7b7d-4ce9-9773-90a4fe41b474/console-operator/2.log" Sep 17 10:12:10.341497 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:10.341484 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-b9wnf_762d3e10-7b7d-4ce9-9773-90a4fe41b474/console-operator/1.log" Sep 17 10:12:10.341547 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:10.341512 2581 generic.go:356] "Generic (PLEG): container finished" podID="762d3e10-7b7d-4ce9-9773-90a4fe41b474" containerID="a6dc1122273355ff1bf6475889fb31b10461ff4435c72f5a7fb6c7dd1fae5200" exitCode=255 Sep 17 10:12:10.341579 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:10.341551 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-b9wnf" event={"ID":"762d3e10-7b7d-4ce9-9773-90a4fe41b474","Type":"ContainerDied","Data":"a6dc1122273355ff1bf6475889fb31b10461ff4435c72f5a7fb6c7dd1fae5200"} Sep 17 10:12:10.341618 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:10.341588 2581 scope.go:117] "RemoveContainer" containerID="4ce48daa3f4f6488190f4a4717a8a6b0b6637089749dbcfdad5d78b78e4b6b43" Sep 17 10:12:10.341929 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:10.341909 2581 scope.go:117] "RemoveContainer" containerID="a6dc1122273355ff1bf6475889fb31b10461ff4435c72f5a7fb6c7dd1fae5200" Sep 17 10:12:10.342108 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:12:10.342079 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-696884fb9c-b9wnf_openshift-console-operator(762d3e10-7b7d-4ce9-9773-90a4fe41b474)\"" pod="openshift-console-operator/console-operator-696884fb9c-b9wnf" podUID="762d3e10-7b7d-4ce9-9773-90a4fe41b474" Sep 17 10:12:11.348611 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:11.348584 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-b9wnf_762d3e10-7b7d-4ce9-9773-90a4fe41b474/console-operator/2.log" Sep 17 10:12:12.354293 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:12.354259 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2bn9v" event={"ID":"c65e3cab-f0ee-47b2-95f3-e059983bf8f7","Type":"ContainerStarted","Data":"9a9af1a4e3c9ae512d3c23cfa6348e0e24e40f6645ab8bd06dabc76f13d7c686"} Sep 17 10:12:12.376374 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:12.376328 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-2bn9v" podStartSLOduration=17.453651863 podStartE2EDuration="19.376313653s" totalImagesPullingTime="1.92266179s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 10:11:53 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 10:12:12.374387291 +0000 UTC m=+86.922944402" watchObservedRunningTime="2026-09-17 10:12:12.376313653 +0000 UTC m=+86.924870764" Sep 17 10:12:14.932209 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:14.932172 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-86cb58c7dd-v95nh"] Sep 17 10:12:14.934123 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:14.934105 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-86cb58c7dd-v95nh" Sep 17 10:12:14.937600 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:14.937580 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-hhnrv\"" Sep 17 10:12:14.937702 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:14.937602 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Sep 17 10:12:14.939029 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:14.939008 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Sep 17 10:12:14.939121 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:14.939008 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Sep 17 10:12:14.939121 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:14.939070 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Sep 17 10:12:14.946116 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:14.946091 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-86cb58c7dd-v95nh"] Sep 17 10:12:15.031401 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:15.031362 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0bcaf9a7-41a0-434a-b47b-c9ae76893ae1-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-86cb58c7dd-v95nh\" (UID: \"0bcaf9a7-41a0-434a-b47b-c9ae76893ae1\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-86cb58c7dd-v95nh" Sep 17 10:12:15.031609 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:15.031428 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zf6nc\" (UniqueName: \"kubernetes.io/projected/0bcaf9a7-41a0-434a-b47b-c9ae76893ae1-kube-api-access-zf6nc\") pod \"managed-serviceaccount-addon-agent-86cb58c7dd-v95nh\" (UID: \"0bcaf9a7-41a0-434a-b47b-c9ae76893ae1\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-86cb58c7dd-v95nh" Sep 17 10:12:15.132747 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:15.132711 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0bcaf9a7-41a0-434a-b47b-c9ae76893ae1-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-86cb58c7dd-v95nh\" (UID: \"0bcaf9a7-41a0-434a-b47b-c9ae76893ae1\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-86cb58c7dd-v95nh" Sep 17 10:12:15.132984 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:15.132806 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zf6nc\" (UniqueName: \"kubernetes.io/projected/0bcaf9a7-41a0-434a-b47b-c9ae76893ae1-kube-api-access-zf6nc\") pod \"managed-serviceaccount-addon-agent-86cb58c7dd-v95nh\" (UID: \"0bcaf9a7-41a0-434a-b47b-c9ae76893ae1\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-86cb58c7dd-v95nh" Sep 17 10:12:15.135488 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:15.135455 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0bcaf9a7-41a0-434a-b47b-c9ae76893ae1-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-86cb58c7dd-v95nh\" (UID: \"0bcaf9a7-41a0-434a-b47b-c9ae76893ae1\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-86cb58c7dd-v95nh" Sep 17 10:12:15.146217 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:15.146174 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zf6nc\" (UniqueName: \"kubernetes.io/projected/0bcaf9a7-41a0-434a-b47b-c9ae76893ae1-kube-api-access-zf6nc\") pod \"managed-serviceaccount-addon-agent-86cb58c7dd-v95nh\" (UID: \"0bcaf9a7-41a0-434a-b47b-c9ae76893ae1\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-86cb58c7dd-v95nh" Sep 17 10:12:15.253565 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:15.253527 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-86cb58c7dd-v95nh" Sep 17 10:12:15.533513 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:15.533431 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-86cb58c7dd-v95nh"] Sep 17 10:12:15.537180 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:12:15.537153 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0bcaf9a7_41a0_434a_b47b_c9ae76893ae1.slice/crio-ffcb0c74811bd18c690f4a571cef2ae5c43ac3353e4dfc57dec48d1dade94252 WatchSource:0}: Error finding container ffcb0c74811bd18c690f4a571cef2ae5c43ac3353e4dfc57dec48d1dade94252: Status 404 returned error can't find the container with id ffcb0c74811bd18c690f4a571cef2ae5c43ac3353e4dfc57dec48d1dade94252 Sep 17 10:12:16.370583 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:16.370509 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-86cb58c7dd-v95nh" event={"ID":"0bcaf9a7-41a0-434a-b47b-c9ae76893ae1","Type":"ContainerStarted","Data":"ffcb0c74811bd18c690f4a571cef2ae5c43ac3353e4dfc57dec48d1dade94252"} Sep 17 10:12:16.888060 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:16.888021 2581 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-b9wnf" Sep 17 10:12:16.888293 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:16.888089 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-b9wnf" Sep 17 10:12:16.888470 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:16.888454 2581 scope.go:117] "RemoveContainer" containerID="a6dc1122273355ff1bf6475889fb31b10461ff4435c72f5a7fb6c7dd1fae5200" Sep 17 10:12:16.888656 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:12:16.888637 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-696884fb9c-b9wnf_openshift-console-operator(762d3e10-7b7d-4ce9-9773-90a4fe41b474)\"" pod="openshift-console-operator/console-operator-696884fb9c-b9wnf" podUID="762d3e10-7b7d-4ce9-9773-90a4fe41b474" Sep 17 10:12:18.358023 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:18.357978 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e18215ee-6cb8-49d5-bb04-aaa79b65f012-service-ca-bundle\") pod \"router-default-85865fbbd8-bkwfh\" (UID: \"e18215ee-6cb8-49d5-bb04-aaa79b65f012\") " pod="openshift-ingress/router-default-85865fbbd8-bkwfh" Sep 17 10:12:18.358506 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:18.358094 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e18215ee-6cb8-49d5-bb04-aaa79b65f012-metrics-certs\") pod \"router-default-85865fbbd8-bkwfh\" (UID: \"e18215ee-6cb8-49d5-bb04-aaa79b65f012\") " pod="openshift-ingress/router-default-85865fbbd8-bkwfh" Sep 17 10:12:18.358710 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:18.358688 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e18215ee-6cb8-49d5-bb04-aaa79b65f012-service-ca-bundle\") pod \"router-default-85865fbbd8-bkwfh\" (UID: \"e18215ee-6cb8-49d5-bb04-aaa79b65f012\") " pod="openshift-ingress/router-default-85865fbbd8-bkwfh" Sep 17 10:12:18.360408 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:18.360389 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e18215ee-6cb8-49d5-bb04-aaa79b65f012-metrics-certs\") pod \"router-default-85865fbbd8-bkwfh\" (UID: \"e18215ee-6cb8-49d5-bb04-aaa79b65f012\") " pod="openshift-ingress/router-default-85865fbbd8-bkwfh" Sep 17 10:12:18.378141 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:18.378091 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-86cb58c7dd-v95nh" event={"ID":"0bcaf9a7-41a0-434a-b47b-c9ae76893ae1","Type":"ContainerStarted","Data":"3f34d68f068e319ab4417b5c4014cf066825cf9d596da6383e3f199e5b268867"} Sep 17 10:12:18.392031 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:18.392002 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-85865fbbd8-bkwfh" Sep 17 10:12:18.396396 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:18.396351 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-86cb58c7dd-v95nh" podStartSLOduration=1.999191321 podStartE2EDuration="4.396337034s" totalImagesPullingTime="2.397145713s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 10:12:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 10:12:18.395263002 +0000 UTC m=+92.943820113" watchObservedRunningTime="2026-09-17 10:12:18.396337034 +0000 UTC m=+92.944894146" Sep 17 10:12:18.673762 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:18.673691 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-85865fbbd8-bkwfh"] Sep 17 10:12:18.677225 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:12:18.677194 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode18215ee_6cb8_49d5_bb04_aaa79b65f012.slice/crio-764b6c448af67b8fd2198c8ba5d4c3f17792a459a9a57860bc997c9e3d2eb771 WatchSource:0}: Error finding container 764b6c448af67b8fd2198c8ba5d4c3f17792a459a9a57860bc997c9e3d2eb771: Status 404 returned error can't find the container with id 764b6c448af67b8fd2198c8ba5d4c3f17792a459a9a57860bc997c9e3d2eb771 Sep 17 10:12:19.383640 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:19.383603 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-85865fbbd8-bkwfh" event={"ID":"e18215ee-6cb8-49d5-bb04-aaa79b65f012","Type":"ContainerStarted","Data":"a060ff64ec1cd3a769d2774324ec3e5020af4b8ada83e229f00cb17cd1e6f2b7"} Sep 17 10:12:19.384086 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:19.383650 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-85865fbbd8-bkwfh" event={"ID":"e18215ee-6cb8-49d5-bb04-aaa79b65f012","Type":"ContainerStarted","Data":"764b6c448af67b8fd2198c8ba5d4c3f17792a459a9a57860bc997c9e3d2eb771"} Sep 17 10:12:19.392940 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:19.392919 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-85865fbbd8-bkwfh" Sep 17 10:12:19.395332 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:19.395313 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-85865fbbd8-bkwfh" Sep 17 10:12:19.409833 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:19.409767 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-85865fbbd8-bkwfh" podStartSLOduration=33.409754516 podStartE2EDuration="33.409754516s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 10:11:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 10:12:19.408164669 +0000 UTC m=+93.956721781" watchObservedRunningTime="2026-09-17 10:12:19.409754516 +0000 UTC m=+93.958311627" Sep 17 10:12:20.387362 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:20.387328 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-85865fbbd8-bkwfh" Sep 17 10:12:20.388633 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:20.388611 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-85865fbbd8-bkwfh" Sep 17 10:12:20.868092 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:20.868061 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-hfwvn"] Sep 17 10:12:20.870684 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:20.870669 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-hfwvn" Sep 17 10:12:20.874033 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:20.874013 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Sep 17 10:12:20.874143 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:20.874046 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-rsqbn\"" Sep 17 10:12:20.882624 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:20.882592 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-hfwvn"] Sep 17 10:12:20.980066 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:20.980008 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/f3233a3b-aa93-403e-99d2-78a522e8a006-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-hfwvn\" (UID: \"f3233a3b-aa93-403e-99d2-78a522e8a006\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-hfwvn" Sep 17 10:12:21.081326 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:21.081270 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/f3233a3b-aa93-403e-99d2-78a522e8a006-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-hfwvn\" (UID: \"f3233a3b-aa93-403e-99d2-78a522e8a006\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-hfwvn" Sep 17 10:12:21.083649 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:21.083628 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/f3233a3b-aa93-403e-99d2-78a522e8a006-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-hfwvn\" (UID: \"f3233a3b-aa93-403e-99d2-78a522e8a006\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-hfwvn" Sep 17 10:12:21.179893 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:21.179766 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-hfwvn" Sep 17 10:12:21.447537 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:21.447462 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-hfwvn"] Sep 17 10:12:21.450848 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:12:21.450812 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3233a3b_aa93_403e_99d2_78a522e8a006.slice/crio-ff5ab16cc3a2f0573b870f942a604da6793457a88ecdd77900e8131b7175a35f WatchSource:0}: Error finding container ff5ab16cc3a2f0573b870f942a604da6793457a88ecdd77900e8131b7175a35f: Status 404 returned error can't find the container with id ff5ab16cc3a2f0573b870f942a604da6793457a88ecdd77900e8131b7175a35f Sep 17 10:12:22.395506 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:22.395476 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-hfwvn" event={"ID":"f3233a3b-aa93-403e-99d2-78a522e8a006","Type":"ContainerStarted","Data":"ff5ab16cc3a2f0573b870f942a604da6793457a88ecdd77900e8131b7175a35f"} Sep 17 10:12:23.400298 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:23.400259 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-hfwvn" event={"ID":"f3233a3b-aa93-403e-99d2-78a522e8a006","Type":"ContainerStarted","Data":"692e933bb744deb3b792b051c0a98e9704e40c7d2dbd9ad5d9077e901f750db2"} Sep 17 10:12:23.400764 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:23.400464 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-hfwvn" Sep 17 10:12:23.405166 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:23.405142 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-hfwvn" Sep 17 10:12:23.418506 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:23.418458 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-hfwvn" podStartSLOduration=2.385227017 podStartE2EDuration="3.418444195s" totalImagesPullingTime="1.033217178s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 10:12:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 10:12:23.417567738 +0000 UTC m=+97.966124848" watchObservedRunningTime="2026-09-17 10:12:23.418444195 +0000 UTC m=+97.967001306" Sep 17 10:12:23.701891 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:23.701797 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/026a85ed-57e7-474d-9f05-c8824906de72-metrics-tls\") pod \"dns-default-ds6sz\" (UID: \"026a85ed-57e7-474d-9f05-c8824906de72\") " pod="openshift-dns/dns-default-ds6sz" Sep 17 10:12:23.704100 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:23.704074 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/026a85ed-57e7-474d-9f05-c8824906de72-metrics-tls\") pod \"dns-default-ds6sz\" (UID: \"026a85ed-57e7-474d-9f05-c8824906de72\") " pod="openshift-dns/dns-default-ds6sz" Sep 17 10:12:23.723043 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:23.723019 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-sglxr\"" Sep 17 10:12:23.731254 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:23.731232 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ds6sz" Sep 17 10:12:23.927689 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:23.927656 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-vzrdc"] Sep 17 10:12:23.953649 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:23.953580 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-vzrdc"] Sep 17 10:12:23.953796 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:23.953762 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-vzrdc" Sep 17 10:12:23.956959 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:23.956937 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Sep 17 10:12:23.957134 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:23.956998 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Sep 17 10:12:23.957269 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:23.957189 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 17 10:12:23.957269 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:23.957031 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 17 10:12:23.957370 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:23.957076 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 17 10:12:23.958364 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:23.958346 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-kt4dk\"" Sep 17 10:12:24.022499 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:24.022470 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-ds6sz"] Sep 17 10:12:24.025634 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:12:24.025596 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod026a85ed_57e7_474d_9f05_c8824906de72.slice/crio-6d72ed6a22d13fb9a601efdd3f01a2753af3181004661ae28db919b489ab8f2d WatchSource:0}: Error finding container 6d72ed6a22d13fb9a601efdd3f01a2753af3181004661ae28db919b489ab8f2d: Status 404 returned error can't find the container with id 6d72ed6a22d13fb9a601efdd3f01a2753af3181004661ae28db919b489ab8f2d Sep 17 10:12:24.105331 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:24.105295 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4a7d0e53-a082-4e4a-bb1d-3da2d679e92c-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-vzrdc\" (UID: \"4a7d0e53-a082-4e4a-bb1d-3da2d679e92c\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-vzrdc" Sep 17 10:12:24.105499 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:24.105338 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57lg6\" (UniqueName: \"kubernetes.io/projected/4a7d0e53-a082-4e4a-bb1d-3da2d679e92c-kube-api-access-57lg6\") pod \"prometheus-operator-68bcd7f987-vzrdc\" (UID: \"4a7d0e53-a082-4e4a-bb1d-3da2d679e92c\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-vzrdc" Sep 17 10:12:24.105499 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:24.105411 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4a7d0e53-a082-4e4a-bb1d-3da2d679e92c-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-vzrdc\" (UID: \"4a7d0e53-a082-4e4a-bb1d-3da2d679e92c\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-vzrdc" Sep 17 10:12:24.105573 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:24.105504 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/4a7d0e53-a082-4e4a-bb1d-3da2d679e92c-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-vzrdc\" (UID: \"4a7d0e53-a082-4e4a-bb1d-3da2d679e92c\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-vzrdc" Sep 17 10:12:24.206727 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:24.206644 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/4a7d0e53-a082-4e4a-bb1d-3da2d679e92c-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-vzrdc\" (UID: \"4a7d0e53-a082-4e4a-bb1d-3da2d679e92c\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-vzrdc" Sep 17 10:12:24.206727 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:24.206693 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4a7d0e53-a082-4e4a-bb1d-3da2d679e92c-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-vzrdc\" (UID: \"4a7d0e53-a082-4e4a-bb1d-3da2d679e92c\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-vzrdc" Sep 17 10:12:24.206727 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:24.206712 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57lg6\" (UniqueName: \"kubernetes.io/projected/4a7d0e53-a082-4e4a-bb1d-3da2d679e92c-kube-api-access-57lg6\") pod \"prometheus-operator-68bcd7f987-vzrdc\" (UID: \"4a7d0e53-a082-4e4a-bb1d-3da2d679e92c\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-vzrdc" Sep 17 10:12:24.206974 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:12:24.206822 2581 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Sep 17 10:12:24.206974 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:24.206828 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4a7d0e53-a082-4e4a-bb1d-3da2d679e92c-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-vzrdc\" (UID: \"4a7d0e53-a082-4e4a-bb1d-3da2d679e92c\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-vzrdc" Sep 17 10:12:24.206974 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:12:24.206895 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a7d0e53-a082-4e4a-bb1d-3da2d679e92c-prometheus-operator-tls podName:4a7d0e53-a082-4e4a-bb1d-3da2d679e92c nodeName:}" failed. No retries permitted until 2026-09-17 10:12:24.70687702 +0000 UTC m=+99.255434126 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/4a7d0e53-a082-4e4a-bb1d-3da2d679e92c-prometheus-operator-tls") pod "prometheus-operator-68bcd7f987-vzrdc" (UID: "4a7d0e53-a082-4e4a-bb1d-3da2d679e92c") : secret "prometheus-operator-tls" not found Sep 17 10:12:24.207452 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:24.207434 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4a7d0e53-a082-4e4a-bb1d-3da2d679e92c-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-vzrdc\" (UID: \"4a7d0e53-a082-4e4a-bb1d-3da2d679e92c\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-vzrdc" Sep 17 10:12:24.209033 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:24.209012 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4a7d0e53-a082-4e4a-bb1d-3da2d679e92c-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-vzrdc\" (UID: \"4a7d0e53-a082-4e4a-bb1d-3da2d679e92c\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-vzrdc" Sep 17 10:12:24.219243 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:24.219221 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-57lg6\" (UniqueName: \"kubernetes.io/projected/4a7d0e53-a082-4e4a-bb1d-3da2d679e92c-kube-api-access-57lg6\") pod \"prometheus-operator-68bcd7f987-vzrdc\" (UID: \"4a7d0e53-a082-4e4a-bb1d-3da2d679e92c\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-vzrdc" Sep 17 10:12:24.404944 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:24.404905 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ds6sz" event={"ID":"026a85ed-57e7-474d-9f05-c8824906de72","Type":"ContainerStarted","Data":"6d72ed6a22d13fb9a601efdd3f01a2753af3181004661ae28db919b489ab8f2d"} Sep 17 10:12:24.710819 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:24.710765 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/4a7d0e53-a082-4e4a-bb1d-3da2d679e92c-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-vzrdc\" (UID: \"4a7d0e53-a082-4e4a-bb1d-3da2d679e92c\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-vzrdc" Sep 17 10:12:24.713593 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:24.713564 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/4a7d0e53-a082-4e4a-bb1d-3da2d679e92c-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-vzrdc\" (UID: \"4a7d0e53-a082-4e4a-bb1d-3da2d679e92c\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-vzrdc" Sep 17 10:12:24.863322 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:24.863289 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-vzrdc" Sep 17 10:12:25.157593 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:25.157562 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-vzrdc"] Sep 17 10:12:25.525532 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:12:25.525485 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a7d0e53_a082_4e4a_bb1d_3da2d679e92c.slice/crio-d5d3f4f26276e33d748ababf38cf267d0fd29a340695f63fb2254a967333754d WatchSource:0}: Error finding container d5d3f4f26276e33d748ababf38cf267d0fd29a340695f63fb2254a967333754d: Status 404 returned error can't find the container with id d5d3f4f26276e33d748ababf38cf267d0fd29a340695f63fb2254a967333754d Sep 17 10:12:26.414575 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:26.414481 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ds6sz" event={"ID":"026a85ed-57e7-474d-9f05-c8824906de72","Type":"ContainerStarted","Data":"8bf6b99ee408052a2aeb46260160af60df4853ee0162fcd87140a9a60311fb64"} Sep 17 10:12:26.414575 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:26.414545 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ds6sz" event={"ID":"026a85ed-57e7-474d-9f05-c8824906de72","Type":"ContainerStarted","Data":"9d7238c8ea9aa2de593719842995cbde28b836567d84251c78042580a297203b"} Sep 17 10:12:26.414767 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:26.414598 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-ds6sz" Sep 17 10:12:26.416121 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:26.416091 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-vzrdc" event={"ID":"4a7d0e53-a082-4e4a-bb1d-3da2d679e92c","Type":"ContainerStarted","Data":"d5d3f4f26276e33d748ababf38cf267d0fd29a340695f63fb2254a967333754d"} Sep 17 10:12:26.434833 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:26.434760 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-ds6sz" podStartSLOduration=65.589857587 podStartE2EDuration="1m7.434743473s" totalImagesPullingTime="1.844885886s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 10:11:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 10:12:26.432614447 +0000 UTC m=+100.981171559" watchObservedRunningTime="2026-09-17 10:12:26.434743473 +0000 UTC m=+100.983300588" Sep 17 10:12:27.422761 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:27.422665 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-vzrdc" event={"ID":"4a7d0e53-a082-4e4a-bb1d-3da2d679e92c","Type":"ContainerStarted","Data":"f94b7aba6ffda5de9b3dec1dfe07e394b1c796ac0648cbf91fc7c8e2377e6146"} Sep 17 10:12:27.422761 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:27.422728 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-vzrdc" event={"ID":"4a7d0e53-a082-4e4a-bb1d-3da2d679e92c","Type":"ContainerStarted","Data":"0b06c50f8ac184752e614b6cad1967aa1573ec0bef0644ce55f2eb3e1ace1fee"} Sep 17 10:12:27.442506 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:27.442445 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-68bcd7f987-vzrdc" podStartSLOduration=2.887685186 podStartE2EDuration="4.442428337s" totalImagesPullingTime="1.554743151s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 10:12:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 10:12:27.441247611 +0000 UTC m=+101.989804725" watchObservedRunningTime="2026-09-17 10:12:27.442428337 +0000 UTC m=+101.990985447" Sep 17 10:12:27.939421 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:27.939385 2581 scope.go:117] "RemoveContainer" containerID="a6dc1122273355ff1bf6475889fb31b10461ff4435c72f5a7fb6c7dd1fae5200" Sep 17 10:12:27.939605 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:12:27.939587 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-696884fb9c-b9wnf_openshift-console-operator(762d3e10-7b7d-4ce9-9773-90a4fe41b474)\"" pod="openshift-console-operator/console-operator-696884fb9c-b9wnf" podUID="762d3e10-7b7d-4ce9-9773-90a4fe41b474" Sep 17 10:12:28.300422 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:28.300392 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-fppwm" Sep 17 10:12:29.402607 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:29.402569 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-2z4bl"] Sep 17 10:12:29.425111 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:29.425078 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-2z4bl" Sep 17 10:12:29.429053 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:29.429021 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 17 10:12:29.429053 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:29.429043 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 17 10:12:29.429265 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:29.429060 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 17 10:12:29.429265 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:29.429145 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-v255f\"" Sep 17 10:12:29.554376 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:29.554337 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a384f6e7-e487-4749-b0d0-6b679aa60fae-node-exporter-accelerators-collector-config\") pod \"node-exporter-2z4bl\" (UID: \"a384f6e7-e487-4749-b0d0-6b679aa60fae\") " pod="openshift-monitoring/node-exporter-2z4bl" Sep 17 10:12:29.554376 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:29.554374 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a384f6e7-e487-4749-b0d0-6b679aa60fae-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2z4bl\" (UID: \"a384f6e7-e487-4749-b0d0-6b679aa60fae\") " pod="openshift-monitoring/node-exporter-2z4bl" Sep 17 10:12:29.554628 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:29.554405 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a384f6e7-e487-4749-b0d0-6b679aa60fae-node-exporter-textfile\") pod \"node-exporter-2z4bl\" (UID: \"a384f6e7-e487-4749-b0d0-6b679aa60fae\") " pod="openshift-monitoring/node-exporter-2z4bl" Sep 17 10:12:29.554628 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:29.554452 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a384f6e7-e487-4749-b0d0-6b679aa60fae-root\") pod \"node-exporter-2z4bl\" (UID: \"a384f6e7-e487-4749-b0d0-6b679aa60fae\") " pod="openshift-monitoring/node-exporter-2z4bl" Sep 17 10:12:29.554628 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:29.554526 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a384f6e7-e487-4749-b0d0-6b679aa60fae-node-exporter-wtmp\") pod \"node-exporter-2z4bl\" (UID: \"a384f6e7-e487-4749-b0d0-6b679aa60fae\") " pod="openshift-monitoring/node-exporter-2z4bl" Sep 17 10:12:29.554628 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:29.554550 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a384f6e7-e487-4749-b0d0-6b679aa60fae-sys\") pod \"node-exporter-2z4bl\" (UID: \"a384f6e7-e487-4749-b0d0-6b679aa60fae\") " pod="openshift-monitoring/node-exporter-2z4bl" Sep 17 10:12:29.554628 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:29.554568 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rmz9\" (UniqueName: \"kubernetes.io/projected/a384f6e7-e487-4749-b0d0-6b679aa60fae-kube-api-access-9rmz9\") pod \"node-exporter-2z4bl\" (UID: \"a384f6e7-e487-4749-b0d0-6b679aa60fae\") " pod="openshift-monitoring/node-exporter-2z4bl" Sep 17 10:12:29.554628 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:29.554586 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a384f6e7-e487-4749-b0d0-6b679aa60fae-node-exporter-tls\") pod \"node-exporter-2z4bl\" (UID: \"a384f6e7-e487-4749-b0d0-6b679aa60fae\") " pod="openshift-monitoring/node-exporter-2z4bl" Sep 17 10:12:29.554839 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:29.554666 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a384f6e7-e487-4749-b0d0-6b679aa60fae-metrics-client-ca\") pod \"node-exporter-2z4bl\" (UID: \"a384f6e7-e487-4749-b0d0-6b679aa60fae\") " pod="openshift-monitoring/node-exporter-2z4bl" Sep 17 10:12:29.655825 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:29.655723 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a384f6e7-e487-4749-b0d0-6b679aa60fae-node-exporter-textfile\") pod \"node-exporter-2z4bl\" (UID: \"a384f6e7-e487-4749-b0d0-6b679aa60fae\") " pod="openshift-monitoring/node-exporter-2z4bl" Sep 17 10:12:29.655825 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:29.655758 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a384f6e7-e487-4749-b0d0-6b679aa60fae-root\") pod \"node-exporter-2z4bl\" (UID: \"a384f6e7-e487-4749-b0d0-6b679aa60fae\") " pod="openshift-monitoring/node-exporter-2z4bl" Sep 17 10:12:29.656010 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:29.655838 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a384f6e7-e487-4749-b0d0-6b679aa60fae-root\") pod \"node-exporter-2z4bl\" (UID: \"a384f6e7-e487-4749-b0d0-6b679aa60fae\") " pod="openshift-monitoring/node-exporter-2z4bl" Sep 17 10:12:29.656010 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:29.655969 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a384f6e7-e487-4749-b0d0-6b679aa60fae-node-exporter-wtmp\") pod \"node-exporter-2z4bl\" (UID: \"a384f6e7-e487-4749-b0d0-6b679aa60fae\") " pod="openshift-monitoring/node-exporter-2z4bl" Sep 17 10:12:29.656010 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:29.656005 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a384f6e7-e487-4749-b0d0-6b679aa60fae-sys\") pod \"node-exporter-2z4bl\" (UID: \"a384f6e7-e487-4749-b0d0-6b679aa60fae\") " pod="openshift-monitoring/node-exporter-2z4bl" Sep 17 10:12:29.656126 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:29.656021 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rmz9\" (UniqueName: \"kubernetes.io/projected/a384f6e7-e487-4749-b0d0-6b679aa60fae-kube-api-access-9rmz9\") pod \"node-exporter-2z4bl\" (UID: \"a384f6e7-e487-4749-b0d0-6b679aa60fae\") " pod="openshift-monitoring/node-exporter-2z4bl" Sep 17 10:12:29.656126 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:29.656039 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a384f6e7-e487-4749-b0d0-6b679aa60fae-node-exporter-tls\") pod \"node-exporter-2z4bl\" (UID: \"a384f6e7-e487-4749-b0d0-6b679aa60fae\") " pod="openshift-monitoring/node-exporter-2z4bl" Sep 17 10:12:29.656126 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:29.656077 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a384f6e7-e487-4749-b0d0-6b679aa60fae-metrics-client-ca\") pod \"node-exporter-2z4bl\" (UID: \"a384f6e7-e487-4749-b0d0-6b679aa60fae\") " pod="openshift-monitoring/node-exporter-2z4bl" Sep 17 10:12:29.656126 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:29.656083 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a384f6e7-e487-4749-b0d0-6b679aa60fae-sys\") pod \"node-exporter-2z4bl\" (UID: \"a384f6e7-e487-4749-b0d0-6b679aa60fae\") " pod="openshift-monitoring/node-exporter-2z4bl" Sep 17 10:12:29.656126 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:29.656099 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a384f6e7-e487-4749-b0d0-6b679aa60fae-node-exporter-accelerators-collector-config\") pod \"node-exporter-2z4bl\" (UID: \"a384f6e7-e487-4749-b0d0-6b679aa60fae\") " pod="openshift-monitoring/node-exporter-2z4bl" Sep 17 10:12:29.656126 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:29.656108 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a384f6e7-e487-4749-b0d0-6b679aa60fae-node-exporter-textfile\") pod \"node-exporter-2z4bl\" (UID: \"a384f6e7-e487-4749-b0d0-6b679aa60fae\") " pod="openshift-monitoring/node-exporter-2z4bl" Sep 17 10:12:29.656126 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:29.656122 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a384f6e7-e487-4749-b0d0-6b679aa60fae-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2z4bl\" (UID: \"a384f6e7-e487-4749-b0d0-6b679aa60fae\") " pod="openshift-monitoring/node-exporter-2z4bl" Sep 17 10:12:29.656437 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:29.656135 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a384f6e7-e487-4749-b0d0-6b679aa60fae-node-exporter-wtmp\") pod \"node-exporter-2z4bl\" (UID: \"a384f6e7-e487-4749-b0d0-6b679aa60fae\") " pod="openshift-monitoring/node-exporter-2z4bl" Sep 17 10:12:29.656620 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:29.656598 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a384f6e7-e487-4749-b0d0-6b679aa60fae-node-exporter-accelerators-collector-config\") pod \"node-exporter-2z4bl\" (UID: \"a384f6e7-e487-4749-b0d0-6b679aa60fae\") " pod="openshift-monitoring/node-exporter-2z4bl" Sep 17 10:12:29.656731 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:29.656694 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a384f6e7-e487-4749-b0d0-6b679aa60fae-metrics-client-ca\") pod \"node-exporter-2z4bl\" (UID: \"a384f6e7-e487-4749-b0d0-6b679aa60fae\") " pod="openshift-monitoring/node-exporter-2z4bl" Sep 17 10:12:29.658557 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:29.658526 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a384f6e7-e487-4749-b0d0-6b679aa60fae-node-exporter-tls\") pod \"node-exporter-2z4bl\" (UID: \"a384f6e7-e487-4749-b0d0-6b679aa60fae\") " pod="openshift-monitoring/node-exporter-2z4bl" Sep 17 10:12:29.668540 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:29.668505 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a384f6e7-e487-4749-b0d0-6b679aa60fae-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2z4bl\" (UID: \"a384f6e7-e487-4749-b0d0-6b679aa60fae\") " pod="openshift-monitoring/node-exporter-2z4bl" Sep 17 10:12:29.670556 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:29.670505 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rmz9\" (UniqueName: \"kubernetes.io/projected/a384f6e7-e487-4749-b0d0-6b679aa60fae-kube-api-access-9rmz9\") pod \"node-exporter-2z4bl\" (UID: \"a384f6e7-e487-4749-b0d0-6b679aa60fae\") " pod="openshift-monitoring/node-exporter-2z4bl" Sep 17 10:12:29.734490 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:29.734452 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-2z4bl" Sep 17 10:12:29.744376 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:12:29.744348 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda384f6e7_e487_4749_b0d0_6b679aa60fae.slice/crio-4ff2770d23c30d69f54daf51abcd02007021155af7732b081f4d267d47636fac WatchSource:0}: Error finding container 4ff2770d23c30d69f54daf51abcd02007021155af7732b081f4d267d47636fac: Status 404 returned error can't find the container with id 4ff2770d23c30d69f54daf51abcd02007021155af7732b081f4d267d47636fac Sep 17 10:12:30.435034 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:30.434996 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2z4bl" event={"ID":"a384f6e7-e487-4749-b0d0-6b679aa60fae","Type":"ContainerStarted","Data":"4ff2770d23c30d69f54daf51abcd02007021155af7732b081f4d267d47636fac"} Sep 17 10:12:31.440198 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:31.440163 2581 generic.go:356] "Generic (PLEG): container finished" podID="a384f6e7-e487-4749-b0d0-6b679aa60fae" containerID="c75f3716ccaf3482d8f49d3dcbbf4cd5672afd370bd362b813b871bc6704e616" exitCode=0 Sep 17 10:12:31.440568 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:31.440211 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2z4bl" event={"ID":"a384f6e7-e487-4749-b0d0-6b679aa60fae","Type":"ContainerDied","Data":"c75f3716ccaf3482d8f49d3dcbbf4cd5672afd370bd362b813b871bc6704e616"} Sep 17 10:12:32.447023 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:32.446991 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2z4bl" event={"ID":"a384f6e7-e487-4749-b0d0-6b679aa60fae","Type":"ContainerStarted","Data":"c054d05ae41cba350c8d4c5e216fe532ba21e93a430044a9e0676adba8518324"} Sep 17 10:12:32.447410 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:32.447045 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2z4bl" event={"ID":"a384f6e7-e487-4749-b0d0-6b679aa60fae","Type":"ContainerStarted","Data":"75ce3c2d5c93d0bbc6769f5cdb0952616ad24494e270ed2ceb7d46aae25eb39d"} Sep 17 10:12:32.474333 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:32.474301 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-56bd659dfc-dqdhc"] Sep 17 10:12:32.476473 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:32.476430 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-2z4bl" podStartSLOduration=2.614422791 podStartE2EDuration="3.476415066s" totalImagesPullingTime="796.448827ms" totalInitContainerRuntime="65.543448ms" isStatefulPod=true podCreationTimestamp="2026-09-17 10:12:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 10:12:32.475057436 +0000 UTC m=+107.023614547" watchObservedRunningTime="2026-09-17 10:12:32.476415066 +0000 UTC m=+107.024972206" Sep 17 10:12:32.477101 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:32.477085 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-56bd659dfc-dqdhc" Sep 17 10:12:32.480249 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:32.480214 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Sep 17 10:12:32.480361 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:32.480260 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Sep 17 10:12:32.480361 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:32.480260 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Sep 17 10:12:32.480361 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:32.480293 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Sep 17 10:12:32.480361 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:32.480320 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-wpv7x\"" Sep 17 10:12:32.480540 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:32.480502 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-em0qj59ethkod\"" Sep 17 10:12:32.480611 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:32.480595 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Sep 17 10:12:32.493883 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:32.493855 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-56bd659dfc-dqdhc"] Sep 17 10:12:32.574991 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:32.574949 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/8453a7d4-d2f3-4b4b-be12-0c6e6faa675d-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-56bd659dfc-dqdhc\" (UID: \"8453a7d4-d2f3-4b4b-be12-0c6e6faa675d\") " pod="openshift-monitoring/thanos-querier-56bd659dfc-dqdhc" Sep 17 10:12:32.574991 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:32.574993 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8453a7d4-d2f3-4b4b-be12-0c6e6faa675d-metrics-client-ca\") pod \"thanos-querier-56bd659dfc-dqdhc\" (UID: \"8453a7d4-d2f3-4b4b-be12-0c6e6faa675d\") " pod="openshift-monitoring/thanos-querier-56bd659dfc-dqdhc" Sep 17 10:12:32.575248 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:32.575122 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8hxd\" (UniqueName: \"kubernetes.io/projected/8453a7d4-d2f3-4b4b-be12-0c6e6faa675d-kube-api-access-m8hxd\") pod \"thanos-querier-56bd659dfc-dqdhc\" (UID: \"8453a7d4-d2f3-4b4b-be12-0c6e6faa675d\") " pod="openshift-monitoring/thanos-querier-56bd659dfc-dqdhc" Sep 17 10:12:32.575248 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:32.575160 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8453a7d4-d2f3-4b4b-be12-0c6e6faa675d-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-56bd659dfc-dqdhc\" (UID: \"8453a7d4-d2f3-4b4b-be12-0c6e6faa675d\") " pod="openshift-monitoring/thanos-querier-56bd659dfc-dqdhc" Sep 17 10:12:32.575248 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:32.575185 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/8453a7d4-d2f3-4b4b-be12-0c6e6faa675d-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-56bd659dfc-dqdhc\" (UID: \"8453a7d4-d2f3-4b4b-be12-0c6e6faa675d\") " pod="openshift-monitoring/thanos-querier-56bd659dfc-dqdhc" Sep 17 10:12:32.575248 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:32.575220 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8453a7d4-d2f3-4b4b-be12-0c6e6faa675d-secret-grpc-tls\") pod \"thanos-querier-56bd659dfc-dqdhc\" (UID: \"8453a7d4-d2f3-4b4b-be12-0c6e6faa675d\") " pod="openshift-monitoring/thanos-querier-56bd659dfc-dqdhc" Sep 17 10:12:32.575374 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:32.575279 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/8453a7d4-d2f3-4b4b-be12-0c6e6faa675d-secret-thanos-querier-tls\") pod \"thanos-querier-56bd659dfc-dqdhc\" (UID: \"8453a7d4-d2f3-4b4b-be12-0c6e6faa675d\") " pod="openshift-monitoring/thanos-querier-56bd659dfc-dqdhc" Sep 17 10:12:32.575374 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:32.575299 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8453a7d4-d2f3-4b4b-be12-0c6e6faa675d-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-56bd659dfc-dqdhc\" (UID: \"8453a7d4-d2f3-4b4b-be12-0c6e6faa675d\") " pod="openshift-monitoring/thanos-querier-56bd659dfc-dqdhc" Sep 17 10:12:32.675669 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:32.675635 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8hxd\" (UniqueName: \"kubernetes.io/projected/8453a7d4-d2f3-4b4b-be12-0c6e6faa675d-kube-api-access-m8hxd\") pod \"thanos-querier-56bd659dfc-dqdhc\" (UID: \"8453a7d4-d2f3-4b4b-be12-0c6e6faa675d\") " pod="openshift-monitoring/thanos-querier-56bd659dfc-dqdhc" Sep 17 10:12:32.675866 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:32.675673 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8453a7d4-d2f3-4b4b-be12-0c6e6faa675d-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-56bd659dfc-dqdhc\" (UID: \"8453a7d4-d2f3-4b4b-be12-0c6e6faa675d\") " pod="openshift-monitoring/thanos-querier-56bd659dfc-dqdhc" Sep 17 10:12:32.675866 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:32.675694 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/8453a7d4-d2f3-4b4b-be12-0c6e6faa675d-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-56bd659dfc-dqdhc\" (UID: \"8453a7d4-d2f3-4b4b-be12-0c6e6faa675d\") " pod="openshift-monitoring/thanos-querier-56bd659dfc-dqdhc" Sep 17 10:12:32.675866 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:32.675709 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8453a7d4-d2f3-4b4b-be12-0c6e6faa675d-secret-grpc-tls\") pod \"thanos-querier-56bd659dfc-dqdhc\" (UID: \"8453a7d4-d2f3-4b4b-be12-0c6e6faa675d\") " pod="openshift-monitoring/thanos-querier-56bd659dfc-dqdhc" Sep 17 10:12:32.675866 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:32.675730 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/8453a7d4-d2f3-4b4b-be12-0c6e6faa675d-secret-thanos-querier-tls\") pod \"thanos-querier-56bd659dfc-dqdhc\" (UID: \"8453a7d4-d2f3-4b4b-be12-0c6e6faa675d\") " pod="openshift-monitoring/thanos-querier-56bd659dfc-dqdhc" Sep 17 10:12:32.675866 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:32.675830 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8453a7d4-d2f3-4b4b-be12-0c6e6faa675d-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-56bd659dfc-dqdhc\" (UID: \"8453a7d4-d2f3-4b4b-be12-0c6e6faa675d\") " pod="openshift-monitoring/thanos-querier-56bd659dfc-dqdhc" Sep 17 10:12:32.676126 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:32.675881 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/8453a7d4-d2f3-4b4b-be12-0c6e6faa675d-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-56bd659dfc-dqdhc\" (UID: \"8453a7d4-d2f3-4b4b-be12-0c6e6faa675d\") " pod="openshift-monitoring/thanos-querier-56bd659dfc-dqdhc" Sep 17 10:12:32.676126 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:32.675912 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8453a7d4-d2f3-4b4b-be12-0c6e6faa675d-metrics-client-ca\") pod \"thanos-querier-56bd659dfc-dqdhc\" (UID: \"8453a7d4-d2f3-4b4b-be12-0c6e6faa675d\") " pod="openshift-monitoring/thanos-querier-56bd659dfc-dqdhc" Sep 17 10:12:32.676669 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:32.676644 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8453a7d4-d2f3-4b4b-be12-0c6e6faa675d-metrics-client-ca\") pod \"thanos-querier-56bd659dfc-dqdhc\" (UID: \"8453a7d4-d2f3-4b4b-be12-0c6e6faa675d\") " pod="openshift-monitoring/thanos-querier-56bd659dfc-dqdhc" Sep 17 10:12:32.678893 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:32.678844 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/8453a7d4-d2f3-4b4b-be12-0c6e6faa675d-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-56bd659dfc-dqdhc\" (UID: \"8453a7d4-d2f3-4b4b-be12-0c6e6faa675d\") " pod="openshift-monitoring/thanos-querier-56bd659dfc-dqdhc" Sep 17 10:12:32.678893 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:32.678867 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8453a7d4-d2f3-4b4b-be12-0c6e6faa675d-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-56bd659dfc-dqdhc\" (UID: \"8453a7d4-d2f3-4b4b-be12-0c6e6faa675d\") " pod="openshift-monitoring/thanos-querier-56bd659dfc-dqdhc" Sep 17 10:12:32.679043 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:32.679017 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8453a7d4-d2f3-4b4b-be12-0c6e6faa675d-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-56bd659dfc-dqdhc\" (UID: \"8453a7d4-d2f3-4b4b-be12-0c6e6faa675d\") " pod="openshift-monitoring/thanos-querier-56bd659dfc-dqdhc" Sep 17 10:12:32.679121 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:32.679073 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8453a7d4-d2f3-4b4b-be12-0c6e6faa675d-secret-grpc-tls\") pod \"thanos-querier-56bd659dfc-dqdhc\" (UID: \"8453a7d4-d2f3-4b4b-be12-0c6e6faa675d\") " pod="openshift-monitoring/thanos-querier-56bd659dfc-dqdhc" Sep 17 10:12:32.679173 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:32.679137 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/8453a7d4-d2f3-4b4b-be12-0c6e6faa675d-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-56bd659dfc-dqdhc\" (UID: \"8453a7d4-d2f3-4b4b-be12-0c6e6faa675d\") " pod="openshift-monitoring/thanos-querier-56bd659dfc-dqdhc" Sep 17 10:12:32.679558 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:32.679536 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/8453a7d4-d2f3-4b4b-be12-0c6e6faa675d-secret-thanos-querier-tls\") pod \"thanos-querier-56bd659dfc-dqdhc\" (UID: \"8453a7d4-d2f3-4b4b-be12-0c6e6faa675d\") " pod="openshift-monitoring/thanos-querier-56bd659dfc-dqdhc" Sep 17 10:12:32.691452 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:32.691424 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8hxd\" (UniqueName: \"kubernetes.io/projected/8453a7d4-d2f3-4b4b-be12-0c6e6faa675d-kube-api-access-m8hxd\") pod \"thanos-querier-56bd659dfc-dqdhc\" (UID: \"8453a7d4-d2f3-4b4b-be12-0c6e6faa675d\") " pod="openshift-monitoring/thanos-querier-56bd659dfc-dqdhc" Sep 17 10:12:32.785496 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:32.785453 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-56bd659dfc-dqdhc" Sep 17 10:12:33.060320 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:33.060245 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-56bd659dfc-dqdhc"] Sep 17 10:12:33.063529 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:12:33.063500 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8453a7d4_d2f3_4b4b_be12_0c6e6faa675d.slice/crio-c3d8a99d4f8a524242c51dd6cb57439fee9bfc53c69f5133ff791229ab7cb89e WatchSource:0}: Error finding container c3d8a99d4f8a524242c51dd6cb57439fee9bfc53c69f5133ff791229ab7cb89e: Status 404 returned error can't find the container with id c3d8a99d4f8a524242c51dd6cb57439fee9bfc53c69f5133ff791229ab7cb89e Sep 17 10:12:33.451720 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:33.451631 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-56bd659dfc-dqdhc" event={"ID":"8453a7d4-d2f3-4b4b-be12-0c6e6faa675d","Type":"ContainerStarted","Data":"c3d8a99d4f8a524242c51dd6cb57439fee9bfc53c69f5133ff791229ab7cb89e"} Sep 17 10:12:34.091109 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:34.091029 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-fkbh2"] Sep 17 10:12:34.099830 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:34.099802 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-fkbh2" Sep 17 10:12:34.102617 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:34.102588 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Sep 17 10:12:34.102934 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:34.102642 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-44tmj\"" Sep 17 10:12:34.105587 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:34.105565 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-fkbh2"] Sep 17 10:12:34.185343 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:34.185302 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/43bccb30-f06f-4511-8927-b8face2f6dda-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-fkbh2\" (UID: \"43bccb30-f06f-4511-8927-b8face2f6dda\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-fkbh2" Sep 17 10:12:34.185591 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:34.185365 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9bfz\" (UniqueName: \"kubernetes.io/projected/43bccb30-f06f-4511-8927-b8face2f6dda-kube-api-access-k9bfz\") pod \"monitoring-plugin-5d85778b9d-fkbh2\" (UID: \"43bccb30-f06f-4511-8927-b8face2f6dda\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-fkbh2" Sep 17 10:12:34.286193 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:34.286157 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/43bccb30-f06f-4511-8927-b8face2f6dda-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-fkbh2\" (UID: \"43bccb30-f06f-4511-8927-b8face2f6dda\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-fkbh2" Sep 17 10:12:34.286193 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:34.286194 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9bfz\" (UniqueName: \"kubernetes.io/projected/43bccb30-f06f-4511-8927-b8face2f6dda-kube-api-access-k9bfz\") pod \"monitoring-plugin-5d85778b9d-fkbh2\" (UID: \"43bccb30-f06f-4511-8927-b8face2f6dda\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-fkbh2" Sep 17 10:12:34.286441 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:12:34.286308 2581 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Sep 17 10:12:34.286441 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:12:34.286383 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43bccb30-f06f-4511-8927-b8face2f6dda-monitoring-plugin-cert podName:43bccb30-f06f-4511-8927-b8face2f6dda nodeName:}" failed. No retries permitted until 2026-09-17 10:12:34.786363216 +0000 UTC m=+109.334920319 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/43bccb30-f06f-4511-8927-b8face2f6dda-monitoring-plugin-cert") pod "monitoring-plugin-5d85778b9d-fkbh2" (UID: "43bccb30-f06f-4511-8927-b8face2f6dda") : secret "monitoring-plugin-cert" not found Sep 17 10:12:34.296380 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:34.296349 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9bfz\" (UniqueName: \"kubernetes.io/projected/43bccb30-f06f-4511-8927-b8face2f6dda-kube-api-access-k9bfz\") pod \"monitoring-plugin-5d85778b9d-fkbh2\" (UID: \"43bccb30-f06f-4511-8927-b8face2f6dda\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-fkbh2" Sep 17 10:12:34.788774 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:34.788732 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/43bccb30-f06f-4511-8927-b8face2f6dda-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-fkbh2\" (UID: \"43bccb30-f06f-4511-8927-b8face2f6dda\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-fkbh2" Sep 17 10:12:34.791502 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:34.791475 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/43bccb30-f06f-4511-8927-b8face2f6dda-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-fkbh2\" (UID: \"43bccb30-f06f-4511-8927-b8face2f6dda\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-fkbh2" Sep 17 10:12:35.013864 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:35.013831 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-fkbh2" Sep 17 10:12:35.292887 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:35.292862 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-fkbh2"] Sep 17 10:12:35.297897 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:12:35.297865 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43bccb30_f06f_4511_8927_b8face2f6dda.slice/crio-7ad54032a06ecca4d07bc73938d6275b43ac9c8cef613fdc502d8d980901200a WatchSource:0}: Error finding container 7ad54032a06ecca4d07bc73938d6275b43ac9c8cef613fdc502d8d980901200a: Status 404 returned error can't find the container with id 7ad54032a06ecca4d07bc73938d6275b43ac9c8cef613fdc502d8d980901200a Sep 17 10:12:35.461850 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:35.461808 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-fkbh2" event={"ID":"43bccb30-f06f-4511-8927-b8face2f6dda","Type":"ContainerStarted","Data":"7ad54032a06ecca4d07bc73938d6275b43ac9c8cef613fdc502d8d980901200a"} Sep 17 10:12:35.464200 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:35.464174 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-56bd659dfc-dqdhc" event={"ID":"8453a7d4-d2f3-4b4b-be12-0c6e6faa675d","Type":"ContainerStarted","Data":"b250849baa2dfcd58aeb1c6a56dfa1e7d512813e2051a078e99dd9658c2e0b06"} Sep 17 10:12:35.464321 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:35.464239 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-56bd659dfc-dqdhc" event={"ID":"8453a7d4-d2f3-4b4b-be12-0c6e6faa675d","Type":"ContainerStarted","Data":"ef3b0947a00c0f2dcb0f7fcc361831128ec4c65e6e98d8771c8334290f6b3951"} Sep 17 10:12:35.464321 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:35.464266 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-56bd659dfc-dqdhc" event={"ID":"8453a7d4-d2f3-4b4b-be12-0c6e6faa675d","Type":"ContainerStarted","Data":"e6965afa37297999b974b6c0600c1aa6e288178bb02bb374a11fb9791b9fba5e"} Sep 17 10:12:36.425268 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:36.425186 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-ds6sz" Sep 17 10:12:36.473483 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:36.473428 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-56bd659dfc-dqdhc" event={"ID":"8453a7d4-d2f3-4b4b-be12-0c6e6faa675d","Type":"ContainerStarted","Data":"f189db1721a33686d4cae5cbd27a319cd080b5d63e1535a632c1f4c6b2f3c6ef"} Sep 17 10:12:36.473656 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:36.473515 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-56bd659dfc-dqdhc" event={"ID":"8453a7d4-d2f3-4b4b-be12-0c6e6faa675d","Type":"ContainerStarted","Data":"d5bd2f54a1c45c8980ecce8730b89ab10961b635c9eee44cb9c9361622e84a08"} Sep 17 10:12:36.473656 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:36.473554 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-56bd659dfc-dqdhc" event={"ID":"8453a7d4-d2f3-4b4b-be12-0c6e6faa675d","Type":"ContainerStarted","Data":"756bc528642717d148016a8e44759062b425819a2e777d7a1b727958213d3861"} Sep 17 10:12:36.473656 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:36.473626 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-56bd659dfc-dqdhc" Sep 17 10:12:36.497474 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:36.497405 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-56bd659dfc-dqdhc" podStartSLOduration=1.8103711900000001 podStartE2EDuration="4.497383129s" totalImagesPullingTime="2.687011939s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 10:12:32 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 10:12:36.494360224 +0000 UTC m=+111.042917340" watchObservedRunningTime="2026-09-17 10:12:36.497383129 +0000 UTC m=+111.045940241" Sep 17 10:12:37.478446 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:37.478409 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-fkbh2" event={"ID":"43bccb30-f06f-4511-8927-b8face2f6dda","Type":"ContainerStarted","Data":"fc682ba6e3df45121fd78985d813610bcb7a9ba02b483321ae0a870ddef62ff0"} Sep 17 10:12:37.478898 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:37.478631 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-fkbh2" Sep 17 10:12:37.483078 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:37.483059 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-fkbh2" Sep 17 10:12:37.495370 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:37.495305 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-fkbh2" podStartSLOduration=2.16389568 podStartE2EDuration="3.495291564s" totalImagesPullingTime="1.331395884s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 10:12:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 10:12:37.49420032 +0000 UTC m=+112.042757432" watchObservedRunningTime="2026-09-17 10:12:37.495291564 +0000 UTC m=+112.043848675" Sep 17 10:12:42.485415 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:42.485387 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-56bd659dfc-dqdhc" Sep 17 10:12:42.938747 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:42.938654 2581 scope.go:117] "RemoveContainer" containerID="a6dc1122273355ff1bf6475889fb31b10461ff4435c72f5a7fb6c7dd1fae5200" Sep 17 10:12:43.503488 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:43.503458 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-b9wnf_762d3e10-7b7d-4ce9-9773-90a4fe41b474/console-operator/2.log" Sep 17 10:12:43.503978 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:43.503595 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-b9wnf" event={"ID":"762d3e10-7b7d-4ce9-9773-90a4fe41b474","Type":"ContainerStarted","Data":"807f5d515061013fe35161628ec58ea291d1f193ad86d692c11348f00940ed23"} Sep 17 10:12:43.504641 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:43.504616 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-b9wnf" Sep 17 10:12:43.522910 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:43.522848 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-696884fb9c-b9wnf" podStartSLOduration=55.152967235 podStartE2EDuration="57.522829306s" totalImagesPullingTime="2.369862071s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 10:11:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 10:12:43.521154312 +0000 UTC m=+118.069711424" watchObservedRunningTime="2026-09-17 10:12:43.522829306 +0000 UTC m=+118.071386418" Sep 17 10:12:43.572839 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:43.572808 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-696884fb9c-b9wnf" Sep 17 10:12:43.749824 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:43.749733 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-76f99584f4-klnsv"] Sep 17 10:12:43.752624 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:43.752596 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-klnsv" Sep 17 10:12:43.755380 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:43.755319 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-pr5tz\"" Sep 17 10:12:43.755380 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:43.755333 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 17 10:12:43.755380 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:43.755348 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 17 10:12:43.765338 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:43.765311 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-klnsv"] Sep 17 10:12:43.859969 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:43.859927 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tr2vr\" (UniqueName: \"kubernetes.io/projected/97aaddeb-c0ee-423c-a916-f6ad15782799-kube-api-access-tr2vr\") pod \"downloads-76f99584f4-klnsv\" (UID: \"97aaddeb-c0ee-423c-a916-f6ad15782799\") " pod="openshift-console/downloads-76f99584f4-klnsv" Sep 17 10:12:43.960516 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:43.960481 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tr2vr\" (UniqueName: \"kubernetes.io/projected/97aaddeb-c0ee-423c-a916-f6ad15782799-kube-api-access-tr2vr\") pod \"downloads-76f99584f4-klnsv\" (UID: \"97aaddeb-c0ee-423c-a916-f6ad15782799\") " pod="openshift-console/downloads-76f99584f4-klnsv" Sep 17 10:12:43.970658 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:43.970624 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tr2vr\" (UniqueName: \"kubernetes.io/projected/97aaddeb-c0ee-423c-a916-f6ad15782799-kube-api-access-tr2vr\") pod \"downloads-76f99584f4-klnsv\" (UID: \"97aaddeb-c0ee-423c-a916-f6ad15782799\") " pod="openshift-console/downloads-76f99584f4-klnsv" Sep 17 10:12:44.062703 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:44.062617 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-klnsv" Sep 17 10:12:44.335890 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:44.335811 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-klnsv"] Sep 17 10:12:44.339188 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:12:44.339161 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97aaddeb_c0ee_423c_a916_f6ad15782799.slice/crio-06982e19e095648d41d1f2091024e60fcc01358bf062fe5db3cdd5aab75605a1 WatchSource:0}: Error finding container 06982e19e095648d41d1f2091024e60fcc01358bf062fe5db3cdd5aab75605a1: Status 404 returned error can't find the container with id 06982e19e095648d41d1f2091024e60fcc01358bf062fe5db3cdd5aab75605a1 Sep 17 10:12:44.508942 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:44.508895 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-klnsv" event={"ID":"97aaddeb-c0ee-423c-a916-f6ad15782799","Type":"ContainerStarted","Data":"06982e19e095648d41d1f2091024e60fcc01358bf062fe5db3cdd5aab75605a1"} Sep 17 10:12:54.097563 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:54.097527 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-798b5b8784-qz9zj"] Sep 17 10:12:54.101640 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:54.101612 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-798b5b8784-qz9zj" Sep 17 10:12:54.105976 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:54.105727 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 17 10:12:54.105976 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:54.105738 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 17 10:12:54.105976 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:54.105733 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-6xrtj\"" Sep 17 10:12:54.105976 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:54.105745 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 17 10:12:54.105976 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:54.105733 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 17 10:12:54.105976 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:54.105806 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 17 10:12:54.113670 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:54.113646 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-798b5b8784-qz9zj"] Sep 17 10:12:54.146593 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:54.146557 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/437ef216-f348-40b6-9c84-29580f39ac68-service-ca\") pod \"console-798b5b8784-qz9zj\" (UID: \"437ef216-f348-40b6-9c84-29580f39ac68\") " pod="openshift-console/console-798b5b8784-qz9zj" Sep 17 10:12:54.146829 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:54.146613 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/437ef216-f348-40b6-9c84-29580f39ac68-console-oauth-config\") pod \"console-798b5b8784-qz9zj\" (UID: \"437ef216-f348-40b6-9c84-29580f39ac68\") " pod="openshift-console/console-798b5b8784-qz9zj" Sep 17 10:12:54.146829 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:54.146675 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/437ef216-f348-40b6-9c84-29580f39ac68-console-config\") pod \"console-798b5b8784-qz9zj\" (UID: \"437ef216-f348-40b6-9c84-29580f39ac68\") " pod="openshift-console/console-798b5b8784-qz9zj" Sep 17 10:12:54.146829 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:54.146773 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/437ef216-f348-40b6-9c84-29580f39ac68-oauth-serving-cert\") pod \"console-798b5b8784-qz9zj\" (UID: \"437ef216-f348-40b6-9c84-29580f39ac68\") " pod="openshift-console/console-798b5b8784-qz9zj" Sep 17 10:12:54.147072 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:54.146827 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwplr\" (UniqueName: \"kubernetes.io/projected/437ef216-f348-40b6-9c84-29580f39ac68-kube-api-access-vwplr\") pod \"console-798b5b8784-qz9zj\" (UID: \"437ef216-f348-40b6-9c84-29580f39ac68\") " pod="openshift-console/console-798b5b8784-qz9zj" Sep 17 10:12:54.147072 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:54.146963 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/437ef216-f348-40b6-9c84-29580f39ac68-console-serving-cert\") pod \"console-798b5b8784-qz9zj\" (UID: \"437ef216-f348-40b6-9c84-29580f39ac68\") " pod="openshift-console/console-798b5b8784-qz9zj" Sep 17 10:12:54.247703 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:54.247666 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/437ef216-f348-40b6-9c84-29580f39ac68-console-serving-cert\") pod \"console-798b5b8784-qz9zj\" (UID: \"437ef216-f348-40b6-9c84-29580f39ac68\") " pod="openshift-console/console-798b5b8784-qz9zj" Sep 17 10:12:54.247899 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:54.247712 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/437ef216-f348-40b6-9c84-29580f39ac68-service-ca\") pod \"console-798b5b8784-qz9zj\" (UID: \"437ef216-f348-40b6-9c84-29580f39ac68\") " pod="openshift-console/console-798b5b8784-qz9zj" Sep 17 10:12:54.247899 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:54.247748 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/437ef216-f348-40b6-9c84-29580f39ac68-console-oauth-config\") pod \"console-798b5b8784-qz9zj\" (UID: \"437ef216-f348-40b6-9c84-29580f39ac68\") " pod="openshift-console/console-798b5b8784-qz9zj" Sep 17 10:12:54.247899 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:54.247771 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/437ef216-f348-40b6-9c84-29580f39ac68-console-config\") pod \"console-798b5b8784-qz9zj\" (UID: \"437ef216-f348-40b6-9c84-29580f39ac68\") " pod="openshift-console/console-798b5b8784-qz9zj" Sep 17 10:12:54.247899 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:54.247836 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/437ef216-f348-40b6-9c84-29580f39ac68-oauth-serving-cert\") pod \"console-798b5b8784-qz9zj\" (UID: \"437ef216-f348-40b6-9c84-29580f39ac68\") " pod="openshift-console/console-798b5b8784-qz9zj" Sep 17 10:12:54.247899 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:54.247857 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwplr\" (UniqueName: \"kubernetes.io/projected/437ef216-f348-40b6-9c84-29580f39ac68-kube-api-access-vwplr\") pod \"console-798b5b8784-qz9zj\" (UID: \"437ef216-f348-40b6-9c84-29580f39ac68\") " pod="openshift-console/console-798b5b8784-qz9zj" Sep 17 10:12:54.248723 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:54.248668 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/437ef216-f348-40b6-9c84-29580f39ac68-service-ca\") pod \"console-798b5b8784-qz9zj\" (UID: \"437ef216-f348-40b6-9c84-29580f39ac68\") " pod="openshift-console/console-798b5b8784-qz9zj" Sep 17 10:12:54.248861 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:54.248718 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/437ef216-f348-40b6-9c84-29580f39ac68-console-config\") pod \"console-798b5b8784-qz9zj\" (UID: \"437ef216-f348-40b6-9c84-29580f39ac68\") " pod="openshift-console/console-798b5b8784-qz9zj" Sep 17 10:12:54.248861 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:54.248819 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/437ef216-f348-40b6-9c84-29580f39ac68-oauth-serving-cert\") pod \"console-798b5b8784-qz9zj\" (UID: \"437ef216-f348-40b6-9c84-29580f39ac68\") " pod="openshift-console/console-798b5b8784-qz9zj" Sep 17 10:12:54.250517 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:54.250490 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/437ef216-f348-40b6-9c84-29580f39ac68-console-oauth-config\") pod \"console-798b5b8784-qz9zj\" (UID: \"437ef216-f348-40b6-9c84-29580f39ac68\") " pod="openshift-console/console-798b5b8784-qz9zj" Sep 17 10:12:54.250686 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:54.250660 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/437ef216-f348-40b6-9c84-29580f39ac68-console-serving-cert\") pod \"console-798b5b8784-qz9zj\" (UID: \"437ef216-f348-40b6-9c84-29580f39ac68\") " pod="openshift-console/console-798b5b8784-qz9zj" Sep 17 10:12:54.259343 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:54.259316 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwplr\" (UniqueName: \"kubernetes.io/projected/437ef216-f348-40b6-9c84-29580f39ac68-kube-api-access-vwplr\") pod \"console-798b5b8784-qz9zj\" (UID: \"437ef216-f348-40b6-9c84-29580f39ac68\") " pod="openshift-console/console-798b5b8784-qz9zj" Sep 17 10:12:54.412952 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:54.412857 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-798b5b8784-qz9zj" Sep 17 10:12:55.658179 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:55.658139 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c84a7003-1537-4470-8094-ea10a89a86a2-metrics-certs\") pod \"network-metrics-daemon-wfccr\" (UID: \"c84a7003-1537-4470-8094-ea10a89a86a2\") " pod="openshift-multus/network-metrics-daemon-wfccr" Sep 17 10:12:55.660833 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:55.660805 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c84a7003-1537-4470-8094-ea10a89a86a2-metrics-certs\") pod \"network-metrics-daemon-wfccr\" (UID: \"c84a7003-1537-4470-8094-ea10a89a86a2\") " pod="openshift-multus/network-metrics-daemon-wfccr" Sep 17 10:12:55.956319 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:55.956241 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-thcs5\"" Sep 17 10:12:55.964061 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:12:55.964030 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wfccr" Sep 17 10:13:00.470722 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:13:00.470697 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-798b5b8784-qz9zj"] Sep 17 10:13:00.473757 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:13:00.473727 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod437ef216_f348_40b6_9c84_29580f39ac68.slice/crio-95f6aa5c982c55b65e27e54f99afd98b501bcb2d9a1adc3bc3e9323ee6b439e0 WatchSource:0}: Error finding container 95f6aa5c982c55b65e27e54f99afd98b501bcb2d9a1adc3bc3e9323ee6b439e0: Status 404 returned error can't find the container with id 95f6aa5c982c55b65e27e54f99afd98b501bcb2d9a1adc3bc3e9323ee6b439e0 Sep 17 10:13:00.482763 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:13:00.482737 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-wfccr"] Sep 17 10:13:00.484653 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:13:00.484626 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc84a7003_1537_4470_8094_ea10a89a86a2.slice/crio-4df3d16bbc472e56d82c6cc45c4929c8fba6627b8c7a38d4172ebc6b13a801d9 WatchSource:0}: Error finding container 4df3d16bbc472e56d82c6cc45c4929c8fba6627b8c7a38d4172ebc6b13a801d9: Status 404 returned error can't find the container with id 4df3d16bbc472e56d82c6cc45c4929c8fba6627b8c7a38d4172ebc6b13a801d9 Sep 17 10:13:00.581163 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:13:00.581119 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wfccr" event={"ID":"c84a7003-1537-4470-8094-ea10a89a86a2","Type":"ContainerStarted","Data":"4df3d16bbc472e56d82c6cc45c4929c8fba6627b8c7a38d4172ebc6b13a801d9"} Sep 17 10:13:00.583155 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:13:00.583114 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-klnsv" event={"ID":"97aaddeb-c0ee-423c-a916-f6ad15782799","Type":"ContainerStarted","Data":"11176cb6d38f8200450c629d351e8eafd7a84e95628683e29ee3165fe46b5eac"} Sep 17 10:13:00.583379 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:13:00.583353 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-76f99584f4-klnsv" Sep 17 10:13:00.584652 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:13:00.584625 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-798b5b8784-qz9zj" event={"ID":"437ef216-f348-40b6-9c84-29580f39ac68","Type":"ContainerStarted","Data":"95f6aa5c982c55b65e27e54f99afd98b501bcb2d9a1adc3bc3e9323ee6b439e0"} Sep 17 10:13:00.601845 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:13:00.601807 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-76f99584f4-klnsv" Sep 17 10:13:00.603103 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:13:00.603050 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-76f99584f4-klnsv" podStartSLOduration=1.652542272 podStartE2EDuration="17.603036264s" totalImagesPullingTime="15.950493992s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 10:12:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 10:13:00.602282508 +0000 UTC m=+135.150839621" watchObservedRunningTime="2026-09-17 10:13:00.603036264 +0000 UTC m=+135.151593374" Sep 17 10:13:02.597029 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:13:02.596985 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wfccr" event={"ID":"c84a7003-1537-4470-8094-ea10a89a86a2","Type":"ContainerStarted","Data":"1e2480159f974a3d8f417343860ce5a53832973adb38a7868ebe044de961b52b"} Sep 17 10:13:02.597493 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:13:02.597061 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wfccr" event={"ID":"c84a7003-1537-4470-8094-ea10a89a86a2","Type":"ContainerStarted","Data":"8f210533e5b608784292221a59187a5d2281331ac01a90adf7853e7aaa9c7156"} Sep 17 10:13:02.626353 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:13:02.626282 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-wfccr" podStartSLOduration=135.562779621 podStartE2EDuration="2m16.626259033s" totalImagesPullingTime="1.063479412s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 10:10:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 10:13:02.625515022 +0000 UTC m=+137.174072135" watchObservedRunningTime="2026-09-17 10:13:02.626259033 +0000 UTC m=+137.174816145" Sep 17 10:13:02.711436 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:13:02.711397 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-74cfbd76cd-kfnrc"] Sep 17 10:13:02.737880 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:13:02.737774 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-74cfbd76cd-kfnrc"] Sep 17 10:13:02.738052 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:13:02.737944 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74cfbd76cd-kfnrc" Sep 17 10:13:02.749817 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:13:02.749773 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Sep 17 10:13:02.822856 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:13:02.822502 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/724f7002-97bb-4bd5-869c-6676b87d5a8e-console-serving-cert\") pod \"console-74cfbd76cd-kfnrc\" (UID: \"724f7002-97bb-4bd5-869c-6676b87d5a8e\") " pod="openshift-console/console-74cfbd76cd-kfnrc" Sep 17 10:13:02.822856 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:13:02.822552 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f65t7\" (UniqueName: \"kubernetes.io/projected/724f7002-97bb-4bd5-869c-6676b87d5a8e-kube-api-access-f65t7\") pod \"console-74cfbd76cd-kfnrc\" (UID: \"724f7002-97bb-4bd5-869c-6676b87d5a8e\") " pod="openshift-console/console-74cfbd76cd-kfnrc" Sep 17 10:13:02.822856 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:13:02.822593 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/724f7002-97bb-4bd5-869c-6676b87d5a8e-oauth-serving-cert\") pod \"console-74cfbd76cd-kfnrc\" (UID: \"724f7002-97bb-4bd5-869c-6676b87d5a8e\") " pod="openshift-console/console-74cfbd76cd-kfnrc" Sep 17 10:13:02.822856 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:13:02.822619 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/724f7002-97bb-4bd5-869c-6676b87d5a8e-trusted-ca-bundle\") pod \"console-74cfbd76cd-kfnrc\" (UID: \"724f7002-97bb-4bd5-869c-6676b87d5a8e\") " pod="openshift-console/console-74cfbd76cd-kfnrc" Sep 17 10:13:02.822856 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:13:02.822655 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/724f7002-97bb-4bd5-869c-6676b87d5a8e-service-ca\") pod \"console-74cfbd76cd-kfnrc\" (UID: \"724f7002-97bb-4bd5-869c-6676b87d5a8e\") " pod="openshift-console/console-74cfbd76cd-kfnrc" Sep 17 10:13:02.822856 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:13:02.822680 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/724f7002-97bb-4bd5-869c-6676b87d5a8e-console-config\") pod \"console-74cfbd76cd-kfnrc\" (UID: \"724f7002-97bb-4bd5-869c-6676b87d5a8e\") " pod="openshift-console/console-74cfbd76cd-kfnrc" Sep 17 10:13:02.822856 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:13:02.822703 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/724f7002-97bb-4bd5-869c-6676b87d5a8e-console-oauth-config\") pod \"console-74cfbd76cd-kfnrc\" (UID: \"724f7002-97bb-4bd5-869c-6676b87d5a8e\") " pod="openshift-console/console-74cfbd76cd-kfnrc" Sep 17 10:13:02.923451 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:13:02.923355 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/724f7002-97bb-4bd5-869c-6676b87d5a8e-oauth-serving-cert\") pod \"console-74cfbd76cd-kfnrc\" (UID: \"724f7002-97bb-4bd5-869c-6676b87d5a8e\") " pod="openshift-console/console-74cfbd76cd-kfnrc" Sep 17 10:13:02.923451 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:13:02.923422 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/724f7002-97bb-4bd5-869c-6676b87d5a8e-trusted-ca-bundle\") pod \"console-74cfbd76cd-kfnrc\" (UID: \"724f7002-97bb-4bd5-869c-6676b87d5a8e\") " pod="openshift-console/console-74cfbd76cd-kfnrc" Sep 17 10:13:02.923642 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:13:02.923472 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/724f7002-97bb-4bd5-869c-6676b87d5a8e-service-ca\") pod \"console-74cfbd76cd-kfnrc\" (UID: \"724f7002-97bb-4bd5-869c-6676b87d5a8e\") " pod="openshift-console/console-74cfbd76cd-kfnrc" Sep 17 10:13:02.923642 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:13:02.923522 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/724f7002-97bb-4bd5-869c-6676b87d5a8e-console-config\") pod \"console-74cfbd76cd-kfnrc\" (UID: \"724f7002-97bb-4bd5-869c-6676b87d5a8e\") " pod="openshift-console/console-74cfbd76cd-kfnrc" Sep 17 10:13:02.923642 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:13:02.923563 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/724f7002-97bb-4bd5-869c-6676b87d5a8e-console-oauth-config\") pod \"console-74cfbd76cd-kfnrc\" (UID: \"724f7002-97bb-4bd5-869c-6676b87d5a8e\") " pod="openshift-console/console-74cfbd76cd-kfnrc" Sep 17 10:13:02.923815 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:13:02.923673 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/724f7002-97bb-4bd5-869c-6676b87d5a8e-console-serving-cert\") pod \"console-74cfbd76cd-kfnrc\" (UID: \"724f7002-97bb-4bd5-869c-6676b87d5a8e\") " pod="openshift-console/console-74cfbd76cd-kfnrc" Sep 17 10:13:02.923815 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:13:02.923705 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f65t7\" (UniqueName: \"kubernetes.io/projected/724f7002-97bb-4bd5-869c-6676b87d5a8e-kube-api-access-f65t7\") pod \"console-74cfbd76cd-kfnrc\" (UID: \"724f7002-97bb-4bd5-869c-6676b87d5a8e\") " pod="openshift-console/console-74cfbd76cd-kfnrc" Sep 17 10:13:02.924916 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:13:02.924894 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/724f7002-97bb-4bd5-869c-6676b87d5a8e-trusted-ca-bundle\") pod \"console-74cfbd76cd-kfnrc\" (UID: \"724f7002-97bb-4bd5-869c-6676b87d5a8e\") " pod="openshift-console/console-74cfbd76cd-kfnrc" Sep 17 10:13:02.925029 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:13:02.924916 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/724f7002-97bb-4bd5-869c-6676b87d5a8e-service-ca\") pod \"console-74cfbd76cd-kfnrc\" (UID: \"724f7002-97bb-4bd5-869c-6676b87d5a8e\") " pod="openshift-console/console-74cfbd76cd-kfnrc" Sep 17 10:13:02.925413 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:13:02.925392 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/724f7002-97bb-4bd5-869c-6676b87d5a8e-oauth-serving-cert\") pod \"console-74cfbd76cd-kfnrc\" (UID: \"724f7002-97bb-4bd5-869c-6676b87d5a8e\") " pod="openshift-console/console-74cfbd76cd-kfnrc" Sep 17 10:13:02.925519 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:13:02.925487 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/724f7002-97bb-4bd5-869c-6676b87d5a8e-console-config\") pod \"console-74cfbd76cd-kfnrc\" (UID: \"724f7002-97bb-4bd5-869c-6676b87d5a8e\") " pod="openshift-console/console-74cfbd76cd-kfnrc" Sep 17 10:13:02.927339 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:13:02.927296 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/724f7002-97bb-4bd5-869c-6676b87d5a8e-console-serving-cert\") pod \"console-74cfbd76cd-kfnrc\" (UID: \"724f7002-97bb-4bd5-869c-6676b87d5a8e\") " pod="openshift-console/console-74cfbd76cd-kfnrc" Sep 17 10:13:02.927772 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:13:02.927725 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/724f7002-97bb-4bd5-869c-6676b87d5a8e-console-oauth-config\") pod \"console-74cfbd76cd-kfnrc\" (UID: \"724f7002-97bb-4bd5-869c-6676b87d5a8e\") " pod="openshift-console/console-74cfbd76cd-kfnrc" Sep 17 10:13:02.938341 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:13:02.938279 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f65t7\" (UniqueName: \"kubernetes.io/projected/724f7002-97bb-4bd5-869c-6676b87d5a8e-kube-api-access-f65t7\") pod \"console-74cfbd76cd-kfnrc\" (UID: \"724f7002-97bb-4bd5-869c-6676b87d5a8e\") " pod="openshift-console/console-74cfbd76cd-kfnrc" Sep 17 10:13:03.053558 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:13:03.053520 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74cfbd76cd-kfnrc" Sep 17 10:13:03.361996 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:13:03.361910 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-74cfbd76cd-kfnrc"] Sep 17 10:13:04.486724 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:13:04.486685 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod724f7002_97bb_4bd5_869c_6676b87d5a8e.slice/crio-34d46fee99720f2126127c28760ef19c601670886f4b770b7379d6559b5c3d36 WatchSource:0}: Error finding container 34d46fee99720f2126127c28760ef19c601670886f4b770b7379d6559b5c3d36: Status 404 returned error can't find the container with id 34d46fee99720f2126127c28760ef19c601670886f4b770b7379d6559b5c3d36 Sep 17 10:13:04.613571 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:13:04.613535 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74cfbd76cd-kfnrc" event={"ID":"724f7002-97bb-4bd5-869c-6676b87d5a8e","Type":"ContainerStarted","Data":"34d46fee99720f2126127c28760ef19c601670886f4b770b7379d6559b5c3d36"} Sep 17 10:13:05.619831 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:13:05.619764 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74cfbd76cd-kfnrc" event={"ID":"724f7002-97bb-4bd5-869c-6676b87d5a8e","Type":"ContainerStarted","Data":"9414524e8e3d323d3cd5d5ffc05f526d8d3af0aa467e4d7c62268ae99f112946"} Sep 17 10:13:05.621728 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:13:05.621699 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-798b5b8784-qz9zj" event={"ID":"437ef216-f348-40b6-9c84-29580f39ac68","Type":"ContainerStarted","Data":"30678f278ae9635419b8ff46cd7fee5e1e5784f3793458c71ebf48e578775a24"} Sep 17 10:13:05.643676 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:13:05.643596 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-74cfbd76cd-kfnrc" podStartSLOduration=3.075983795 podStartE2EDuration="3.64357822s" totalImagesPullingTime="567.594425ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 10:13:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 10:13:05.64189416 +0000 UTC m=+140.190451271" watchObservedRunningTime="2026-09-17 10:13:05.64357822 +0000 UTC m=+140.192135331" Sep 17 10:13:05.663037 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:13:05.662983 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-798b5b8784-qz9zj" podStartSLOduration=7.220091481 podStartE2EDuration="11.662968796s" totalImagesPullingTime="4.442877315s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 10:12:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 10:13:05.661666889 +0000 UTC m=+140.210224001" watchObservedRunningTime="2026-09-17 10:13:05.662968796 +0000 UTC m=+140.211525906" Sep 17 10:13:13.053756 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:13:13.053723 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-74cfbd76cd-kfnrc" Sep 17 10:13:13.054207 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:13:13.053804 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-74cfbd76cd-kfnrc" Sep 17 10:13:13.058288 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:13:13.058262 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-74cfbd76cd-kfnrc" Sep 17 10:13:13.661397 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:13:13.661369 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-74cfbd76cd-kfnrc" Sep 17 10:13:13.708235 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:13:13.708207 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-798b5b8784-qz9zj"] Sep 17 10:13:14.413412 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:13:14.413373 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-798b5b8784-qz9zj" Sep 17 10:13:21.687176 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:13:21.687135 2581 generic.go:356] "Generic (PLEG): container finished" podID="da9abd04-2db6-44cc-b6bb-c5d2999ec8c4" containerID="89582ce03c7bf2530b70eb6a12035ec9057ee99d65658b4be38079f84d8e310c" exitCode=0 Sep 17 10:13:21.687585 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:13:21.687212 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-mjspg" event={"ID":"da9abd04-2db6-44cc-b6bb-c5d2999ec8c4","Type":"ContainerDied","Data":"89582ce03c7bf2530b70eb6a12035ec9057ee99d65658b4be38079f84d8e310c"} Sep 17 10:13:21.687585 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:13:21.687533 2581 scope.go:117] "RemoveContainer" containerID="89582ce03c7bf2530b70eb6a12035ec9057ee99d65658b4be38079f84d8e310c" Sep 17 10:13:22.692369 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:13:22.692293 2581 generic.go:356] "Generic (PLEG): container finished" podID="a13ce06b-1f1b-4614-a6f1-7e95dbd6f1d4" containerID="20f73970eac769a69eb7deb515b62c4a3c5ae6be2fa9d6421e76b577e0892988" exitCode=0 Sep 17 10:13:22.692751 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:13:22.692370 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-fdc72" event={"ID":"a13ce06b-1f1b-4614-a6f1-7e95dbd6f1d4","Type":"ContainerDied","Data":"20f73970eac769a69eb7deb515b62c4a3c5ae6be2fa9d6421e76b577e0892988"} Sep 17 10:13:22.692751 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:13:22.692730 2581 scope.go:117] "RemoveContainer" containerID="20f73970eac769a69eb7deb515b62c4a3c5ae6be2fa9d6421e76b577e0892988" Sep 17 10:13:22.694307 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:13:22.694286 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-mjspg" event={"ID":"da9abd04-2db6-44cc-b6bb-c5d2999ec8c4","Type":"ContainerStarted","Data":"1dd7f171928081a2670212e6eed952c046f4e697b9252076e062e1ed9669cac3"} Sep 17 10:13:23.700820 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:13:23.700769 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-fdc72" event={"ID":"a13ce06b-1f1b-4614-a6f1-7e95dbd6f1d4","Type":"ContainerStarted","Data":"55df77c67386630fb70a55f294edbe4494f540e80d66942c8c0bd71b47eb941e"} Sep 17 10:13:38.729454 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:13:38.729339 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-798b5b8784-qz9zj" podUID="437ef216-f348-40b6-9c84-29580f39ac68" containerName="console" containerID="cri-o://30678f278ae9635419b8ff46cd7fee5e1e5784f3793458c71ebf48e578775a24" gracePeriod=15 Sep 17 10:13:39.000550 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:13:39.000489 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-798b5b8784-qz9zj" Sep 17 10:13:39.020703 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:13:39.020673 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/437ef216-f348-40b6-9c84-29580f39ac68-oauth-serving-cert\") pod \"437ef216-f348-40b6-9c84-29580f39ac68\" (UID: \"437ef216-f348-40b6-9c84-29580f39ac68\") " Sep 17 10:13:39.020896 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:13:39.020737 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/437ef216-f348-40b6-9c84-29580f39ac68-console-serving-cert\") pod \"437ef216-f348-40b6-9c84-29580f39ac68\" (UID: \"437ef216-f348-40b6-9c84-29580f39ac68\") " Sep 17 10:13:39.020896 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:13:39.020758 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vwplr\" (UniqueName: \"kubernetes.io/projected/437ef216-f348-40b6-9c84-29580f39ac68-kube-api-access-vwplr\") pod \"437ef216-f348-40b6-9c84-29580f39ac68\" (UID: \"437ef216-f348-40b6-9c84-29580f39ac68\") " Sep 17 10:13:39.020896 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:13:39.020775 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/437ef216-f348-40b6-9c84-29580f39ac68-service-ca\") pod \"437ef216-f348-40b6-9c84-29580f39ac68\" (UID: \"437ef216-f348-40b6-9c84-29580f39ac68\") " Sep 17 10:13:39.020896 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:13:39.020806 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/437ef216-f348-40b6-9c84-29580f39ac68-console-oauth-config\") pod \"437ef216-f348-40b6-9c84-29580f39ac68\" (UID: \"437ef216-f348-40b6-9c84-29580f39ac68\") " Sep 17 10:13:39.020896 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:13:39.020832 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/437ef216-f348-40b6-9c84-29580f39ac68-console-config\") pod \"437ef216-f348-40b6-9c84-29580f39ac68\" (UID: \"437ef216-f348-40b6-9c84-29580f39ac68\") " Sep 17 10:13:39.021603 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:13:39.021324 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/437ef216-f348-40b6-9c84-29580f39ac68-service-ca" (OuterVolumeSpecName: "service-ca") pod "437ef216-f348-40b6-9c84-29580f39ac68" (UID: "437ef216-f348-40b6-9c84-29580f39ac68"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 10:13:39.021603 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:13:39.021344 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/437ef216-f348-40b6-9c84-29580f39ac68-console-config" (OuterVolumeSpecName: "console-config") pod "437ef216-f348-40b6-9c84-29580f39ac68" (UID: "437ef216-f348-40b6-9c84-29580f39ac68"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 10:13:39.021603 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:13:39.021500 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/437ef216-f348-40b6-9c84-29580f39ac68-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "437ef216-f348-40b6-9c84-29580f39ac68" (UID: "437ef216-f348-40b6-9c84-29580f39ac68"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 10:13:39.023414 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:13:39.023387 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/437ef216-f348-40b6-9c84-29580f39ac68-kube-api-access-vwplr" (OuterVolumeSpecName: "kube-api-access-vwplr") pod "437ef216-f348-40b6-9c84-29580f39ac68" (UID: "437ef216-f348-40b6-9c84-29580f39ac68"). InnerVolumeSpecName "kube-api-access-vwplr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 10:13:39.023571 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:13:39.023543 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/437ef216-f348-40b6-9c84-29580f39ac68-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "437ef216-f348-40b6-9c84-29580f39ac68" (UID: "437ef216-f348-40b6-9c84-29580f39ac68"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 10:13:39.023663 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:13:39.023587 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/437ef216-f348-40b6-9c84-29580f39ac68-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "437ef216-f348-40b6-9c84-29580f39ac68" (UID: "437ef216-f348-40b6-9c84-29580f39ac68"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 10:13:39.121419 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:13:39.121385 2581 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/437ef216-f348-40b6-9c84-29580f39ac68-oauth-serving-cert\") on node \"ip-10-0-137-203.ec2.internal\" DevicePath \"\"" Sep 17 10:13:39.121419 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:13:39.121413 2581 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/437ef216-f348-40b6-9c84-29580f39ac68-console-serving-cert\") on node \"ip-10-0-137-203.ec2.internal\" DevicePath \"\"" Sep 17 10:13:39.121419 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:13:39.121421 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwplr\" (UniqueName: \"kubernetes.io/projected/437ef216-f348-40b6-9c84-29580f39ac68-kube-api-access-vwplr\") on node \"ip-10-0-137-203.ec2.internal\" DevicePath \"\"" Sep 17 10:13:39.121419 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:13:39.121431 2581 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/437ef216-f348-40b6-9c84-29580f39ac68-service-ca\") on node \"ip-10-0-137-203.ec2.internal\" DevicePath \"\"" Sep 17 10:13:39.121676 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:13:39.121440 2581 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/437ef216-f348-40b6-9c84-29580f39ac68-console-oauth-config\") on node \"ip-10-0-137-203.ec2.internal\" DevicePath \"\"" Sep 17 10:13:39.121676 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:13:39.121448 2581 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/437ef216-f348-40b6-9c84-29580f39ac68-console-config\") on node \"ip-10-0-137-203.ec2.internal\" DevicePath \"\"" Sep 17 10:13:39.760638 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:13:39.760599 2581 generic.go:356] "Generic (PLEG): container finished" podID="437ef216-f348-40b6-9c84-29580f39ac68" containerID="30678f278ae9635419b8ff46cd7fee5e1e5784f3793458c71ebf48e578775a24" exitCode=0 Sep 17 10:13:39.761044 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:13:39.760645 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-798b5b8784-qz9zj" Sep 17 10:13:39.761044 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:13:39.760694 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-798b5b8784-qz9zj" event={"ID":"437ef216-f348-40b6-9c84-29580f39ac68","Type":"ContainerDied","Data":"30678f278ae9635419b8ff46cd7fee5e1e5784f3793458c71ebf48e578775a24"} Sep 17 10:13:39.761044 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:13:39.760737 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-798b5b8784-qz9zj" event={"ID":"437ef216-f348-40b6-9c84-29580f39ac68","Type":"ContainerDied","Data":"95f6aa5c982c55b65e27e54f99afd98b501bcb2d9a1adc3bc3e9323ee6b439e0"} Sep 17 10:13:39.761044 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:13:39.760758 2581 scope.go:117] "RemoveContainer" containerID="30678f278ae9635419b8ff46cd7fee5e1e5784f3793458c71ebf48e578775a24" Sep 17 10:13:39.769147 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:13:39.769130 2581 scope.go:117] "RemoveContainer" containerID="30678f278ae9635419b8ff46cd7fee5e1e5784f3793458c71ebf48e578775a24" Sep 17 10:13:39.769415 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:13:39.769398 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30678f278ae9635419b8ff46cd7fee5e1e5784f3793458c71ebf48e578775a24\": container with ID starting with 30678f278ae9635419b8ff46cd7fee5e1e5784f3793458c71ebf48e578775a24 not found: ID does not exist" containerID="30678f278ae9635419b8ff46cd7fee5e1e5784f3793458c71ebf48e578775a24" Sep 17 10:13:39.769464 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:13:39.769422 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30678f278ae9635419b8ff46cd7fee5e1e5784f3793458c71ebf48e578775a24"} err="failed to get container status \"30678f278ae9635419b8ff46cd7fee5e1e5784f3793458c71ebf48e578775a24\": rpc error: code = NotFound desc = could not find container \"30678f278ae9635419b8ff46cd7fee5e1e5784f3793458c71ebf48e578775a24\": container with ID starting with 30678f278ae9635419b8ff46cd7fee5e1e5784f3793458c71ebf48e578775a24 not found: ID does not exist" Sep 17 10:13:39.789010 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:13:39.788976 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-798b5b8784-qz9zj"] Sep 17 10:13:39.793800 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:13:39.793751 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-798b5b8784-qz9zj"] Sep 17 10:13:39.944251 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:13:39.944223 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="437ef216-f348-40b6-9c84-29580f39ac68" path="/var/lib/kubelet/pods/437ef216-f348-40b6-9c84-29580f39ac68/volumes" Sep 17 10:13:57.304896 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:13:57.304859 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6fd5d66c4-vltj4"] Sep 17 10:13:57.305295 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:13:57.305157 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="437ef216-f348-40b6-9c84-29580f39ac68" containerName="console" Sep 17 10:13:57.305295 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:13:57.305169 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="437ef216-f348-40b6-9c84-29580f39ac68" containerName="console" Sep 17 10:13:57.305295 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:13:57.305228 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="437ef216-f348-40b6-9c84-29580f39ac68" containerName="console" Sep 17 10:13:57.307927 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:13:57.307903 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6fd5d66c4-vltj4" Sep 17 10:13:57.322576 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:13:57.322550 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6fd5d66c4-vltj4"] Sep 17 10:13:57.354652 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:13:57.354625 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/edd08497-99e0-4c46-b90e-be0bce94e70c-oauth-serving-cert\") pod \"console-6fd5d66c4-vltj4\" (UID: \"edd08497-99e0-4c46-b90e-be0bce94e70c\") " pod="openshift-console/console-6fd5d66c4-vltj4" Sep 17 10:13:57.354850 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:13:57.354659 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/edd08497-99e0-4c46-b90e-be0bce94e70c-console-serving-cert\") pod \"console-6fd5d66c4-vltj4\" (UID: \"edd08497-99e0-4c46-b90e-be0bce94e70c\") " pod="openshift-console/console-6fd5d66c4-vltj4" Sep 17 10:13:57.354850 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:13:57.354679 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/edd08497-99e0-4c46-b90e-be0bce94e70c-console-config\") pod \"console-6fd5d66c4-vltj4\" (UID: \"edd08497-99e0-4c46-b90e-be0bce94e70c\") " pod="openshift-console/console-6fd5d66c4-vltj4" Sep 17 10:13:57.354850 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:13:57.354703 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjkk5\" (UniqueName: \"kubernetes.io/projected/edd08497-99e0-4c46-b90e-be0bce94e70c-kube-api-access-hjkk5\") pod \"console-6fd5d66c4-vltj4\" (UID: \"edd08497-99e0-4c46-b90e-be0bce94e70c\") " pod="openshift-console/console-6fd5d66c4-vltj4" Sep 17 10:13:57.354850 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:13:57.354763 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/edd08497-99e0-4c46-b90e-be0bce94e70c-service-ca\") pod \"console-6fd5d66c4-vltj4\" (UID: \"edd08497-99e0-4c46-b90e-be0bce94e70c\") " pod="openshift-console/console-6fd5d66c4-vltj4" Sep 17 10:13:57.354850 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:13:57.354831 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/edd08497-99e0-4c46-b90e-be0bce94e70c-trusted-ca-bundle\") pod \"console-6fd5d66c4-vltj4\" (UID: \"edd08497-99e0-4c46-b90e-be0bce94e70c\") " pod="openshift-console/console-6fd5d66c4-vltj4" Sep 17 10:13:57.355041 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:13:57.354867 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/edd08497-99e0-4c46-b90e-be0bce94e70c-console-oauth-config\") pod \"console-6fd5d66c4-vltj4\" (UID: \"edd08497-99e0-4c46-b90e-be0bce94e70c\") " pod="openshift-console/console-6fd5d66c4-vltj4" Sep 17 10:13:57.455362 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:13:57.455326 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/edd08497-99e0-4c46-b90e-be0bce94e70c-trusted-ca-bundle\") pod \"console-6fd5d66c4-vltj4\" (UID: \"edd08497-99e0-4c46-b90e-be0bce94e70c\") " pod="openshift-console/console-6fd5d66c4-vltj4" Sep 17 10:13:57.455532 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:13:57.455370 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/edd08497-99e0-4c46-b90e-be0bce94e70c-console-oauth-config\") pod \"console-6fd5d66c4-vltj4\" (UID: \"edd08497-99e0-4c46-b90e-be0bce94e70c\") " pod="openshift-console/console-6fd5d66c4-vltj4" Sep 17 10:13:57.455532 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:13:57.455397 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/edd08497-99e0-4c46-b90e-be0bce94e70c-oauth-serving-cert\") pod \"console-6fd5d66c4-vltj4\" (UID: \"edd08497-99e0-4c46-b90e-be0bce94e70c\") " pod="openshift-console/console-6fd5d66c4-vltj4" Sep 17 10:13:57.455532 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:13:57.455428 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/edd08497-99e0-4c46-b90e-be0bce94e70c-console-serving-cert\") pod \"console-6fd5d66c4-vltj4\" (UID: \"edd08497-99e0-4c46-b90e-be0bce94e70c\") " pod="openshift-console/console-6fd5d66c4-vltj4" Sep 17 10:13:57.455532 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:13:57.455455 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/edd08497-99e0-4c46-b90e-be0bce94e70c-console-config\") pod \"console-6fd5d66c4-vltj4\" (UID: \"edd08497-99e0-4c46-b90e-be0bce94e70c\") " pod="openshift-console/console-6fd5d66c4-vltj4" Sep 17 10:13:57.455532 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:13:57.455481 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjkk5\" (UniqueName: \"kubernetes.io/projected/edd08497-99e0-4c46-b90e-be0bce94e70c-kube-api-access-hjkk5\") pod \"console-6fd5d66c4-vltj4\" (UID: \"edd08497-99e0-4c46-b90e-be0bce94e70c\") " pod="openshift-console/console-6fd5d66c4-vltj4" Sep 17 10:13:57.455532 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:13:57.455517 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/edd08497-99e0-4c46-b90e-be0bce94e70c-service-ca\") pod \"console-6fd5d66c4-vltj4\" (UID: \"edd08497-99e0-4c46-b90e-be0bce94e70c\") " pod="openshift-console/console-6fd5d66c4-vltj4" Sep 17 10:13:57.456189 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:13:57.456166 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/edd08497-99e0-4c46-b90e-be0bce94e70c-service-ca\") pod \"console-6fd5d66c4-vltj4\" (UID: \"edd08497-99e0-4c46-b90e-be0bce94e70c\") " pod="openshift-console/console-6fd5d66c4-vltj4" Sep 17 10:13:57.456303 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:13:57.456278 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/edd08497-99e0-4c46-b90e-be0bce94e70c-oauth-serving-cert\") pod \"console-6fd5d66c4-vltj4\" (UID: \"edd08497-99e0-4c46-b90e-be0bce94e70c\") " pod="openshift-console/console-6fd5d66c4-vltj4" Sep 17 10:13:57.456349 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:13:57.456306 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/edd08497-99e0-4c46-b90e-be0bce94e70c-trusted-ca-bundle\") pod \"console-6fd5d66c4-vltj4\" (UID: \"edd08497-99e0-4c46-b90e-be0bce94e70c\") " pod="openshift-console/console-6fd5d66c4-vltj4" Sep 17 10:13:57.456349 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:13:57.456317 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/edd08497-99e0-4c46-b90e-be0bce94e70c-console-config\") pod \"console-6fd5d66c4-vltj4\" (UID: \"edd08497-99e0-4c46-b90e-be0bce94e70c\") " pod="openshift-console/console-6fd5d66c4-vltj4" Sep 17 10:13:57.457895 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:13:57.457864 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/edd08497-99e0-4c46-b90e-be0bce94e70c-console-serving-cert\") pod \"console-6fd5d66c4-vltj4\" (UID: \"edd08497-99e0-4c46-b90e-be0bce94e70c\") " pod="openshift-console/console-6fd5d66c4-vltj4" Sep 17 10:13:57.457999 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:13:57.457917 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/edd08497-99e0-4c46-b90e-be0bce94e70c-console-oauth-config\") pod \"console-6fd5d66c4-vltj4\" (UID: \"edd08497-99e0-4c46-b90e-be0bce94e70c\") " pod="openshift-console/console-6fd5d66c4-vltj4" Sep 17 10:13:57.468385 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:13:57.468363 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjkk5\" (UniqueName: \"kubernetes.io/projected/edd08497-99e0-4c46-b90e-be0bce94e70c-kube-api-access-hjkk5\") pod \"console-6fd5d66c4-vltj4\" (UID: \"edd08497-99e0-4c46-b90e-be0bce94e70c\") " pod="openshift-console/console-6fd5d66c4-vltj4" Sep 17 10:13:57.618581 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:13:57.618495 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6fd5d66c4-vltj4" Sep 17 10:13:57.894292 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:13:57.894266 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6fd5d66c4-vltj4"] Sep 17 10:13:57.896121 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:13:57.896095 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedd08497_99e0_4c46_b90e_be0bce94e70c.slice/crio-7c19a69d4fc6575beed98b7c4801ec8b3fd82b55dfa49b166177d5fdf4f35fdc WatchSource:0}: Error finding container 7c19a69d4fc6575beed98b7c4801ec8b3fd82b55dfa49b166177d5fdf4f35fdc: Status 404 returned error can't find the container with id 7c19a69d4fc6575beed98b7c4801ec8b3fd82b55dfa49b166177d5fdf4f35fdc Sep 17 10:13:58.835052 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:13:58.835017 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6fd5d66c4-vltj4" event={"ID":"edd08497-99e0-4c46-b90e-be0bce94e70c","Type":"ContainerStarted","Data":"cd7958d472899db433c3ddfb710700d10d7bbc5530836e95cc42a357b708be56"} Sep 17 10:13:58.835453 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:13:58.835070 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6fd5d66c4-vltj4" event={"ID":"edd08497-99e0-4c46-b90e-be0bce94e70c","Type":"ContainerStarted","Data":"7c19a69d4fc6575beed98b7c4801ec8b3fd82b55dfa49b166177d5fdf4f35fdc"} Sep 17 10:13:58.857280 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:13:58.857231 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6fd5d66c4-vltj4" podStartSLOduration=1.8572184919999999 podStartE2EDuration="1.857218492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 10:13:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 10:13:58.85677599 +0000 UTC m=+193.405333102" watchObservedRunningTime="2026-09-17 10:13:58.857218492 +0000 UTC m=+193.405775603" Sep 17 10:14:07.619650 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:14:07.619610 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6fd5d66c4-vltj4" Sep 17 10:14:07.620043 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:14:07.619685 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6fd5d66c4-vltj4" Sep 17 10:14:07.624366 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:14:07.624345 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6fd5d66c4-vltj4" Sep 17 10:14:07.878801 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:14:07.878714 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6fd5d66c4-vltj4" Sep 17 10:14:07.930099 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:14:07.930062 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-74cfbd76cd-kfnrc"] Sep 17 10:14:32.953828 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:14:32.953757 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-74cfbd76cd-kfnrc" podUID="724f7002-97bb-4bd5-869c-6676b87d5a8e" containerName="console" containerID="cri-o://9414524e8e3d323d3cd5d5ffc05f526d8d3af0aa467e4d7c62268ae99f112946" gracePeriod=15 Sep 17 10:14:33.189906 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:14:33.189883 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74cfbd76cd-kfnrc" Sep 17 10:14:33.335648 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:14:33.335609 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/724f7002-97bb-4bd5-869c-6676b87d5a8e-oauth-serving-cert\") pod \"724f7002-97bb-4bd5-869c-6676b87d5a8e\" (UID: \"724f7002-97bb-4bd5-869c-6676b87d5a8e\") " Sep 17 10:14:33.335648 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:14:33.335648 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/724f7002-97bb-4bd5-869c-6676b87d5a8e-trusted-ca-bundle\") pod \"724f7002-97bb-4bd5-869c-6676b87d5a8e\" (UID: \"724f7002-97bb-4bd5-869c-6676b87d5a8e\") " Sep 17 10:14:33.335924 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:14:33.335691 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/724f7002-97bb-4bd5-869c-6676b87d5a8e-console-serving-cert\") pod \"724f7002-97bb-4bd5-869c-6676b87d5a8e\" (UID: \"724f7002-97bb-4bd5-869c-6676b87d5a8e\") " Sep 17 10:14:33.335924 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:14:33.335719 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f65t7\" (UniqueName: \"kubernetes.io/projected/724f7002-97bb-4bd5-869c-6676b87d5a8e-kube-api-access-f65t7\") pod \"724f7002-97bb-4bd5-869c-6676b87d5a8e\" (UID: \"724f7002-97bb-4bd5-869c-6676b87d5a8e\") " Sep 17 10:14:33.335924 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:14:33.335775 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/724f7002-97bb-4bd5-869c-6676b87d5a8e-console-oauth-config\") pod \"724f7002-97bb-4bd5-869c-6676b87d5a8e\" (UID: \"724f7002-97bb-4bd5-869c-6676b87d5a8e\") " Sep 17 10:14:33.335924 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:14:33.335827 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/724f7002-97bb-4bd5-869c-6676b87d5a8e-console-config\") pod \"724f7002-97bb-4bd5-869c-6676b87d5a8e\" (UID: \"724f7002-97bb-4bd5-869c-6676b87d5a8e\") " Sep 17 10:14:33.335924 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:14:33.335862 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/724f7002-97bb-4bd5-869c-6676b87d5a8e-service-ca\") pod \"724f7002-97bb-4bd5-869c-6676b87d5a8e\" (UID: \"724f7002-97bb-4bd5-869c-6676b87d5a8e\") " Sep 17 10:14:33.336146 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:14:33.336075 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/724f7002-97bb-4bd5-869c-6676b87d5a8e-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "724f7002-97bb-4bd5-869c-6676b87d5a8e" (UID: "724f7002-97bb-4bd5-869c-6676b87d5a8e"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 10:14:33.336280 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:14:33.336257 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/724f7002-97bb-4bd5-869c-6676b87d5a8e-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "724f7002-97bb-4bd5-869c-6676b87d5a8e" (UID: "724f7002-97bb-4bd5-869c-6676b87d5a8e"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 10:14:33.336349 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:14:33.336317 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/724f7002-97bb-4bd5-869c-6676b87d5a8e-service-ca" (OuterVolumeSpecName: "service-ca") pod "724f7002-97bb-4bd5-869c-6676b87d5a8e" (UID: "724f7002-97bb-4bd5-869c-6676b87d5a8e"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 10:14:33.336406 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:14:33.336345 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/724f7002-97bb-4bd5-869c-6676b87d5a8e-console-config" (OuterVolumeSpecName: "console-config") pod "724f7002-97bb-4bd5-869c-6676b87d5a8e" (UID: "724f7002-97bb-4bd5-869c-6676b87d5a8e"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 10:14:33.338037 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:14:33.338010 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/724f7002-97bb-4bd5-869c-6676b87d5a8e-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "724f7002-97bb-4bd5-869c-6676b87d5a8e" (UID: "724f7002-97bb-4bd5-869c-6676b87d5a8e"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 10:14:33.338132 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:14:33.338017 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/724f7002-97bb-4bd5-869c-6676b87d5a8e-kube-api-access-f65t7" (OuterVolumeSpecName: "kube-api-access-f65t7") pod "724f7002-97bb-4bd5-869c-6676b87d5a8e" (UID: "724f7002-97bb-4bd5-869c-6676b87d5a8e"). InnerVolumeSpecName "kube-api-access-f65t7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 10:14:33.338132 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:14:33.338041 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/724f7002-97bb-4bd5-869c-6676b87d5a8e-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "724f7002-97bb-4bd5-869c-6676b87d5a8e" (UID: "724f7002-97bb-4bd5-869c-6676b87d5a8e"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 10:14:33.437059 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:14:33.437033 2581 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/724f7002-97bb-4bd5-869c-6676b87d5a8e-console-oauth-config\") on node \"ip-10-0-137-203.ec2.internal\" DevicePath \"\"" Sep 17 10:14:33.437059 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:14:33.437056 2581 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/724f7002-97bb-4bd5-869c-6676b87d5a8e-console-config\") on node \"ip-10-0-137-203.ec2.internal\" DevicePath \"\"" Sep 17 10:14:33.437059 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:14:33.437065 2581 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/724f7002-97bb-4bd5-869c-6676b87d5a8e-service-ca\") on node \"ip-10-0-137-203.ec2.internal\" DevicePath \"\"" Sep 17 10:14:33.437240 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:14:33.437074 2581 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/724f7002-97bb-4bd5-869c-6676b87d5a8e-oauth-serving-cert\") on node \"ip-10-0-137-203.ec2.internal\" DevicePath \"\"" Sep 17 10:14:33.437240 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:14:33.437082 2581 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/724f7002-97bb-4bd5-869c-6676b87d5a8e-trusted-ca-bundle\") on node \"ip-10-0-137-203.ec2.internal\" DevicePath \"\"" Sep 17 10:14:33.437240 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:14:33.437089 2581 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/724f7002-97bb-4bd5-869c-6676b87d5a8e-console-serving-cert\") on node \"ip-10-0-137-203.ec2.internal\" DevicePath \"\"" Sep 17 10:14:33.437240 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:14:33.437097 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f65t7\" (UniqueName: \"kubernetes.io/projected/724f7002-97bb-4bd5-869c-6676b87d5a8e-kube-api-access-f65t7\") on node \"ip-10-0-137-203.ec2.internal\" DevicePath \"\"" Sep 17 10:14:33.971487 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:14:33.971454 2581 generic.go:356] "Generic (PLEG): container finished" podID="724f7002-97bb-4bd5-869c-6676b87d5a8e" containerID="9414524e8e3d323d3cd5d5ffc05f526d8d3af0aa467e4d7c62268ae99f112946" exitCode=0 Sep 17 10:14:33.971895 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:14:33.971501 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74cfbd76cd-kfnrc" Sep 17 10:14:33.971895 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:14:33.971549 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74cfbd76cd-kfnrc" event={"ID":"724f7002-97bb-4bd5-869c-6676b87d5a8e","Type":"ContainerDied","Data":"9414524e8e3d323d3cd5d5ffc05f526d8d3af0aa467e4d7c62268ae99f112946"} Sep 17 10:14:33.971895 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:14:33.971610 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74cfbd76cd-kfnrc" event={"ID":"724f7002-97bb-4bd5-869c-6676b87d5a8e","Type":"ContainerDied","Data":"34d46fee99720f2126127c28760ef19c601670886f4b770b7379d6559b5c3d36"} Sep 17 10:14:33.971895 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:14:33.971640 2581 scope.go:117] "RemoveContainer" containerID="9414524e8e3d323d3cd5d5ffc05f526d8d3af0aa467e4d7c62268ae99f112946" Sep 17 10:14:33.979327 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:14:33.979311 2581 scope.go:117] "RemoveContainer" containerID="9414524e8e3d323d3cd5d5ffc05f526d8d3af0aa467e4d7c62268ae99f112946" Sep 17 10:14:33.979584 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:14:33.979563 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9414524e8e3d323d3cd5d5ffc05f526d8d3af0aa467e4d7c62268ae99f112946\": container with ID starting with 9414524e8e3d323d3cd5d5ffc05f526d8d3af0aa467e4d7c62268ae99f112946 not found: ID does not exist" containerID="9414524e8e3d323d3cd5d5ffc05f526d8d3af0aa467e4d7c62268ae99f112946" Sep 17 10:14:33.979675 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:14:33.979588 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9414524e8e3d323d3cd5d5ffc05f526d8d3af0aa467e4d7c62268ae99f112946"} err="failed to get container status \"9414524e8e3d323d3cd5d5ffc05f526d8d3af0aa467e4d7c62268ae99f112946\": rpc error: code = NotFound desc = could not find container \"9414524e8e3d323d3cd5d5ffc05f526d8d3af0aa467e4d7c62268ae99f112946\": container with ID starting with 9414524e8e3d323d3cd5d5ffc05f526d8d3af0aa467e4d7c62268ae99f112946 not found: ID does not exist" Sep 17 10:14:33.991205 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:14:33.991174 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-74cfbd76cd-kfnrc"] Sep 17 10:14:33.995325 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:14:33.995306 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-74cfbd76cd-kfnrc"] Sep 17 10:14:35.942950 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:14:35.942918 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="724f7002-97bb-4bd5-869c-6676b87d5a8e" path="/var/lib/kubelet/pods/724f7002-97bb-4bd5-869c-6676b87d5a8e/volumes" Sep 17 10:15:28.544991 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:15:28.544962 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5cc7c64bbf-dmfkl"] Sep 17 10:15:28.545386 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:15:28.545247 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="724f7002-97bb-4bd5-869c-6676b87d5a8e" containerName="console" Sep 17 10:15:28.545386 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:15:28.545257 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="724f7002-97bb-4bd5-869c-6676b87d5a8e" containerName="console" Sep 17 10:15:28.545386 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:15:28.545305 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="724f7002-97bb-4bd5-869c-6676b87d5a8e" containerName="console" Sep 17 10:15:28.547157 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:15:28.547133 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5cc7c64bbf-dmfkl" Sep 17 10:15:28.561318 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:15:28.561287 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5cc7c64bbf-dmfkl"] Sep 17 10:15:28.636365 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:15:28.636328 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2mf2\" (UniqueName: \"kubernetes.io/projected/f14a6868-bbda-4e64-b336-e3dbe3b47906-kube-api-access-g2mf2\") pod \"console-5cc7c64bbf-dmfkl\" (UID: \"f14a6868-bbda-4e64-b336-e3dbe3b47906\") " pod="openshift-console/console-5cc7c64bbf-dmfkl" Sep 17 10:15:28.636365 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:15:28.636369 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f14a6868-bbda-4e64-b336-e3dbe3b47906-console-oauth-config\") pod \"console-5cc7c64bbf-dmfkl\" (UID: \"f14a6868-bbda-4e64-b336-e3dbe3b47906\") " pod="openshift-console/console-5cc7c64bbf-dmfkl" Sep 17 10:15:28.636587 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:15:28.636391 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f14a6868-bbda-4e64-b336-e3dbe3b47906-service-ca\") pod \"console-5cc7c64bbf-dmfkl\" (UID: \"f14a6868-bbda-4e64-b336-e3dbe3b47906\") " pod="openshift-console/console-5cc7c64bbf-dmfkl" Sep 17 10:15:28.636587 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:15:28.636414 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f14a6868-bbda-4e64-b336-e3dbe3b47906-console-config\") pod \"console-5cc7c64bbf-dmfkl\" (UID: \"f14a6868-bbda-4e64-b336-e3dbe3b47906\") " pod="openshift-console/console-5cc7c64bbf-dmfkl" Sep 17 10:15:28.636587 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:15:28.636455 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f14a6868-bbda-4e64-b336-e3dbe3b47906-console-serving-cert\") pod \"console-5cc7c64bbf-dmfkl\" (UID: \"f14a6868-bbda-4e64-b336-e3dbe3b47906\") " pod="openshift-console/console-5cc7c64bbf-dmfkl" Sep 17 10:15:28.636587 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:15:28.636481 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f14a6868-bbda-4e64-b336-e3dbe3b47906-trusted-ca-bundle\") pod \"console-5cc7c64bbf-dmfkl\" (UID: \"f14a6868-bbda-4e64-b336-e3dbe3b47906\") " pod="openshift-console/console-5cc7c64bbf-dmfkl" Sep 17 10:15:28.636587 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:15:28.636497 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f14a6868-bbda-4e64-b336-e3dbe3b47906-oauth-serving-cert\") pod \"console-5cc7c64bbf-dmfkl\" (UID: \"f14a6868-bbda-4e64-b336-e3dbe3b47906\") " pod="openshift-console/console-5cc7c64bbf-dmfkl" Sep 17 10:15:28.737636 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:15:28.737599 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f14a6868-bbda-4e64-b336-e3dbe3b47906-console-oauth-config\") pod \"console-5cc7c64bbf-dmfkl\" (UID: \"f14a6868-bbda-4e64-b336-e3dbe3b47906\") " pod="openshift-console/console-5cc7c64bbf-dmfkl" Sep 17 10:15:28.737846 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:15:28.737644 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f14a6868-bbda-4e64-b336-e3dbe3b47906-service-ca\") pod \"console-5cc7c64bbf-dmfkl\" (UID: \"f14a6868-bbda-4e64-b336-e3dbe3b47906\") " pod="openshift-console/console-5cc7c64bbf-dmfkl" Sep 17 10:15:28.737846 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:15:28.737747 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f14a6868-bbda-4e64-b336-e3dbe3b47906-console-config\") pod \"console-5cc7c64bbf-dmfkl\" (UID: \"f14a6868-bbda-4e64-b336-e3dbe3b47906\") " pod="openshift-console/console-5cc7c64bbf-dmfkl" Sep 17 10:15:28.737966 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:15:28.737923 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f14a6868-bbda-4e64-b336-e3dbe3b47906-console-serving-cert\") pod \"console-5cc7c64bbf-dmfkl\" (UID: \"f14a6868-bbda-4e64-b336-e3dbe3b47906\") " pod="openshift-console/console-5cc7c64bbf-dmfkl" Sep 17 10:15:28.738022 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:15:28.737968 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f14a6868-bbda-4e64-b336-e3dbe3b47906-trusted-ca-bundle\") pod \"console-5cc7c64bbf-dmfkl\" (UID: \"f14a6868-bbda-4e64-b336-e3dbe3b47906\") " pod="openshift-console/console-5cc7c64bbf-dmfkl" Sep 17 10:15:28.738022 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:15:28.737997 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f14a6868-bbda-4e64-b336-e3dbe3b47906-oauth-serving-cert\") pod \"console-5cc7c64bbf-dmfkl\" (UID: \"f14a6868-bbda-4e64-b336-e3dbe3b47906\") " pod="openshift-console/console-5cc7c64bbf-dmfkl" Sep 17 10:15:28.738119 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:15:28.738062 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2mf2\" (UniqueName: \"kubernetes.io/projected/f14a6868-bbda-4e64-b336-e3dbe3b47906-kube-api-access-g2mf2\") pod \"console-5cc7c64bbf-dmfkl\" (UID: \"f14a6868-bbda-4e64-b336-e3dbe3b47906\") " pod="openshift-console/console-5cc7c64bbf-dmfkl" Sep 17 10:15:28.738548 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:15:28.738516 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f14a6868-bbda-4e64-b336-e3dbe3b47906-console-config\") pod \"console-5cc7c64bbf-dmfkl\" (UID: \"f14a6868-bbda-4e64-b336-e3dbe3b47906\") " pod="openshift-console/console-5cc7c64bbf-dmfkl" Sep 17 10:15:28.738712 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:15:28.738695 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f14a6868-bbda-4e64-b336-e3dbe3b47906-service-ca\") pod \"console-5cc7c64bbf-dmfkl\" (UID: \"f14a6868-bbda-4e64-b336-e3dbe3b47906\") " pod="openshift-console/console-5cc7c64bbf-dmfkl" Sep 17 10:15:28.738838 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:15:28.738813 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f14a6868-bbda-4e64-b336-e3dbe3b47906-oauth-serving-cert\") pod \"console-5cc7c64bbf-dmfkl\" (UID: \"f14a6868-bbda-4e64-b336-e3dbe3b47906\") " pod="openshift-console/console-5cc7c64bbf-dmfkl" Sep 17 10:15:28.738912 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:15:28.738867 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f14a6868-bbda-4e64-b336-e3dbe3b47906-trusted-ca-bundle\") pod \"console-5cc7c64bbf-dmfkl\" (UID: \"f14a6868-bbda-4e64-b336-e3dbe3b47906\") " pod="openshift-console/console-5cc7c64bbf-dmfkl" Sep 17 10:15:28.740336 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:15:28.740311 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f14a6868-bbda-4e64-b336-e3dbe3b47906-console-oauth-config\") pod \"console-5cc7c64bbf-dmfkl\" (UID: \"f14a6868-bbda-4e64-b336-e3dbe3b47906\") " pod="openshift-console/console-5cc7c64bbf-dmfkl" Sep 17 10:15:28.740421 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:15:28.740315 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f14a6868-bbda-4e64-b336-e3dbe3b47906-console-serving-cert\") pod \"console-5cc7c64bbf-dmfkl\" (UID: \"f14a6868-bbda-4e64-b336-e3dbe3b47906\") " pod="openshift-console/console-5cc7c64bbf-dmfkl" Sep 17 10:15:28.747290 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:15:28.747265 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2mf2\" (UniqueName: \"kubernetes.io/projected/f14a6868-bbda-4e64-b336-e3dbe3b47906-kube-api-access-g2mf2\") pod \"console-5cc7c64bbf-dmfkl\" (UID: \"f14a6868-bbda-4e64-b336-e3dbe3b47906\") " pod="openshift-console/console-5cc7c64bbf-dmfkl" Sep 17 10:15:28.857902 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:15:28.857797 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5cc7c64bbf-dmfkl" Sep 17 10:15:29.135062 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:15:29.134984 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5cc7c64bbf-dmfkl"] Sep 17 10:15:29.137898 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:15:29.137867 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf14a6868_bbda_4e64_b336_e3dbe3b47906.slice/crio-90a4953ef45531ca390ac9dc5cc0b983eae66850052812ce1be3ec375f1a6e34 WatchSource:0}: Error finding container 90a4953ef45531ca390ac9dc5cc0b983eae66850052812ce1be3ec375f1a6e34: Status 404 returned error can't find the container with id 90a4953ef45531ca390ac9dc5cc0b983eae66850052812ce1be3ec375f1a6e34 Sep 17 10:15:29.173292 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:15:29.173267 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5cc7c64bbf-dmfkl" event={"ID":"f14a6868-bbda-4e64-b336-e3dbe3b47906","Type":"ContainerStarted","Data":"90a4953ef45531ca390ac9dc5cc0b983eae66850052812ce1be3ec375f1a6e34"} Sep 17 10:15:30.178771 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:15:30.178737 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5cc7c64bbf-dmfkl" event={"ID":"f14a6868-bbda-4e64-b336-e3dbe3b47906","Type":"ContainerStarted","Data":"2c0246b431c977f0218ba271ad5694917f1b749b47470527f7eb480f8f0fd52b"} Sep 17 10:15:30.201119 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:15:30.201069 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5cc7c64bbf-dmfkl" podStartSLOduration=2.201053887 podStartE2EDuration="2.201053887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 10:15:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 10:15:30.200183474 +0000 UTC m=+284.748740595" watchObservedRunningTime="2026-09-17 10:15:30.201053887 +0000 UTC m=+284.749610997" Sep 17 10:15:38.858882 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:15:38.858847 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5cc7c64bbf-dmfkl" Sep 17 10:15:38.859287 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:15:38.858943 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5cc7c64bbf-dmfkl" Sep 17 10:15:38.863263 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:15:38.863245 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5cc7c64bbf-dmfkl" Sep 17 10:15:39.216891 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:15:39.216814 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5cc7c64bbf-dmfkl" Sep 17 10:15:39.266127 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:15:39.266090 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6fd5d66c4-vltj4"] Sep 17 10:15:45.822800 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:15:45.822757 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-b9wnf_762d3e10-7b7d-4ce9-9773-90a4fe41b474/console-operator/2.log" Sep 17 10:15:45.827569 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:15:45.827543 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-b9wnf_762d3e10-7b7d-4ce9-9773-90a4fe41b474/console-operator/2.log" Sep 17 10:15:45.833125 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:15:45.833092 2581 kubelet.go:1648] "Image garbage collection succeeded" Sep 17 10:16:04.291167 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:16:04.291124 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6fd5d66c4-vltj4" podUID="edd08497-99e0-4c46-b90e-be0bce94e70c" containerName="console" containerID="cri-o://cd7958d472899db433c3ddfb710700d10d7bbc5530836e95cc42a357b708be56" gracePeriod=15 Sep 17 10:16:04.526273 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:16:04.526251 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6fd5d66c4-vltj4" Sep 17 10:16:04.615177 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:16:04.615084 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/edd08497-99e0-4c46-b90e-be0bce94e70c-console-config\") pod \"edd08497-99e0-4c46-b90e-be0bce94e70c\" (UID: \"edd08497-99e0-4c46-b90e-be0bce94e70c\") " Sep 17 10:16:04.615177 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:16:04.615153 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/edd08497-99e0-4c46-b90e-be0bce94e70c-console-serving-cert\") pod \"edd08497-99e0-4c46-b90e-be0bce94e70c\" (UID: \"edd08497-99e0-4c46-b90e-be0bce94e70c\") " Sep 17 10:16:04.615177 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:16:04.615176 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/edd08497-99e0-4c46-b90e-be0bce94e70c-console-oauth-config\") pod \"edd08497-99e0-4c46-b90e-be0bce94e70c\" (UID: \"edd08497-99e0-4c46-b90e-be0bce94e70c\") " Sep 17 10:16:04.615458 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:16:04.615217 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/edd08497-99e0-4c46-b90e-be0bce94e70c-trusted-ca-bundle\") pod \"edd08497-99e0-4c46-b90e-be0bce94e70c\" (UID: \"edd08497-99e0-4c46-b90e-be0bce94e70c\") " Sep 17 10:16:04.615458 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:16:04.615237 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/edd08497-99e0-4c46-b90e-be0bce94e70c-oauth-serving-cert\") pod \"edd08497-99e0-4c46-b90e-be0bce94e70c\" (UID: \"edd08497-99e0-4c46-b90e-be0bce94e70c\") " Sep 17 10:16:04.615458 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:16:04.615274 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/edd08497-99e0-4c46-b90e-be0bce94e70c-service-ca\") pod \"edd08497-99e0-4c46-b90e-be0bce94e70c\" (UID: \"edd08497-99e0-4c46-b90e-be0bce94e70c\") " Sep 17 10:16:04.615458 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:16:04.615300 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hjkk5\" (UniqueName: \"kubernetes.io/projected/edd08497-99e0-4c46-b90e-be0bce94e70c-kube-api-access-hjkk5\") pod \"edd08497-99e0-4c46-b90e-be0bce94e70c\" (UID: \"edd08497-99e0-4c46-b90e-be0bce94e70c\") " Sep 17 10:16:04.615654 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:16:04.615571 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edd08497-99e0-4c46-b90e-be0bce94e70c-console-config" (OuterVolumeSpecName: "console-config") pod "edd08497-99e0-4c46-b90e-be0bce94e70c" (UID: "edd08497-99e0-4c46-b90e-be0bce94e70c"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 10:16:04.615726 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:16:04.615697 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edd08497-99e0-4c46-b90e-be0bce94e70c-service-ca" (OuterVolumeSpecName: "service-ca") pod "edd08497-99e0-4c46-b90e-be0bce94e70c" (UID: "edd08497-99e0-4c46-b90e-be0bce94e70c"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 10:16:04.615726 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:16:04.615711 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edd08497-99e0-4c46-b90e-be0bce94e70c-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "edd08497-99e0-4c46-b90e-be0bce94e70c" (UID: "edd08497-99e0-4c46-b90e-be0bce94e70c"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 10:16:04.615856 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:16:04.615704 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edd08497-99e0-4c46-b90e-be0bce94e70c-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "edd08497-99e0-4c46-b90e-be0bce94e70c" (UID: "edd08497-99e0-4c46-b90e-be0bce94e70c"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 10:16:04.617386 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:16:04.617360 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/edd08497-99e0-4c46-b90e-be0bce94e70c-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "edd08497-99e0-4c46-b90e-be0bce94e70c" (UID: "edd08497-99e0-4c46-b90e-be0bce94e70c"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 10:16:04.617543 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:16:04.617527 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edd08497-99e0-4c46-b90e-be0bce94e70c-kube-api-access-hjkk5" (OuterVolumeSpecName: "kube-api-access-hjkk5") pod "edd08497-99e0-4c46-b90e-be0bce94e70c" (UID: "edd08497-99e0-4c46-b90e-be0bce94e70c"). InnerVolumeSpecName "kube-api-access-hjkk5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 10:16:04.617634 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:16:04.617612 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/edd08497-99e0-4c46-b90e-be0bce94e70c-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "edd08497-99e0-4c46-b90e-be0bce94e70c" (UID: "edd08497-99e0-4c46-b90e-be0bce94e70c"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 10:16:04.716188 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:16:04.716146 2581 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/edd08497-99e0-4c46-b90e-be0bce94e70c-console-serving-cert\") on node \"ip-10-0-137-203.ec2.internal\" DevicePath \"\"" Sep 17 10:16:04.716188 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:16:04.716182 2581 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/edd08497-99e0-4c46-b90e-be0bce94e70c-console-oauth-config\") on node \"ip-10-0-137-203.ec2.internal\" DevicePath \"\"" Sep 17 10:16:04.716188 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:16:04.716195 2581 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/edd08497-99e0-4c46-b90e-be0bce94e70c-trusted-ca-bundle\") on node \"ip-10-0-137-203.ec2.internal\" DevicePath \"\"" Sep 17 10:16:04.716420 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:16:04.716206 2581 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/edd08497-99e0-4c46-b90e-be0bce94e70c-oauth-serving-cert\") on node \"ip-10-0-137-203.ec2.internal\" DevicePath \"\"" Sep 17 10:16:04.716420 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:16:04.716219 2581 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/edd08497-99e0-4c46-b90e-be0bce94e70c-service-ca\") on node \"ip-10-0-137-203.ec2.internal\" DevicePath \"\"" Sep 17 10:16:04.716420 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:16:04.716230 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjkk5\" (UniqueName: \"kubernetes.io/projected/edd08497-99e0-4c46-b90e-be0bce94e70c-kube-api-access-hjkk5\") on node \"ip-10-0-137-203.ec2.internal\" DevicePath \"\"" Sep 17 10:16:04.716420 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:16:04.716241 2581 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/edd08497-99e0-4c46-b90e-be0bce94e70c-console-config\") on node \"ip-10-0-137-203.ec2.internal\" DevicePath \"\"" Sep 17 10:16:05.311160 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:16:05.311128 2581 generic.go:356] "Generic (PLEG): container finished" podID="edd08497-99e0-4c46-b90e-be0bce94e70c" containerID="cd7958d472899db433c3ddfb710700d10d7bbc5530836e95cc42a357b708be56" exitCode=0 Sep 17 10:16:05.311565 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:16:05.311169 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6fd5d66c4-vltj4" Sep 17 10:16:05.311565 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:16:05.311217 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6fd5d66c4-vltj4" event={"ID":"edd08497-99e0-4c46-b90e-be0bce94e70c","Type":"ContainerDied","Data":"cd7958d472899db433c3ddfb710700d10d7bbc5530836e95cc42a357b708be56"} Sep 17 10:16:05.311565 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:16:05.311273 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6fd5d66c4-vltj4" event={"ID":"edd08497-99e0-4c46-b90e-be0bce94e70c","Type":"ContainerDied","Data":"7c19a69d4fc6575beed98b7c4801ec8b3fd82b55dfa49b166177d5fdf4f35fdc"} Sep 17 10:16:05.311565 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:16:05.311298 2581 scope.go:117] "RemoveContainer" containerID="cd7958d472899db433c3ddfb710700d10d7bbc5530836e95cc42a357b708be56" Sep 17 10:16:05.320486 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:16:05.320467 2581 scope.go:117] "RemoveContainer" containerID="cd7958d472899db433c3ddfb710700d10d7bbc5530836e95cc42a357b708be56" Sep 17 10:16:05.320815 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:16:05.320771 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd7958d472899db433c3ddfb710700d10d7bbc5530836e95cc42a357b708be56\": container with ID starting with cd7958d472899db433c3ddfb710700d10d7bbc5530836e95cc42a357b708be56 not found: ID does not exist" containerID="cd7958d472899db433c3ddfb710700d10d7bbc5530836e95cc42a357b708be56" Sep 17 10:16:05.320912 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:16:05.320823 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd7958d472899db433c3ddfb710700d10d7bbc5530836e95cc42a357b708be56"} err="failed to get container status \"cd7958d472899db433c3ddfb710700d10d7bbc5530836e95cc42a357b708be56\": rpc error: code = NotFound desc = could not find container \"cd7958d472899db433c3ddfb710700d10d7bbc5530836e95cc42a357b708be56\": container with ID starting with cd7958d472899db433c3ddfb710700d10d7bbc5530836e95cc42a357b708be56 not found: ID does not exist" Sep 17 10:16:05.335562 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:16:05.335534 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6fd5d66c4-vltj4"] Sep 17 10:16:05.340007 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:16:05.339984 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6fd5d66c4-vltj4"] Sep 17 10:16:05.943653 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:16:05.943619 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="edd08497-99e0-4c46-b90e-be0bce94e70c" path="/var/lib/kubelet/pods/edd08497-99e0-4c46-b90e-be0bce94e70c/volumes" Sep 17 10:17:29.948611 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:17:29.948568 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-7899657bdc-68q7s"] Sep 17 10:17:29.949052 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:17:29.948896 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="edd08497-99e0-4c46-b90e-be0bce94e70c" containerName="console" Sep 17 10:17:29.949052 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:17:29.948908 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="edd08497-99e0-4c46-b90e-be0bce94e70c" containerName="console" Sep 17 10:17:29.949052 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:17:29.948975 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="edd08497-99e0-4c46-b90e-be0bce94e70c" containerName="console" Sep 17 10:17:29.951274 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:17:29.951259 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-68q7s" Sep 17 10:17:29.954119 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:17:29.954094 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\"" Sep 17 10:17:29.954243 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:17:29.954094 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-offline-tls\"" Sep 17 10:17:29.954243 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:17:29.954129 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-intra-comm\"" Sep 17 10:17:29.954243 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:17:29.954094 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-registry-tls\"" Sep 17 10:17:29.954409 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:17:29.954283 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-7zflc\"" Sep 17 10:17:29.954409 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:17:29.954368 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-ui-tls\"" Sep 17 10:17:29.954515 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:17:29.954483 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-online-tls\"" Sep 17 10:17:29.954573 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:17:29.954534 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\"" Sep 17 10:17:29.965683 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:17:29.965660 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-7899657bdc-68q7s"] Sep 17 10:17:30.045828 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:17:30.045774 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/f27c99ca-6c37-4de3-bb57-e0ed2cab0e14-online-tls\") pod \"feast-simple-feast-setup-7899657bdc-68q7s\" (UID: \"f27c99ca-6c37-4de3-bb57-e0ed2cab0e14\") " pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-68q7s" Sep 17 10:17:30.046003 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:17:30.045837 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/f27c99ca-6c37-4de3-bb57-e0ed2cab0e14-feast-data\") pod \"feast-simple-feast-setup-7899657bdc-68q7s\" (UID: \"f27c99ca-6c37-4de3-bb57-e0ed2cab0e14\") " pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-68q7s" Sep 17 10:17:30.046003 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:17:30.045896 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/f27c99ca-6c37-4de3-bb57-e0ed2cab0e14-offline-tls\") pod \"feast-simple-feast-setup-7899657bdc-68q7s\" (UID: \"f27c99ca-6c37-4de3-bb57-e0ed2cab0e14\") " pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-68q7s" Sep 17 10:17:30.046003 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:17:30.045949 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzjzx\" (UniqueName: \"kubernetes.io/projected/f27c99ca-6c37-4de3-bb57-e0ed2cab0e14-kube-api-access-rzjzx\") pod \"feast-simple-feast-setup-7899657bdc-68q7s\" (UID: \"f27c99ca-6c37-4de3-bb57-e0ed2cab0e14\") " pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-68q7s" Sep 17 10:17:30.046003 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:17:30.045978 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/f27c99ca-6c37-4de3-bb57-e0ed2cab0e14-registry-tls\") pod \"feast-simple-feast-setup-7899657bdc-68q7s\" (UID: \"f27c99ca-6c37-4de3-bb57-e0ed2cab0e14\") " pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-68q7s" Sep 17 10:17:30.046003 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:17:30.045995 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/f27c99ca-6c37-4de3-bb57-e0ed2cab0e14-ui-tls\") pod \"feast-simple-feast-setup-7899657bdc-68q7s\" (UID: \"f27c99ca-6c37-4de3-bb57-e0ed2cab0e14\") " pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-68q7s" Sep 17 10:17:30.146917 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:17:30.146876 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/f27c99ca-6c37-4de3-bb57-e0ed2cab0e14-feast-data\") pod \"feast-simple-feast-setup-7899657bdc-68q7s\" (UID: \"f27c99ca-6c37-4de3-bb57-e0ed2cab0e14\") " pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-68q7s" Sep 17 10:17:30.147071 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:17:30.146926 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/f27c99ca-6c37-4de3-bb57-e0ed2cab0e14-offline-tls\") pod \"feast-simple-feast-setup-7899657bdc-68q7s\" (UID: \"f27c99ca-6c37-4de3-bb57-e0ed2cab0e14\") " pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-68q7s" Sep 17 10:17:30.147071 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:17:30.146961 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzjzx\" (UniqueName: \"kubernetes.io/projected/f27c99ca-6c37-4de3-bb57-e0ed2cab0e14-kube-api-access-rzjzx\") pod \"feast-simple-feast-setup-7899657bdc-68q7s\" (UID: \"f27c99ca-6c37-4de3-bb57-e0ed2cab0e14\") " pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-68q7s" Sep 17 10:17:30.147071 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:17:30.146982 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/f27c99ca-6c37-4de3-bb57-e0ed2cab0e14-registry-tls\") pod \"feast-simple-feast-setup-7899657bdc-68q7s\" (UID: \"f27c99ca-6c37-4de3-bb57-e0ed2cab0e14\") " pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-68q7s" Sep 17 10:17:30.147071 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:17:30.147001 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/f27c99ca-6c37-4de3-bb57-e0ed2cab0e14-ui-tls\") pod \"feast-simple-feast-setup-7899657bdc-68q7s\" (UID: \"f27c99ca-6c37-4de3-bb57-e0ed2cab0e14\") " pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-68q7s" Sep 17 10:17:30.147071 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:17:30.147026 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/f27c99ca-6c37-4de3-bb57-e0ed2cab0e14-online-tls\") pod \"feast-simple-feast-setup-7899657bdc-68q7s\" (UID: \"f27c99ca-6c37-4de3-bb57-e0ed2cab0e14\") " pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-68q7s" Sep 17 10:17:30.147315 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:17:30.147127 2581 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-online-tls: secret "feast-simple-feast-setup-online-tls" not found Sep 17 10:17:30.147315 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:17:30.147195 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f27c99ca-6c37-4de3-bb57-e0ed2cab0e14-online-tls podName:f27c99ca-6c37-4de3-bb57-e0ed2cab0e14 nodeName:}" failed. No retries permitted until 2026-09-17 10:17:30.647172973 +0000 UTC m=+405.195730079 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/f27c99ca-6c37-4de3-bb57-e0ed2cab0e14-online-tls") pod "feast-simple-feast-setup-7899657bdc-68q7s" (UID: "f27c99ca-6c37-4de3-bb57-e0ed2cab0e14") : secret "feast-simple-feast-setup-online-tls" not found Sep 17 10:17:30.147315 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:17:30.147129 2581 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-offline-tls: secret "feast-simple-feast-setup-offline-tls" not found Sep 17 10:17:30.147315 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:17:30.147278 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f27c99ca-6c37-4de3-bb57-e0ed2cab0e14-offline-tls podName:f27c99ca-6c37-4de3-bb57-e0ed2cab0e14 nodeName:}" failed. No retries permitted until 2026-09-17 10:17:30.647259364 +0000 UTC m=+405.195816471 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/f27c99ca-6c37-4de3-bb57-e0ed2cab0e14-offline-tls") pod "feast-simple-feast-setup-7899657bdc-68q7s" (UID: "f27c99ca-6c37-4de3-bb57-e0ed2cab0e14") : secret "feast-simple-feast-setup-offline-tls" not found Sep 17 10:17:30.147315 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:17:30.147128 2581 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-registry-tls: secret "feast-simple-feast-setup-registry-tls" not found Sep 17 10:17:30.147502 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:17:30.147322 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/f27c99ca-6c37-4de3-bb57-e0ed2cab0e14-feast-data\") pod \"feast-simple-feast-setup-7899657bdc-68q7s\" (UID: \"f27c99ca-6c37-4de3-bb57-e0ed2cab0e14\") " pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-68q7s" Sep 17 10:17:30.147502 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:17:30.147355 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f27c99ca-6c37-4de3-bb57-e0ed2cab0e14-registry-tls podName:f27c99ca-6c37-4de3-bb57-e0ed2cab0e14 nodeName:}" failed. No retries permitted until 2026-09-17 10:17:30.647336154 +0000 UTC m=+405.195893260 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/f27c99ca-6c37-4de3-bb57-e0ed2cab0e14-registry-tls") pod "feast-simple-feast-setup-7899657bdc-68q7s" (UID: "f27c99ca-6c37-4de3-bb57-e0ed2cab0e14") : secret "feast-simple-feast-setup-registry-tls" not found Sep 17 10:17:30.149579 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:17:30.149557 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/f27c99ca-6c37-4de3-bb57-e0ed2cab0e14-ui-tls\") pod \"feast-simple-feast-setup-7899657bdc-68q7s\" (UID: \"f27c99ca-6c37-4de3-bb57-e0ed2cab0e14\") " pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-68q7s" Sep 17 10:17:30.159438 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:17:30.159414 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzjzx\" (UniqueName: \"kubernetes.io/projected/f27c99ca-6c37-4de3-bb57-e0ed2cab0e14-kube-api-access-rzjzx\") pod \"feast-simple-feast-setup-7899657bdc-68q7s\" (UID: \"f27c99ca-6c37-4de3-bb57-e0ed2cab0e14\") " pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-68q7s" Sep 17 10:17:30.651398 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:17:30.651358 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/f27c99ca-6c37-4de3-bb57-e0ed2cab0e14-registry-tls\") pod \"feast-simple-feast-setup-7899657bdc-68q7s\" (UID: \"f27c99ca-6c37-4de3-bb57-e0ed2cab0e14\") " pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-68q7s" Sep 17 10:17:30.651587 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:17:30.651410 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/f27c99ca-6c37-4de3-bb57-e0ed2cab0e14-online-tls\") pod \"feast-simple-feast-setup-7899657bdc-68q7s\" (UID: \"f27c99ca-6c37-4de3-bb57-e0ed2cab0e14\") " pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-68q7s" Sep 17 10:17:30.651587 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:17:30.651461 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/f27c99ca-6c37-4de3-bb57-e0ed2cab0e14-offline-tls\") pod \"feast-simple-feast-setup-7899657bdc-68q7s\" (UID: \"f27c99ca-6c37-4de3-bb57-e0ed2cab0e14\") " pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-68q7s" Sep 17 10:17:30.653996 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:17:30.653963 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/f27c99ca-6c37-4de3-bb57-e0ed2cab0e14-offline-tls\") pod \"feast-simple-feast-setup-7899657bdc-68q7s\" (UID: \"f27c99ca-6c37-4de3-bb57-e0ed2cab0e14\") " pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-68q7s" Sep 17 10:17:30.653996 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:17:30.653990 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/f27c99ca-6c37-4de3-bb57-e0ed2cab0e14-registry-tls\") pod \"feast-simple-feast-setup-7899657bdc-68q7s\" (UID: \"f27c99ca-6c37-4de3-bb57-e0ed2cab0e14\") " pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-68q7s" Sep 17 10:17:30.654152 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:17:30.654012 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/f27c99ca-6c37-4de3-bb57-e0ed2cab0e14-online-tls\") pod \"feast-simple-feast-setup-7899657bdc-68q7s\" (UID: \"f27c99ca-6c37-4de3-bb57-e0ed2cab0e14\") " pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-68q7s" Sep 17 10:17:30.860470 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:17:30.860427 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-68q7s" Sep 17 10:17:31.137929 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:17:31.137903 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-7899657bdc-68q7s"] Sep 17 10:17:31.138916 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:17:31.138893 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf27c99ca_6c37_4de3_bb57_e0ed2cab0e14.slice/crio-34cbba3b12cf91bec9291956b60d39f1701190357968e525783dbcc94d3da939 WatchSource:0}: Error finding container 34cbba3b12cf91bec9291956b60d39f1701190357968e525783dbcc94d3da939: Status 404 returned error can't find the container with id 34cbba3b12cf91bec9291956b60d39f1701190357968e525783dbcc94d3da939 Sep 17 10:17:31.140745 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:17:31.140730 2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 17 10:17:31.631154 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:17:31.631114 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-68q7s" event={"ID":"f27c99ca-6c37-4de3-bb57-e0ed2cab0e14","Type":"ContainerStarted","Data":"34cbba3b12cf91bec9291956b60d39f1701190357968e525783dbcc94d3da939"} Sep 17 10:17:55.095110 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:17:55.095067 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-8fcc87966-pblxr"] Sep 17 10:17:55.156886 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:17:55.156843 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8fcc87966-pblxr"] Sep 17 10:17:55.156886 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:17:55.156990 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8fcc87966-pblxr" Sep 17 10:17:55.263184 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:17:55.263146 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d85944aa-7e5a-4243-9b09-8a5f451b3bd1-trusted-ca-bundle\") pod \"console-8fcc87966-pblxr\" (UID: \"d85944aa-7e5a-4243-9b09-8a5f451b3bd1\") " pod="openshift-console/console-8fcc87966-pblxr" Sep 17 10:17:55.263354 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:17:55.263196 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d85944aa-7e5a-4243-9b09-8a5f451b3bd1-console-config\") pod \"console-8fcc87966-pblxr\" (UID: \"d85944aa-7e5a-4243-9b09-8a5f451b3bd1\") " pod="openshift-console/console-8fcc87966-pblxr" Sep 17 10:17:55.263354 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:17:55.263268 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d85944aa-7e5a-4243-9b09-8a5f451b3bd1-console-serving-cert\") pod \"console-8fcc87966-pblxr\" (UID: \"d85944aa-7e5a-4243-9b09-8a5f451b3bd1\") " pod="openshift-console/console-8fcc87966-pblxr" Sep 17 10:17:55.263354 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:17:55.263290 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d85944aa-7e5a-4243-9b09-8a5f451b3bd1-service-ca\") pod \"console-8fcc87966-pblxr\" (UID: \"d85944aa-7e5a-4243-9b09-8a5f451b3bd1\") " pod="openshift-console/console-8fcc87966-pblxr" Sep 17 10:17:55.263354 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:17:55.263317 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d85944aa-7e5a-4243-9b09-8a5f451b3bd1-oauth-serving-cert\") pod \"console-8fcc87966-pblxr\" (UID: \"d85944aa-7e5a-4243-9b09-8a5f451b3bd1\") " pod="openshift-console/console-8fcc87966-pblxr" Sep 17 10:17:55.263354 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:17:55.263337 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d85944aa-7e5a-4243-9b09-8a5f451b3bd1-console-oauth-config\") pod \"console-8fcc87966-pblxr\" (UID: \"d85944aa-7e5a-4243-9b09-8a5f451b3bd1\") " pod="openshift-console/console-8fcc87966-pblxr" Sep 17 10:17:55.263525 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:17:55.263429 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dx6m7\" (UniqueName: \"kubernetes.io/projected/d85944aa-7e5a-4243-9b09-8a5f451b3bd1-kube-api-access-dx6m7\") pod \"console-8fcc87966-pblxr\" (UID: \"d85944aa-7e5a-4243-9b09-8a5f451b3bd1\") " pod="openshift-console/console-8fcc87966-pblxr" Sep 17 10:17:55.364538 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:17:55.364452 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d85944aa-7e5a-4243-9b09-8a5f451b3bd1-console-serving-cert\") pod \"console-8fcc87966-pblxr\" (UID: \"d85944aa-7e5a-4243-9b09-8a5f451b3bd1\") " pod="openshift-console/console-8fcc87966-pblxr" Sep 17 10:17:55.364538 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:17:55.364501 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d85944aa-7e5a-4243-9b09-8a5f451b3bd1-service-ca\") pod \"console-8fcc87966-pblxr\" (UID: \"d85944aa-7e5a-4243-9b09-8a5f451b3bd1\") " pod="openshift-console/console-8fcc87966-pblxr" Sep 17 10:17:55.364756 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:17:55.364547 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d85944aa-7e5a-4243-9b09-8a5f451b3bd1-oauth-serving-cert\") pod \"console-8fcc87966-pblxr\" (UID: \"d85944aa-7e5a-4243-9b09-8a5f451b3bd1\") " pod="openshift-console/console-8fcc87966-pblxr" Sep 17 10:17:55.364756 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:17:55.364577 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d85944aa-7e5a-4243-9b09-8a5f451b3bd1-console-oauth-config\") pod \"console-8fcc87966-pblxr\" (UID: \"d85944aa-7e5a-4243-9b09-8a5f451b3bd1\") " pod="openshift-console/console-8fcc87966-pblxr" Sep 17 10:17:55.364756 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:17:55.364603 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dx6m7\" (UniqueName: \"kubernetes.io/projected/d85944aa-7e5a-4243-9b09-8a5f451b3bd1-kube-api-access-dx6m7\") pod \"console-8fcc87966-pblxr\" (UID: \"d85944aa-7e5a-4243-9b09-8a5f451b3bd1\") " pod="openshift-console/console-8fcc87966-pblxr" Sep 17 10:17:55.364756 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:17:55.364634 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d85944aa-7e5a-4243-9b09-8a5f451b3bd1-trusted-ca-bundle\") pod \"console-8fcc87966-pblxr\" (UID: \"d85944aa-7e5a-4243-9b09-8a5f451b3bd1\") " pod="openshift-console/console-8fcc87966-pblxr" Sep 17 10:17:55.364756 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:17:55.364666 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d85944aa-7e5a-4243-9b09-8a5f451b3bd1-console-config\") pod \"console-8fcc87966-pblxr\" (UID: \"d85944aa-7e5a-4243-9b09-8a5f451b3bd1\") " pod="openshift-console/console-8fcc87966-pblxr" Sep 17 10:17:55.365453 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:17:55.365390 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d85944aa-7e5a-4243-9b09-8a5f451b3bd1-service-ca\") pod \"console-8fcc87966-pblxr\" (UID: \"d85944aa-7e5a-4243-9b09-8a5f451b3bd1\") " pod="openshift-console/console-8fcc87966-pblxr" Sep 17 10:17:55.365572 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:17:55.365521 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d85944aa-7e5a-4243-9b09-8a5f451b3bd1-oauth-serving-cert\") pod \"console-8fcc87966-pblxr\" (UID: \"d85944aa-7e5a-4243-9b09-8a5f451b3bd1\") " pod="openshift-console/console-8fcc87966-pblxr" Sep 17 10:17:55.365752 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:17:55.365725 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d85944aa-7e5a-4243-9b09-8a5f451b3bd1-console-config\") pod \"console-8fcc87966-pblxr\" (UID: \"d85944aa-7e5a-4243-9b09-8a5f451b3bd1\") " pod="openshift-console/console-8fcc87966-pblxr" Sep 17 10:17:55.365869 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:17:55.365765 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d85944aa-7e5a-4243-9b09-8a5f451b3bd1-trusted-ca-bundle\") pod \"console-8fcc87966-pblxr\" (UID: \"d85944aa-7e5a-4243-9b09-8a5f451b3bd1\") " pod="openshift-console/console-8fcc87966-pblxr" Sep 17 10:17:55.367380 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:17:55.367361 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d85944aa-7e5a-4243-9b09-8a5f451b3bd1-console-serving-cert\") pod \"console-8fcc87966-pblxr\" (UID: \"d85944aa-7e5a-4243-9b09-8a5f451b3bd1\") " pod="openshift-console/console-8fcc87966-pblxr" Sep 17 10:17:55.367528 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:17:55.367494 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d85944aa-7e5a-4243-9b09-8a5f451b3bd1-console-oauth-config\") pod \"console-8fcc87966-pblxr\" (UID: \"d85944aa-7e5a-4243-9b09-8a5f451b3bd1\") " pod="openshift-console/console-8fcc87966-pblxr" Sep 17 10:17:55.375593 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:17:55.375573 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dx6m7\" (UniqueName: \"kubernetes.io/projected/d85944aa-7e5a-4243-9b09-8a5f451b3bd1-kube-api-access-dx6m7\") pod \"console-8fcc87966-pblxr\" (UID: \"d85944aa-7e5a-4243-9b09-8a5f451b3bd1\") " pod="openshift-console/console-8fcc87966-pblxr" Sep 17 10:17:55.469726 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:17:55.469666 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8fcc87966-pblxr" Sep 17 10:17:59.055216 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:17:59.055187 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8fcc87966-pblxr"] Sep 17 10:17:59.057386 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:17:59.057353 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd85944aa_7e5a_4243_9b09_8a5f451b3bd1.slice/crio-80a13687e76514f705bbd9537aafde7618ab82e8606f04eaa8b127ba9d5ba653 WatchSource:0}: Error finding container 80a13687e76514f705bbd9537aafde7618ab82e8606f04eaa8b127ba9d5ba653: Status 404 returned error can't find the container with id 80a13687e76514f705bbd9537aafde7618ab82e8606f04eaa8b127ba9d5ba653 Sep 17 10:17:59.761407 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:17:59.761369 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8fcc87966-pblxr" event={"ID":"d85944aa-7e5a-4243-9b09-8a5f451b3bd1","Type":"ContainerStarted","Data":"c15a4bdee14f2e1ce0896694eb02d76d9f1cd8b4ef9176a9d8a0515a515faf1d"} Sep 17 10:17:59.761599 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:17:59.761424 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8fcc87966-pblxr" event={"ID":"d85944aa-7e5a-4243-9b09-8a5f451b3bd1","Type":"ContainerStarted","Data":"80a13687e76514f705bbd9537aafde7618ab82e8606f04eaa8b127ba9d5ba653"} Sep 17 10:17:59.762907 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:17:59.762880 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-68q7s" event={"ID":"f27c99ca-6c37-4de3-bb57-e0ed2cab0e14","Type":"ContainerStarted","Data":"f96a872a7b6f6d17a7ee820a0d04e576c401fff2c3f432f73c1b1400a523b7cf"} Sep 17 10:17:59.780761 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:17:59.780713 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-8fcc87966-pblxr" podStartSLOduration=4.780700485 podStartE2EDuration="4.780700485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 10:17:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 10:17:59.779124914 +0000 UTC m=+434.327682026" watchObservedRunningTime="2026-09-17 10:17:59.780700485 +0000 UTC m=+434.329257597" Sep 17 10:18:04.787051 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:18:04.787015 2581 generic.go:356] "Generic (PLEG): container finished" podID="f27c99ca-6c37-4de3-bb57-e0ed2cab0e14" containerID="f96a872a7b6f6d17a7ee820a0d04e576c401fff2c3f432f73c1b1400a523b7cf" exitCode=0 Sep 17 10:18:04.787439 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:18:04.787102 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-68q7s" event={"ID":"f27c99ca-6c37-4de3-bb57-e0ed2cab0e14","Type":"ContainerDied","Data":"f96a872a7b6f6d17a7ee820a0d04e576c401fff2c3f432f73c1b1400a523b7cf"} Sep 17 10:18:05.470700 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:18:05.470667 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-8fcc87966-pblxr" Sep 17 10:18:05.471025 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:18:05.470984 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-8fcc87966-pblxr" Sep 17 10:18:05.475766 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:18:05.475740 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-8fcc87966-pblxr" Sep 17 10:18:05.793698 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:18:05.793663 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-68q7s" event={"ID":"f27c99ca-6c37-4de3-bb57-e0ed2cab0e14","Type":"ContainerStarted","Data":"bef3a4294c147ea2b16eb231114afb2c457a1917fab86a17948d39b0359bf0b0"} Sep 17 10:18:05.797544 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:18:05.797521 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-8fcc87966-pblxr" Sep 17 10:18:05.868075 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:18:05.868041 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5cc7c64bbf-dmfkl"] Sep 17 10:18:10.817427 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:18:10.817395 2581 generic.go:356] "Generic (PLEG): container finished" podID="f27c99ca-6c37-4de3-bb57-e0ed2cab0e14" containerID="bef3a4294c147ea2b16eb231114afb2c457a1917fab86a17948d39b0359bf0b0" exitCode=0 Sep 17 10:18:10.817755 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:18:10.817437 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-68q7s" event={"ID":"f27c99ca-6c37-4de3-bb57-e0ed2cab0e14","Type":"ContainerDied","Data":"bef3a4294c147ea2b16eb231114afb2c457a1917fab86a17948d39b0359bf0b0"} Sep 17 10:18:10.823194 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:18:10.823167 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-intra-comm\"" Sep 17 10:18:11.827460 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:18:11.827420 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-68q7s" event={"ID":"f27c99ca-6c37-4de3-bb57-e0ed2cab0e14","Type":"ContainerStarted","Data":"fea0c75bf16433402585081186269b372071d21ec90e52fdf019bf6dd0b5f3e9"} Sep 17 10:18:11.827951 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:18:11.827484 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-68q7s" event={"ID":"f27c99ca-6c37-4de3-bb57-e0ed2cab0e14","Type":"ContainerStarted","Data":"e7b70d84ec1a0baa6acb85e3c76ae2f6b412ad19b0c91ab29c715617ec8e5141"} Sep 17 10:18:11.827951 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:18:11.827506 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-68q7s" event={"ID":"f27c99ca-6c37-4de3-bb57-e0ed2cab0e14","Type":"ContainerStarted","Data":"b8ed4c8223befc38f3dfa290538153c5afe97fb2e11d6b3547ffa06bf847514d"} Sep 17 10:18:11.827951 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:18:11.827525 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-68q7s" event={"ID":"f27c99ca-6c37-4de3-bb57-e0ed2cab0e14","Type":"ContainerStarted","Data":"1afe3d8222e9c58133d3373f500e8f4e21e430bdaa4d583eab4b66ccc57cc8dc"} Sep 17 10:18:11.855862 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:18:11.855799 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-68q7s" podStartSLOduration=15.09360917 podStartE2EDuration="42.85576024s" totalImagesPullingTime="27.76215107s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 10:17:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 10:18:11.854192623 +0000 UTC m=+446.402749762" watchObservedRunningTime="2026-09-17 10:18:11.85576024 +0000 UTC m=+446.404317355" Sep 17 10:18:12.861561 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:18:12.861444 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-68q7s" Sep 17 10:18:12.861561 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:18:12.861505 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-68q7s" Sep 17 10:18:12.861561 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:18:12.861524 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-68q7s" Sep 17 10:18:12.861561 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:18:12.861556 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-68q7s" Sep 17 10:18:12.863823 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:18:12.863768 2581 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-68q7s" podUID="f27c99ca-6c37-4de3-bb57-e0ed2cab0e14" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.25:8443: connect: connection refused" Sep 17 10:18:12.864051 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:18:12.863826 2581 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-68q7s" podUID="f27c99ca-6c37-4de3-bb57-e0ed2cab0e14" containerName="online" probeResult="failure" output="Get \"https://10.134.0.25:6567/health\": dial tcp 10.134.0.25:6567: connect: connection refused" Sep 17 10:18:12.864198 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:18:12.863867 2581 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-68q7s" podUID="f27c99ca-6c37-4de3-bb57-e0ed2cab0e14" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.25:8816: connect: connection refused" Sep 17 10:18:12.864354 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:18:12.863866 2581 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-68q7s" podUID="f27c99ca-6c37-4de3-bb57-e0ed2cab0e14" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.25:6571: connect: connection refused" Sep 17 10:18:15.861528 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:18:15.861475 2581 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-68q7s" podUID="f27c99ca-6c37-4de3-bb57-e0ed2cab0e14" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.25:8443: connect: connection refused" Sep 17 10:18:15.862225 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:18:15.862159 2581 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-68q7s" podUID="f27c99ca-6c37-4de3-bb57-e0ed2cab0e14" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.25:6571: connect: connection refused" Sep 17 10:18:15.862225 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:18:15.862190 2581 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-68q7s" podUID="f27c99ca-6c37-4de3-bb57-e0ed2cab0e14" containerName="online" probeResult="failure" output="Get \"https://10.134.0.25:6567/health\": dial tcp 10.134.0.25:6567: connect: connection refused" Sep 17 10:18:15.862460 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:18:15.862181 2581 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-68q7s" podUID="f27c99ca-6c37-4de3-bb57-e0ed2cab0e14" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.25:8816: connect: connection refused" Sep 17 10:18:18.861110 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:18:18.861065 2581 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-68q7s" podUID="f27c99ca-6c37-4de3-bb57-e0ed2cab0e14" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.25:8443: connect: connection refused" Sep 17 10:18:18.867975 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:18:18.867938 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-68q7s" Sep 17 10:18:18.868146 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:18:18.868133 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-68q7s" Sep 17 10:18:18.868631 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:18:18.868563 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-68q7s" Sep 17 10:18:18.868631 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:18:18.868604 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-68q7s" Sep 17 10:18:18.868979 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:18:18.868954 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-68q7s" Sep 17 10:18:18.869084 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:18:18.869035 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-68q7s" Sep 17 10:18:18.873005 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:18:18.872984 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-68q7s" Sep 17 10:18:18.879392 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:18:18.879373 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-68q7s" Sep 17 10:18:18.882096 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:18:18.882076 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-68q7s" Sep 17 10:18:20.860663 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:18:20.860629 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-68q7s" Sep 17 10:18:21.861852 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:18:21.861822 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-68q7s" Sep 17 10:18:21.891936 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:18:21.891913 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-68q7s" Sep 17 10:18:30.892542 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:18:30.892506 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5cc7c64bbf-dmfkl" podUID="f14a6868-bbda-4e64-b336-e3dbe3b47906" containerName="console" containerID="cri-o://2c0246b431c977f0218ba271ad5694917f1b749b47470527f7eb480f8f0fd52b" gracePeriod=15 Sep 17 10:18:31.129446 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:18:31.129423 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5cc7c64bbf-dmfkl" Sep 17 10:18:31.175923 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:18:31.175851 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f14a6868-bbda-4e64-b336-e3dbe3b47906-console-config\") pod \"f14a6868-bbda-4e64-b336-e3dbe3b47906\" (UID: \"f14a6868-bbda-4e64-b336-e3dbe3b47906\") " Sep 17 10:18:31.175923 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:18:31.175901 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f14a6868-bbda-4e64-b336-e3dbe3b47906-oauth-serving-cert\") pod \"f14a6868-bbda-4e64-b336-e3dbe3b47906\" (UID: \"f14a6868-bbda-4e64-b336-e3dbe3b47906\") " Sep 17 10:18:31.176117 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:18:31.175945 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f14a6868-bbda-4e64-b336-e3dbe3b47906-service-ca\") pod \"f14a6868-bbda-4e64-b336-e3dbe3b47906\" (UID: \"f14a6868-bbda-4e64-b336-e3dbe3b47906\") " Sep 17 10:18:31.176117 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:18:31.176062 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f14a6868-bbda-4e64-b336-e3dbe3b47906-console-serving-cert\") pod \"f14a6868-bbda-4e64-b336-e3dbe3b47906\" (UID: \"f14a6868-bbda-4e64-b336-e3dbe3b47906\") " Sep 17 10:18:31.176280 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:18:31.176130 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f14a6868-bbda-4e64-b336-e3dbe3b47906-trusted-ca-bundle\") pod \"f14a6868-bbda-4e64-b336-e3dbe3b47906\" (UID: \"f14a6868-bbda-4e64-b336-e3dbe3b47906\") " Sep 17 10:18:31.176280 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:18:31.176171 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g2mf2\" (UniqueName: \"kubernetes.io/projected/f14a6868-bbda-4e64-b336-e3dbe3b47906-kube-api-access-g2mf2\") pod \"f14a6868-bbda-4e64-b336-e3dbe3b47906\" (UID: \"f14a6868-bbda-4e64-b336-e3dbe3b47906\") " Sep 17 10:18:31.176280 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:18:31.176226 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f14a6868-bbda-4e64-b336-e3dbe3b47906-console-oauth-config\") pod \"f14a6868-bbda-4e64-b336-e3dbe3b47906\" (UID: \"f14a6868-bbda-4e64-b336-e3dbe3b47906\") " Sep 17 10:18:31.176429 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:18:31.176327 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f14a6868-bbda-4e64-b336-e3dbe3b47906-service-ca" (OuterVolumeSpecName: "service-ca") pod "f14a6868-bbda-4e64-b336-e3dbe3b47906" (UID: "f14a6868-bbda-4e64-b336-e3dbe3b47906"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 10:18:31.176491 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:18:31.176325 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f14a6868-bbda-4e64-b336-e3dbe3b47906-console-config" (OuterVolumeSpecName: "console-config") pod "f14a6868-bbda-4e64-b336-e3dbe3b47906" (UID: "f14a6868-bbda-4e64-b336-e3dbe3b47906"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 10:18:31.176491 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:18:31.176463 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f14a6868-bbda-4e64-b336-e3dbe3b47906-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "f14a6868-bbda-4e64-b336-e3dbe3b47906" (UID: "f14a6868-bbda-4e64-b336-e3dbe3b47906"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 10:18:31.176589 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:18:31.176567 2581 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f14a6868-bbda-4e64-b336-e3dbe3b47906-console-config\") on node \"ip-10-0-137-203.ec2.internal\" DevicePath \"\"" Sep 17 10:18:31.176589 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:18:31.176581 2581 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f14a6868-bbda-4e64-b336-e3dbe3b47906-oauth-serving-cert\") on node \"ip-10-0-137-203.ec2.internal\" DevicePath \"\"" Sep 17 10:18:31.176697 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:18:31.176601 2581 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f14a6868-bbda-4e64-b336-e3dbe3b47906-service-ca\") on node \"ip-10-0-137-203.ec2.internal\" DevicePath \"\"" Sep 17 10:18:31.176697 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:18:31.176560 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f14a6868-bbda-4e64-b336-e3dbe3b47906-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "f14a6868-bbda-4e64-b336-e3dbe3b47906" (UID: "f14a6868-bbda-4e64-b336-e3dbe3b47906"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 10:18:31.178272 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:18:31.178243 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f14a6868-bbda-4e64-b336-e3dbe3b47906-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "f14a6868-bbda-4e64-b336-e3dbe3b47906" (UID: "f14a6868-bbda-4e64-b336-e3dbe3b47906"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 10:18:31.178356 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:18:31.178272 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f14a6868-bbda-4e64-b336-e3dbe3b47906-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "f14a6868-bbda-4e64-b336-e3dbe3b47906" (UID: "f14a6868-bbda-4e64-b336-e3dbe3b47906"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 10:18:31.178356 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:18:31.178335 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f14a6868-bbda-4e64-b336-e3dbe3b47906-kube-api-access-g2mf2" (OuterVolumeSpecName: "kube-api-access-g2mf2") pod "f14a6868-bbda-4e64-b336-e3dbe3b47906" (UID: "f14a6868-bbda-4e64-b336-e3dbe3b47906"). InnerVolumeSpecName "kube-api-access-g2mf2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 10:18:31.276972 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:18:31.276943 2581 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f14a6868-bbda-4e64-b336-e3dbe3b47906-trusted-ca-bundle\") on node \"ip-10-0-137-203.ec2.internal\" DevicePath \"\"" Sep 17 10:18:31.276972 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:18:31.276969 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2mf2\" (UniqueName: \"kubernetes.io/projected/f14a6868-bbda-4e64-b336-e3dbe3b47906-kube-api-access-g2mf2\") on node \"ip-10-0-137-203.ec2.internal\" DevicePath \"\"" Sep 17 10:18:31.276972 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:18:31.276979 2581 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f14a6868-bbda-4e64-b336-e3dbe3b47906-console-oauth-config\") on node \"ip-10-0-137-203.ec2.internal\" DevicePath \"\"" Sep 17 10:18:31.276972 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:18:31.276987 2581 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f14a6868-bbda-4e64-b336-e3dbe3b47906-console-serving-cert\") on node \"ip-10-0-137-203.ec2.internal\" DevicePath \"\"" Sep 17 10:18:31.931750 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:18:31.931718 2581 generic.go:356] "Generic (PLEG): container finished" podID="f14a6868-bbda-4e64-b336-e3dbe3b47906" containerID="2c0246b431c977f0218ba271ad5694917f1b749b47470527f7eb480f8f0fd52b" exitCode=0 Sep 17 10:18:31.932231 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:18:31.931760 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5cc7c64bbf-dmfkl" event={"ID":"f14a6868-bbda-4e64-b336-e3dbe3b47906","Type":"ContainerDied","Data":"2c0246b431c977f0218ba271ad5694917f1b749b47470527f7eb480f8f0fd52b"} Sep 17 10:18:31.932231 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:18:31.931775 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5cc7c64bbf-dmfkl" Sep 17 10:18:31.932231 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:18:31.931845 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5cc7c64bbf-dmfkl" event={"ID":"f14a6868-bbda-4e64-b336-e3dbe3b47906","Type":"ContainerDied","Data":"90a4953ef45531ca390ac9dc5cc0b983eae66850052812ce1be3ec375f1a6e34"} Sep 17 10:18:31.932231 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:18:31.931866 2581 scope.go:117] "RemoveContainer" containerID="2c0246b431c977f0218ba271ad5694917f1b749b47470527f7eb480f8f0fd52b" Sep 17 10:18:31.941821 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:18:31.941800 2581 scope.go:117] "RemoveContainer" containerID="2c0246b431c977f0218ba271ad5694917f1b749b47470527f7eb480f8f0fd52b" Sep 17 10:18:31.942070 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:18:31.942052 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c0246b431c977f0218ba271ad5694917f1b749b47470527f7eb480f8f0fd52b\": container with ID starting with 2c0246b431c977f0218ba271ad5694917f1b749b47470527f7eb480f8f0fd52b not found: ID does not exist" containerID="2c0246b431c977f0218ba271ad5694917f1b749b47470527f7eb480f8f0fd52b" Sep 17 10:18:31.942117 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:18:31.942077 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c0246b431c977f0218ba271ad5694917f1b749b47470527f7eb480f8f0fd52b"} err="failed to get container status \"2c0246b431c977f0218ba271ad5694917f1b749b47470527f7eb480f8f0fd52b\": rpc error: code = NotFound desc = could not find container \"2c0246b431c977f0218ba271ad5694917f1b749b47470527f7eb480f8f0fd52b\": container with ID starting with 2c0246b431c977f0218ba271ad5694917f1b749b47470527f7eb480f8f0fd52b not found: ID does not exist" Sep 17 10:18:31.959239 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:18:31.959210 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5cc7c64bbf-dmfkl"] Sep 17 10:18:31.960564 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:18:31.960538 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5cc7c64bbf-dmfkl"] Sep 17 10:18:33.943224 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:18:33.943193 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f14a6868-bbda-4e64-b336-e3dbe3b47906" path="/var/lib/kubelet/pods/f14a6868-bbda-4e64-b336-e3dbe3b47906/volumes" Sep 17 10:19:24.206645 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:19:24.206559 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-7899657bdc-68q7s"] Sep 17 10:19:24.207222 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:19:24.206886 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-68q7s" podUID="f27c99ca-6c37-4de3-bb57-e0ed2cab0e14" containerName="ui" containerID="cri-o://fea0c75bf16433402585081186269b372071d21ec90e52fdf019bf6dd0b5f3e9" gracePeriod=30 Sep 17 10:19:24.207222 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:19:24.206965 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-68q7s" podUID="f27c99ca-6c37-4de3-bb57-e0ed2cab0e14" containerName="offline" containerID="cri-o://e7b70d84ec1a0baa6acb85e3c76ae2f6b412ad19b0c91ab29c715617ec8e5141" gracePeriod=30 Sep 17 10:19:24.207222 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:19:24.206866 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-68q7s" podUID="f27c99ca-6c37-4de3-bb57-e0ed2cab0e14" containerName="registry" containerID="cri-o://1afe3d8222e9c58133d3373f500e8f4e21e430bdaa4d583eab4b66ccc57cc8dc" gracePeriod=30 Sep 17 10:19:24.207222 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:19:24.206951 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-68q7s" podUID="f27c99ca-6c37-4de3-bb57-e0ed2cab0e14" containerName="online" containerID="cri-o://b8ed4c8223befc38f3dfa290538153c5afe97fb2e11d6b3547ffa06bf847514d" gracePeriod=30 Sep 17 10:19:24.611855 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:19:24.611814 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-7899657bdc-hqsrp"] Sep 17 10:19:24.612807 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:19:24.612762 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f14a6868-bbda-4e64-b336-e3dbe3b47906" containerName="console" Sep 17 10:19:24.612964 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:19:24.612952 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f14a6868-bbda-4e64-b336-e3dbe3b47906" containerName="console" Sep 17 10:19:24.613219 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:19:24.613206 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f14a6868-bbda-4e64-b336-e3dbe3b47906" containerName="console" Sep 17 10:19:24.618299 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:19:24.618273 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-hqsrp" Sep 17 10:19:24.621206 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:19:24.621189 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-jrjwb\"" Sep 17 10:19:24.629565 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:19:24.629539 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-7899657bdc-hqsrp"] Sep 17 10:19:24.706653 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:19:24.706618 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7-feast-data\") pod \"feast-simple-feast-setup-7899657bdc-hqsrp\" (UID: \"7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7\") " pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-hqsrp" Sep 17 10:19:24.706890 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:19:24.706674 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7-registry-tls\") pod \"feast-simple-feast-setup-7899657bdc-hqsrp\" (UID: \"7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7\") " pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-hqsrp" Sep 17 10:19:24.706890 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:19:24.706709 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7-ui-tls\") pod \"feast-simple-feast-setup-7899657bdc-hqsrp\" (UID: \"7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7\") " pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-hqsrp" Sep 17 10:19:24.706890 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:19:24.706824 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7-offline-tls\") pod \"feast-simple-feast-setup-7899657bdc-hqsrp\" (UID: \"7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7\") " pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-hqsrp" Sep 17 10:19:24.706890 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:19:24.706860 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hr9zt\" (UniqueName: \"kubernetes.io/projected/7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7-kube-api-access-hr9zt\") pod \"feast-simple-feast-setup-7899657bdc-hqsrp\" (UID: \"7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7\") " pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-hqsrp" Sep 17 10:19:24.706890 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:19:24.706886 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7-online-tls\") pod \"feast-simple-feast-setup-7899657bdc-hqsrp\" (UID: \"7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7\") " pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-hqsrp" Sep 17 10:19:24.807555 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:19:24.807528 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7-registry-tls\") pod \"feast-simple-feast-setup-7899657bdc-hqsrp\" (UID: \"7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7\") " pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-hqsrp" Sep 17 10:19:24.807633 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:19:24.807566 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7-ui-tls\") pod \"feast-simple-feast-setup-7899657bdc-hqsrp\" (UID: \"7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7\") " pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-hqsrp" Sep 17 10:19:24.807633 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:19:24.807607 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7-offline-tls\") pod \"feast-simple-feast-setup-7899657bdc-hqsrp\" (UID: \"7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7\") " pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-hqsrp" Sep 17 10:19:24.807735 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:19:24.807630 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hr9zt\" (UniqueName: \"kubernetes.io/projected/7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7-kube-api-access-hr9zt\") pod \"feast-simple-feast-setup-7899657bdc-hqsrp\" (UID: \"7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7\") " pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-hqsrp" Sep 17 10:19:24.807735 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:19:24.807663 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7-online-tls\") pod \"feast-simple-feast-setup-7899657bdc-hqsrp\" (UID: \"7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7\") " pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-hqsrp" Sep 17 10:19:24.807735 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:19:24.807693 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7-feast-data\") pod \"feast-simple-feast-setup-7899657bdc-hqsrp\" (UID: \"7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7\") " pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-hqsrp" Sep 17 10:19:24.807735 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:19:24.807701 2581 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-registry-tls: secret "feast-simple-feast-setup-registry-tls" not found Sep 17 10:19:24.807939 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:19:24.807826 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7-registry-tls podName:7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7 nodeName:}" failed. No retries permitted until 2026-09-17 10:19:25.307775543 +0000 UTC m=+519.856332646 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7-registry-tls") pod "feast-simple-feast-setup-7899657bdc-hqsrp" (UID: "7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7") : secret "feast-simple-feast-setup-registry-tls" not found Sep 17 10:19:24.808183 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:19:24.808159 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7-feast-data\") pod \"feast-simple-feast-setup-7899657bdc-hqsrp\" (UID: \"7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7\") " pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-hqsrp" Sep 17 10:19:24.810434 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:19:24.810409 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7-ui-tls\") pod \"feast-simple-feast-setup-7899657bdc-hqsrp\" (UID: \"7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7\") " pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-hqsrp" Sep 17 10:19:24.810566 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:19:24.810542 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7-online-tls\") pod \"feast-simple-feast-setup-7899657bdc-hqsrp\" (UID: \"7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7\") " pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-hqsrp" Sep 17 10:19:24.811059 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:19:24.811036 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7-offline-tls\") pod \"feast-simple-feast-setup-7899657bdc-hqsrp\" (UID: \"7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7\") " pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-hqsrp" Sep 17 10:19:24.819391 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:19:24.819371 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hr9zt\" (UniqueName: \"kubernetes.io/projected/7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7-kube-api-access-hr9zt\") pod \"feast-simple-feast-setup-7899657bdc-hqsrp\" (UID: \"7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7\") " pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-hqsrp" Sep 17 10:19:25.137247 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:19:25.137215 2581 generic.go:356] "Generic (PLEG): container finished" podID="f27c99ca-6c37-4de3-bb57-e0ed2cab0e14" containerID="fea0c75bf16433402585081186269b372071d21ec90e52fdf019bf6dd0b5f3e9" exitCode=0 Sep 17 10:19:25.137247 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:19:25.137238 2581 generic.go:356] "Generic (PLEG): container finished" podID="f27c99ca-6c37-4de3-bb57-e0ed2cab0e14" containerID="b8ed4c8223befc38f3dfa290538153c5afe97fb2e11d6b3547ffa06bf847514d" exitCode=0 Sep 17 10:19:25.137399 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:19:25.137261 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-68q7s" event={"ID":"f27c99ca-6c37-4de3-bb57-e0ed2cab0e14","Type":"ContainerDied","Data":"fea0c75bf16433402585081186269b372071d21ec90e52fdf019bf6dd0b5f3e9"} Sep 17 10:19:25.137399 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:19:25.137291 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-68q7s" event={"ID":"f27c99ca-6c37-4de3-bb57-e0ed2cab0e14","Type":"ContainerDied","Data":"b8ed4c8223befc38f3dfa290538153c5afe97fb2e11d6b3547ffa06bf847514d"} Sep 17 10:19:25.311512 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:19:25.311424 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7-registry-tls\") pod \"feast-simple-feast-setup-7899657bdc-hqsrp\" (UID: \"7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7\") " pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-hqsrp" Sep 17 10:19:25.313788 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:19:25.313759 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7-registry-tls\") pod \"feast-simple-feast-setup-7899657bdc-hqsrp\" (UID: \"7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7\") " pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-hqsrp" Sep 17 10:19:25.528203 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:19:25.528170 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-hqsrp" Sep 17 10:19:25.806671 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:19:25.806645 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-7899657bdc-hqsrp"] Sep 17 10:19:25.807642 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:19:25.807616 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7e10a1f6_18c1_4fab_ae59_bdcacf4ffba7.slice/crio-bf1a3f0cff0db61bf998e8725e5f1ea316728c3889dc7ecab4c31c77ce7643cb WatchSource:0}: Error finding container bf1a3f0cff0db61bf998e8725e5f1ea316728c3889dc7ecab4c31c77ce7643cb: Status 404 returned error can't find the container with id bf1a3f0cff0db61bf998e8725e5f1ea316728c3889dc7ecab4c31c77ce7643cb Sep 17 10:19:26.144616 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:19:26.144522 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-hqsrp" event={"ID":"7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7","Type":"ContainerStarted","Data":"578bffd0279f5b36cffe0e30361e009922da309f2ca00700a7335cb44b9f4cdb"} Sep 17 10:19:26.144616 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:19:26.144576 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-hqsrp" event={"ID":"7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7","Type":"ContainerStarted","Data":"bf1a3f0cff0db61bf998e8725e5f1ea316728c3889dc7ecab4c31c77ce7643cb"} Sep 17 10:19:28.880487 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:19:28.880436 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-68q7s" podUID="f27c99ca-6c37-4de3-bb57-e0ed2cab0e14" containerName="online" probeResult="failure" output="Get \"https://10.134.0.25:6567/health\": dial tcp 10.134.0.25:6567: connect: connection refused" Sep 17 10:19:31.168310 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:19:31.168273 2581 generic.go:356] "Generic (PLEG): container finished" podID="7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7" containerID="578bffd0279f5b36cffe0e30361e009922da309f2ca00700a7335cb44b9f4cdb" exitCode=0 Sep 17 10:19:31.168684 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:19:31.168346 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-hqsrp" event={"ID":"7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7","Type":"ContainerDied","Data":"578bffd0279f5b36cffe0e30361e009922da309f2ca00700a7335cb44b9f4cdb"} Sep 17 10:19:31.892385 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:19:31.892342 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-68q7s" podUID="f27c99ca-6c37-4de3-bb57-e0ed2cab0e14" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.25:8443: connect: connection refused" Sep 17 10:19:32.174194 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:19:32.174114 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-hqsrp" event={"ID":"7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7","Type":"ContainerStarted","Data":"99c285ee3ed8161b981b5d92a8f6d11a01ea2d2102391ff66f9f0e68199c3b00"} Sep 17 10:19:37.195212 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:19:37.195179 2581 generic.go:356] "Generic (PLEG): container finished" podID="7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7" containerID="99c285ee3ed8161b981b5d92a8f6d11a01ea2d2102391ff66f9f0e68199c3b00" exitCode=0 Sep 17 10:19:37.195575 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:19:37.195247 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-hqsrp" event={"ID":"7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7","Type":"ContainerDied","Data":"99c285ee3ed8161b981b5d92a8f6d11a01ea2d2102391ff66f9f0e68199c3b00"} Sep 17 10:19:38.205823 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:19:38.205711 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-hqsrp" event={"ID":"7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7","Type":"ContainerStarted","Data":"2ef6de94729047a8a977a531e5d6429eb69784b33f3e297f3ac60aaaa891d569"} Sep 17 10:19:38.205823 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:19:38.205773 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-hqsrp" event={"ID":"7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7","Type":"ContainerStarted","Data":"4cfc9c280a15f49bcdc3ffa354031afbcd439b042cdfef4270f557ead88f6141"} Sep 17 10:19:38.205823 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:19:38.205815 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-hqsrp" event={"ID":"7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7","Type":"ContainerStarted","Data":"8c9d9489ae7db7ad8e25da6c0ff64446c22d85a7dd5a3c5a299a8028183e80ec"} Sep 17 10:19:38.206405 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:19:38.205833 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-hqsrp" event={"ID":"7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7","Type":"ContainerStarted","Data":"0ba4a2e418a199c3250d5478785cc4e048e043dc86c1acd9e9522e622d831151"} Sep 17 10:19:38.233200 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:19:38.233135 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-hqsrp" podStartSLOduration=14.233112813 podStartE2EDuration="14.233112813s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 10:19:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 10:19:38.230221626 +0000 UTC m=+532.778778740" watchObservedRunningTime="2026-09-17 10:19:38.233112813 +0000 UTC m=+532.781669925" Sep 17 10:19:38.879889 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:19:38.879843 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-68q7s" podUID="f27c99ca-6c37-4de3-bb57-e0ed2cab0e14" containerName="online" probeResult="failure" output="Get \"https://10.134.0.25:6567/health\": dial tcp 10.134.0.25:6567: connect: connection refused" Sep 17 10:19:40.528949 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:19:40.528902 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-hqsrp" Sep 17 10:19:40.529526 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:19:40.529500 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-hqsrp" Sep 17 10:19:40.529526 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:19:40.529534 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-hqsrp" Sep 17 10:19:40.529722 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:19:40.529555 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-hqsrp" Sep 17 10:19:40.531137 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:19:40.531098 2581 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-hqsrp" podUID="7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.27:8816: connect: connection refused" Sep 17 10:19:40.531137 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:19:40.531112 2581 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-hqsrp" podUID="7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.27:6571: connect: connection refused" Sep 17 10:19:40.531325 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:19:40.531154 2581 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-hqsrp" podUID="7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.27:8443: connect: connection refused" Sep 17 10:19:40.531325 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:19:40.531279 2581 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-hqsrp" podUID="7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7" containerName="online" probeResult="failure" output="Get \"https://10.134.0.27:6567/health\": dial tcp 10.134.0.27:6567: connect: connection refused" Sep 17 10:19:41.892091 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:19:41.892038 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-68q7s" podUID="f27c99ca-6c37-4de3-bb57-e0ed2cab0e14" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.25:8443: connect: connection refused" Sep 17 10:19:43.529460 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:19:43.529412 2581 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-hqsrp" podUID="7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.27:8816: connect: connection refused" Sep 17 10:19:43.530142 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:19:43.529432 2581 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-hqsrp" podUID="7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.27:8443: connect: connection refused" Sep 17 10:19:43.530142 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:19:43.529438 2581 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-hqsrp" podUID="7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.27:6571: connect: connection refused" Sep 17 10:19:43.530142 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:19:43.529489 2581 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-hqsrp" podUID="7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7" containerName="online" probeResult="failure" output="Get \"https://10.134.0.27:6567/health\": dial tcp 10.134.0.27:6567: connect: connection refused" Sep 17 10:19:45.528707 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:19:45.528668 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-hqsrp" Sep 17 10:19:45.529112 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:19:45.528731 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-hqsrp" Sep 17 10:19:45.529112 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:19:45.528747 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-hqsrp" Sep 17 10:19:45.529112 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:19:45.528760 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-hqsrp" Sep 17 10:19:46.530120 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:19:46.530085 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-hqsrp" Sep 17 10:19:46.530587 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:19:46.530218 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-hqsrp" Sep 17 10:19:46.530587 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:19:46.530478 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-hqsrp" Sep 17 10:19:46.531089 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:19:46.531058 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-hqsrp" Sep 17 10:19:46.531250 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:19:46.531232 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-hqsrp" Sep 17 10:19:46.531375 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:19:46.531287 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-hqsrp" Sep 17 10:19:46.535772 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:19:46.535746 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-hqsrp" Sep 17 10:19:47.258841 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:19:47.258815 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-hqsrp" Sep 17 10:19:48.880383 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:19:48.880340 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-68q7s" podUID="f27c99ca-6c37-4de3-bb57-e0ed2cab0e14" containerName="online" probeResult="failure" output="Get \"https://10.134.0.25:6567/health\": dial tcp 10.134.0.25:6567: connect: connection refused" Sep 17 10:19:48.880746 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:19:48.880478 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-68q7s" Sep 17 10:19:51.892395 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:19:51.892349 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-68q7s" podUID="f27c99ca-6c37-4de3-bb57-e0ed2cab0e14" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.25:8443: connect: connection refused" Sep 17 10:19:51.892842 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:19:51.892497 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-68q7s" Sep 17 10:19:54.287355 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:19:54.287325 2581 generic.go:356] "Generic (PLEG): container finished" podID="f27c99ca-6c37-4de3-bb57-e0ed2cab0e14" containerID="e7b70d84ec1a0baa6acb85e3c76ae2f6b412ad19b0c91ab29c715617ec8e5141" exitCode=137 Sep 17 10:19:54.287355 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:19:54.287349 2581 generic.go:356] "Generic (PLEG): container finished" podID="f27c99ca-6c37-4de3-bb57-e0ed2cab0e14" containerID="1afe3d8222e9c58133d3373f500e8f4e21e430bdaa4d583eab4b66ccc57cc8dc" exitCode=137 Sep 17 10:19:54.287729 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:19:54.287392 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-68q7s" event={"ID":"f27c99ca-6c37-4de3-bb57-e0ed2cab0e14","Type":"ContainerDied","Data":"e7b70d84ec1a0baa6acb85e3c76ae2f6b412ad19b0c91ab29c715617ec8e5141"} Sep 17 10:19:54.287729 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:19:54.287433 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-68q7s" event={"ID":"f27c99ca-6c37-4de3-bb57-e0ed2cab0e14","Type":"ContainerDied","Data":"1afe3d8222e9c58133d3373f500e8f4e21e430bdaa4d583eab4b66ccc57cc8dc"} Sep 17 10:19:54.846758 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:19:54.846733 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-68q7s" Sep 17 10:19:54.972770 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:19:54.972736 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rzjzx\" (UniqueName: \"kubernetes.io/projected/f27c99ca-6c37-4de3-bb57-e0ed2cab0e14-kube-api-access-rzjzx\") pod \"f27c99ca-6c37-4de3-bb57-e0ed2cab0e14\" (UID: \"f27c99ca-6c37-4de3-bb57-e0ed2cab0e14\") " Sep 17 10:19:54.973094 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:19:54.972799 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/f27c99ca-6c37-4de3-bb57-e0ed2cab0e14-ui-tls\") pod \"f27c99ca-6c37-4de3-bb57-e0ed2cab0e14\" (UID: \"f27c99ca-6c37-4de3-bb57-e0ed2cab0e14\") " Sep 17 10:19:54.973094 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:19:54.972823 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/f27c99ca-6c37-4de3-bb57-e0ed2cab0e14-feast-data\") pod \"f27c99ca-6c37-4de3-bb57-e0ed2cab0e14\" (UID: \"f27c99ca-6c37-4de3-bb57-e0ed2cab0e14\") " Sep 17 10:19:54.973094 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:19:54.973016 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/f27c99ca-6c37-4de3-bb57-e0ed2cab0e14-offline-tls\") pod \"f27c99ca-6c37-4de3-bb57-e0ed2cab0e14\" (UID: \"f27c99ca-6c37-4de3-bb57-e0ed2cab0e14\") " Sep 17 10:19:54.973094 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:19:54.973066 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/f27c99ca-6c37-4de3-bb57-e0ed2cab0e14-registry-tls\") pod \"f27c99ca-6c37-4de3-bb57-e0ed2cab0e14\" (UID: \"f27c99ca-6c37-4de3-bb57-e0ed2cab0e14\") " Sep 17 10:19:54.973311 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:19:54.973125 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/f27c99ca-6c37-4de3-bb57-e0ed2cab0e14-online-tls\") pod \"f27c99ca-6c37-4de3-bb57-e0ed2cab0e14\" (UID: \"f27c99ca-6c37-4de3-bb57-e0ed2cab0e14\") " Sep 17 10:19:54.973357 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:19:54.973304 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f27c99ca-6c37-4de3-bb57-e0ed2cab0e14-feast-data" (OuterVolumeSpecName: "feast-data") pod "f27c99ca-6c37-4de3-bb57-e0ed2cab0e14" (UID: "f27c99ca-6c37-4de3-bb57-e0ed2cab0e14"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 10:19:54.973430 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:19:54.973411 2581 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/f27c99ca-6c37-4de3-bb57-e0ed2cab0e14-feast-data\") on node \"ip-10-0-137-203.ec2.internal\" DevicePath \"\"" Sep 17 10:19:54.975190 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:19:54.975165 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f27c99ca-6c37-4de3-bb57-e0ed2cab0e14-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "f27c99ca-6c37-4de3-bb57-e0ed2cab0e14" (UID: "f27c99ca-6c37-4de3-bb57-e0ed2cab0e14"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 10:19:54.975275 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:19:54.975249 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f27c99ca-6c37-4de3-bb57-e0ed2cab0e14-online-tls" (OuterVolumeSpecName: "online-tls") pod "f27c99ca-6c37-4de3-bb57-e0ed2cab0e14" (UID: "f27c99ca-6c37-4de3-bb57-e0ed2cab0e14"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 10:19:54.975319 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:19:54.975265 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f27c99ca-6c37-4de3-bb57-e0ed2cab0e14-kube-api-access-rzjzx" (OuterVolumeSpecName: "kube-api-access-rzjzx") pod "f27c99ca-6c37-4de3-bb57-e0ed2cab0e14" (UID: "f27c99ca-6c37-4de3-bb57-e0ed2cab0e14"). InnerVolumeSpecName "kube-api-access-rzjzx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 10:19:54.975401 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:19:54.975381 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f27c99ca-6c37-4de3-bb57-e0ed2cab0e14-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "f27c99ca-6c37-4de3-bb57-e0ed2cab0e14" (UID: "f27c99ca-6c37-4de3-bb57-e0ed2cab0e14"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 10:19:54.975454 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:19:54.975432 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f27c99ca-6c37-4de3-bb57-e0ed2cab0e14-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "f27c99ca-6c37-4de3-bb57-e0ed2cab0e14" (UID: "f27c99ca-6c37-4de3-bb57-e0ed2cab0e14"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 10:19:55.074066 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:19:55.074023 2581 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/f27c99ca-6c37-4de3-bb57-e0ed2cab0e14-offline-tls\") on node \"ip-10-0-137-203.ec2.internal\" DevicePath \"\"" Sep 17 10:19:55.074066 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:19:55.074055 2581 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/f27c99ca-6c37-4de3-bb57-e0ed2cab0e14-registry-tls\") on node \"ip-10-0-137-203.ec2.internal\" DevicePath \"\"" Sep 17 10:19:55.074066 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:19:55.074064 2581 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/f27c99ca-6c37-4de3-bb57-e0ed2cab0e14-online-tls\") on node \"ip-10-0-137-203.ec2.internal\" DevicePath \"\"" Sep 17 10:19:55.074066 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:19:55.074073 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzjzx\" (UniqueName: \"kubernetes.io/projected/f27c99ca-6c37-4de3-bb57-e0ed2cab0e14-kube-api-access-rzjzx\") on node \"ip-10-0-137-203.ec2.internal\" DevicePath \"\"" Sep 17 10:19:55.074412 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:19:55.074084 2581 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/f27c99ca-6c37-4de3-bb57-e0ed2cab0e14-ui-tls\") on node \"ip-10-0-137-203.ec2.internal\" DevicePath \"\"" Sep 17 10:19:55.294890 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:19:55.294852 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-68q7s" event={"ID":"f27c99ca-6c37-4de3-bb57-e0ed2cab0e14","Type":"ContainerDied","Data":"34cbba3b12cf91bec9291956b60d39f1701190357968e525783dbcc94d3da939"} Sep 17 10:19:55.295317 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:19:55.294907 2581 scope.go:117] "RemoveContainer" containerID="fea0c75bf16433402585081186269b372071d21ec90e52fdf019bf6dd0b5f3e9" Sep 17 10:19:55.295317 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:19:55.294911 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-68q7s" Sep 17 10:19:55.303668 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:19:55.303647 2581 scope.go:117] "RemoveContainer" containerID="e7b70d84ec1a0baa6acb85e3c76ae2f6b412ad19b0c91ab29c715617ec8e5141" Sep 17 10:19:55.311042 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:19:55.311025 2581 scope.go:117] "RemoveContainer" containerID="b8ed4c8223befc38f3dfa290538153c5afe97fb2e11d6b3547ffa06bf847514d" Sep 17 10:19:55.318290 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:19:55.318213 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-7899657bdc-68q7s"] Sep 17 10:19:55.319530 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:19:55.319446 2581 scope.go:117] "RemoveContainer" containerID="1afe3d8222e9c58133d3373f500e8f4e21e430bdaa4d583eab4b66ccc57cc8dc" Sep 17 10:19:55.322595 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:19:55.322574 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-7899657bdc-68q7s"] Sep 17 10:19:55.327490 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:19:55.327476 2581 scope.go:117] "RemoveContainer" containerID="bef3a4294c147ea2b16eb231114afb2c457a1917fab86a17948d39b0359bf0b0" Sep 17 10:19:55.334413 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:19:55.334396 2581 scope.go:117] "RemoveContainer" containerID="f96a872a7b6f6d17a7ee820a0d04e576c401fff2c3f432f73c1b1400a523b7cf" Sep 17 10:19:55.945168 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:19:55.945130 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f27c99ca-6c37-4de3-bb57-e0ed2cab0e14" path="/var/lib/kubelet/pods/f27c99ca-6c37-4de3-bb57-e0ed2cab0e14/volumes" Sep 17 10:20:45.855861 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:20:45.855824 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-b9wnf_762d3e10-7b7d-4ce9-9773-90a4fe41b474/console-operator/2.log" Sep 17 10:20:45.864304 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:20:45.864285 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-b9wnf_762d3e10-7b7d-4ce9-9773-90a4fe41b474/console-operator/2.log" Sep 17 10:20:49.693745 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:20:49.693709 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-64c5997d6f-drqht"] Sep 17 10:20:49.694127 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:20:49.694023 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f27c99ca-6c37-4de3-bb57-e0ed2cab0e14" containerName="ui" Sep 17 10:20:49.694127 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:20:49.694034 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f27c99ca-6c37-4de3-bb57-e0ed2cab0e14" containerName="ui" Sep 17 10:20:49.694127 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:20:49.694045 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f27c99ca-6c37-4de3-bb57-e0ed2cab0e14" containerName="registry" Sep 17 10:20:49.694127 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:20:49.694050 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f27c99ca-6c37-4de3-bb57-e0ed2cab0e14" containerName="registry" Sep 17 10:20:49.694127 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:20:49.694060 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f27c99ca-6c37-4de3-bb57-e0ed2cab0e14" containerName="feast-init" Sep 17 10:20:49.694127 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:20:49.694066 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f27c99ca-6c37-4de3-bb57-e0ed2cab0e14" containerName="feast-init" Sep 17 10:20:49.694127 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:20:49.694072 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f27c99ca-6c37-4de3-bb57-e0ed2cab0e14" containerName="offline" Sep 17 10:20:49.694127 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:20:49.694077 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f27c99ca-6c37-4de3-bb57-e0ed2cab0e14" containerName="offline" Sep 17 10:20:49.694127 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:20:49.694093 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f27c99ca-6c37-4de3-bb57-e0ed2cab0e14" containerName="feast-apply" Sep 17 10:20:49.694127 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:20:49.694098 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f27c99ca-6c37-4de3-bb57-e0ed2cab0e14" containerName="feast-apply" Sep 17 10:20:49.694127 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:20:49.694105 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f27c99ca-6c37-4de3-bb57-e0ed2cab0e14" containerName="online" Sep 17 10:20:49.694127 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:20:49.694110 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f27c99ca-6c37-4de3-bb57-e0ed2cab0e14" containerName="online" Sep 17 10:20:49.694505 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:20:49.694153 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f27c99ca-6c37-4de3-bb57-e0ed2cab0e14" containerName="online" Sep 17 10:20:49.694505 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:20:49.694162 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f27c99ca-6c37-4de3-bb57-e0ed2cab0e14" containerName="offline" Sep 17 10:20:49.694505 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:20:49.694169 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f27c99ca-6c37-4de3-bb57-e0ed2cab0e14" containerName="registry" Sep 17 10:20:49.694505 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:20:49.694175 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f27c99ca-6c37-4de3-bb57-e0ed2cab0e14" containerName="ui" Sep 17 10:20:49.698563 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:20:49.698546 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-64c5997d6f-drqht" Sep 17 10:20:49.701402 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:20:49.701320 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-offline-tls\"" Sep 17 10:20:49.701402 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:20:49.701334 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-intra-comm\"" Sep 17 10:20:49.701402 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:20:49.701387 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-online-tls\"" Sep 17 10:20:49.701402 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:20:49.701396 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-client-ca\"" Sep 17 10:20:49.701402 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:20:49.701398 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"openshift-service-ca.crt\"" Sep 17 10:20:49.701685 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:20:49.701411 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-dockercfg-lwcs6\"" Sep 17 10:20:49.701685 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:20:49.701631 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-ui-tls\"" Sep 17 10:20:49.701766 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:20:49.701751 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"kube-root-ca.crt\"" Sep 17 10:20:49.708130 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:20:49.708109 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-64c5997d6f-drqht"] Sep 17 10:20:49.787674 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:20:49.787635 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/f25b7469-ba0c-4359-b965-e96df8ed1830-ui-tls\") pod \"feast-simple-feast-remote-setup-64c5997d6f-drqht\" (UID: \"f25b7469-ba0c-4359-b965-e96df8ed1830\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-64c5997d6f-drqht" Sep 17 10:20:49.787674 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:20:49.787678 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/f25b7469-ba0c-4359-b965-e96df8ed1830-registry-tls\") pod \"feast-simple-feast-remote-setup-64c5997d6f-drqht\" (UID: \"f25b7469-ba0c-4359-b965-e96df8ed1830\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-64c5997d6f-drqht" Sep 17 10:20:49.787926 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:20:49.787698 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/f25b7469-ba0c-4359-b965-e96df8ed1830-offline-tls\") pod \"feast-simple-feast-remote-setup-64c5997d6f-drqht\" (UID: \"f25b7469-ba0c-4359-b965-e96df8ed1830\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-64c5997d6f-drqht" Sep 17 10:20:49.787926 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:20:49.787760 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/f25b7469-ba0c-4359-b965-e96df8ed1830-online-tls\") pod \"feast-simple-feast-remote-setup-64c5997d6f-drqht\" (UID: \"f25b7469-ba0c-4359-b965-e96df8ed1830\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-64c5997d6f-drqht" Sep 17 10:20:49.787926 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:20:49.787850 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/f25b7469-ba0c-4359-b965-e96df8ed1830-feast-data\") pod \"feast-simple-feast-remote-setup-64c5997d6f-drqht\" (UID: \"f25b7469-ba0c-4359-b965-e96df8ed1830\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-64c5997d6f-drqht" Sep 17 10:20:49.787926 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:20:49.787888 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8vjk\" (UniqueName: \"kubernetes.io/projected/f25b7469-ba0c-4359-b965-e96df8ed1830-kube-api-access-r8vjk\") pod \"feast-simple-feast-remote-setup-64c5997d6f-drqht\" (UID: \"f25b7469-ba0c-4359-b965-e96df8ed1830\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-64c5997d6f-drqht" Sep 17 10:20:49.889110 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:20:49.889067 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/f25b7469-ba0c-4359-b965-e96df8ed1830-ui-tls\") pod \"feast-simple-feast-remote-setup-64c5997d6f-drqht\" (UID: \"f25b7469-ba0c-4359-b965-e96df8ed1830\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-64c5997d6f-drqht" Sep 17 10:20:49.889305 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:20:49.889117 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/f25b7469-ba0c-4359-b965-e96df8ed1830-registry-tls\") pod \"feast-simple-feast-remote-setup-64c5997d6f-drqht\" (UID: \"f25b7469-ba0c-4359-b965-e96df8ed1830\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-64c5997d6f-drqht" Sep 17 10:20:49.889305 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:20:49.889146 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/f25b7469-ba0c-4359-b965-e96df8ed1830-offline-tls\") pod \"feast-simple-feast-remote-setup-64c5997d6f-drqht\" (UID: \"f25b7469-ba0c-4359-b965-e96df8ed1830\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-64c5997d6f-drqht" Sep 17 10:20:49.889305 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:20:49.889167 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/f25b7469-ba0c-4359-b965-e96df8ed1830-online-tls\") pod \"feast-simple-feast-remote-setup-64c5997d6f-drqht\" (UID: \"f25b7469-ba0c-4359-b965-e96df8ed1830\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-64c5997d6f-drqht" Sep 17 10:20:49.889305 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:20:49.889201 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/f25b7469-ba0c-4359-b965-e96df8ed1830-feast-data\") pod \"feast-simple-feast-remote-setup-64c5997d6f-drqht\" (UID: \"f25b7469-ba0c-4359-b965-e96df8ed1830\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-64c5997d6f-drqht" Sep 17 10:20:49.889305 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:20:49.889213 2581 secret.go:189] Couldn't get secret test-ns-remote-registry/feast-simple-feast-remote-setup-ui-tls: secret "feast-simple-feast-remote-setup-ui-tls" not found Sep 17 10:20:49.889305 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:20:49.889250 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8vjk\" (UniqueName: \"kubernetes.io/projected/f25b7469-ba0c-4359-b965-e96df8ed1830-kube-api-access-r8vjk\") pod \"feast-simple-feast-remote-setup-64c5997d6f-drqht\" (UID: \"f25b7469-ba0c-4359-b965-e96df8ed1830\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-64c5997d6f-drqht" Sep 17 10:20:49.889305 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:20:49.889281 2581 secret.go:189] Couldn't get secret test-ns-remote-registry/feast-simple-feast-remote-setup-offline-tls: secret "feast-simple-feast-remote-setup-offline-tls" not found Sep 17 10:20:49.889305 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:20:49.889296 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f25b7469-ba0c-4359-b965-e96df8ed1830-ui-tls podName:f25b7469-ba0c-4359-b965-e96df8ed1830 nodeName:}" failed. No retries permitted until 2026-09-17 10:20:50.389274839 +0000 UTC m=+604.937831943 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "ui-tls" (UniqueName: "kubernetes.io/secret/f25b7469-ba0c-4359-b965-e96df8ed1830-ui-tls") pod "feast-simple-feast-remote-setup-64c5997d6f-drqht" (UID: "f25b7469-ba0c-4359-b965-e96df8ed1830") : secret "feast-simple-feast-remote-setup-ui-tls" not found Sep 17 10:20:49.889686 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:20:49.889339 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f25b7469-ba0c-4359-b965-e96df8ed1830-offline-tls podName:f25b7469-ba0c-4359-b965-e96df8ed1830 nodeName:}" failed. No retries permitted until 2026-09-17 10:20:50.389321156 +0000 UTC m=+604.937878246 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/f25b7469-ba0c-4359-b965-e96df8ed1830-offline-tls") pod "feast-simple-feast-remote-setup-64c5997d6f-drqht" (UID: "f25b7469-ba0c-4359-b965-e96df8ed1830") : secret "feast-simple-feast-remote-setup-offline-tls" not found Sep 17 10:20:49.889728 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:20:49.889695 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/f25b7469-ba0c-4359-b965-e96df8ed1830-feast-data\") pod \"feast-simple-feast-remote-setup-64c5997d6f-drqht\" (UID: \"f25b7469-ba0c-4359-b965-e96df8ed1830\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-64c5997d6f-drqht" Sep 17 10:20:49.889904 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:20:49.889886 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/f25b7469-ba0c-4359-b965-e96df8ed1830-registry-tls\") pod \"feast-simple-feast-remote-setup-64c5997d6f-drqht\" (UID: \"f25b7469-ba0c-4359-b965-e96df8ed1830\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-64c5997d6f-drqht" Sep 17 10:20:49.891662 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:20:49.891645 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/f25b7469-ba0c-4359-b965-e96df8ed1830-online-tls\") pod \"feast-simple-feast-remote-setup-64c5997d6f-drqht\" (UID: \"f25b7469-ba0c-4359-b965-e96df8ed1830\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-64c5997d6f-drqht" Sep 17 10:20:49.899064 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:20:49.899042 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8vjk\" (UniqueName: \"kubernetes.io/projected/f25b7469-ba0c-4359-b965-e96df8ed1830-kube-api-access-r8vjk\") pod \"feast-simple-feast-remote-setup-64c5997d6f-drqht\" (UID: \"f25b7469-ba0c-4359-b965-e96df8ed1830\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-64c5997d6f-drqht" Sep 17 10:20:50.392511 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:20:50.392475 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/f25b7469-ba0c-4359-b965-e96df8ed1830-ui-tls\") pod \"feast-simple-feast-remote-setup-64c5997d6f-drqht\" (UID: \"f25b7469-ba0c-4359-b965-e96df8ed1830\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-64c5997d6f-drqht" Sep 17 10:20:50.392680 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:20:50.392520 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/f25b7469-ba0c-4359-b965-e96df8ed1830-offline-tls\") pod \"feast-simple-feast-remote-setup-64c5997d6f-drqht\" (UID: \"f25b7469-ba0c-4359-b965-e96df8ed1830\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-64c5997d6f-drqht" Sep 17 10:20:50.394995 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:20:50.394965 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/f25b7469-ba0c-4359-b965-e96df8ed1830-ui-tls\") pod \"feast-simple-feast-remote-setup-64c5997d6f-drqht\" (UID: \"f25b7469-ba0c-4359-b965-e96df8ed1830\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-64c5997d6f-drqht" Sep 17 10:20:50.395105 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:20:50.395004 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/f25b7469-ba0c-4359-b965-e96df8ed1830-offline-tls\") pod \"feast-simple-feast-remote-setup-64c5997d6f-drqht\" (UID: \"f25b7469-ba0c-4359-b965-e96df8ed1830\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-64c5997d6f-drqht" Sep 17 10:20:50.608849 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:20:50.608815 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-64c5997d6f-drqht" Sep 17 10:20:50.883753 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:20:50.883720 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-64c5997d6f-drqht"] Sep 17 10:20:50.885054 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:20:50.885033 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf25b7469_ba0c_4359_b965_e96df8ed1830.slice/crio-036a4d5a78d0cf2998a499f3c7fbf84bc46b132a612b356e2cbe04fc756ded92 WatchSource:0}: Error finding container 036a4d5a78d0cf2998a499f3c7fbf84bc46b132a612b356e2cbe04fc756ded92: Status 404 returned error can't find the container with id 036a4d5a78d0cf2998a499f3c7fbf84bc46b132a612b356e2cbe04fc756ded92 Sep 17 10:20:51.518655 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:20:51.518615 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-64c5997d6f-drqht" event={"ID":"f25b7469-ba0c-4359-b965-e96df8ed1830","Type":"ContainerStarted","Data":"64510e71fe9d6f9e5e1d2bc23a43f7a7d84428cf393f590871fb29b192512762"} Sep 17 10:20:51.518862 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:20:51.518668 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-64c5997d6f-drqht" event={"ID":"f25b7469-ba0c-4359-b965-e96df8ed1830","Type":"ContainerStarted","Data":"036a4d5a78d0cf2998a499f3c7fbf84bc46b132a612b356e2cbe04fc756ded92"} Sep 17 10:20:56.539030 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:20:56.538998 2581 generic.go:356] "Generic (PLEG): container finished" podID="f25b7469-ba0c-4359-b965-e96df8ed1830" containerID="64510e71fe9d6f9e5e1d2bc23a43f7a7d84428cf393f590871fb29b192512762" exitCode=0 Sep 17 10:20:56.539443 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:20:56.539036 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-64c5997d6f-drqht" event={"ID":"f25b7469-ba0c-4359-b965-e96df8ed1830","Type":"ContainerDied","Data":"64510e71fe9d6f9e5e1d2bc23a43f7a7d84428cf393f590871fb29b192512762"} Sep 17 10:20:57.545242 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:20:57.545211 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-64c5997d6f-drqht" event={"ID":"f25b7469-ba0c-4359-b965-e96df8ed1830","Type":"ContainerStarted","Data":"c722d7aa3758f8faef54ef28ff30eb21dd4523c6456229b87db37f4722113fb8"} Sep 17 10:21:18.631093 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:21:18.631048 2581 generic.go:356] "Generic (PLEG): container finished" podID="f25b7469-ba0c-4359-b965-e96df8ed1830" containerID="c722d7aa3758f8faef54ef28ff30eb21dd4523c6456229b87db37f4722113fb8" exitCode=0 Sep 17 10:21:18.631512 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:21:18.631129 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-64c5997d6f-drqht" event={"ID":"f25b7469-ba0c-4359-b965-e96df8ed1830","Type":"ContainerDied","Data":"c722d7aa3758f8faef54ef28ff30eb21dd4523c6456229b87db37f4722113fb8"} Sep 17 10:21:19.640475 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:21:19.640438 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-64c5997d6f-drqht" event={"ID":"f25b7469-ba0c-4359-b965-e96df8ed1830","Type":"ContainerStarted","Data":"a4d378190937813479b68abe166e786b87bb994de3bce55b336dcbb11a13f09d"} Sep 17 10:21:19.640475 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:21:19.640489 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-64c5997d6f-drqht" event={"ID":"f25b7469-ba0c-4359-b965-e96df8ed1830","Type":"ContainerStarted","Data":"955a29a3f5c91e519956aa85737e75d52a05ecfe2f629b405f1218155591f475"} Sep 17 10:21:19.641061 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:21:19.640506 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-64c5997d6f-drqht" event={"ID":"f25b7469-ba0c-4359-b965-e96df8ed1830","Type":"ContainerStarted","Data":"1d9e6a0b904d4f23a306ef983582d0affed2162fc870f05b008c14ed497c9324"} Sep 17 10:21:19.667364 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:21:19.667303 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-64c5997d6f-drqht" podStartSLOduration=30.667285514 podStartE2EDuration="30.667285514s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 10:20:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 10:21:19.666515883 +0000 UTC m=+634.215073006" watchObservedRunningTime="2026-09-17 10:21:19.667285514 +0000 UTC m=+634.215842626" Sep 17 10:21:20.609372 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:21:20.609241 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-64c5997d6f-drqht" Sep 17 10:21:20.609372 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:21:20.609309 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-64c5997d6f-drqht" Sep 17 10:21:20.609372 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:21:20.609328 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-64c5997d6f-drqht" Sep 17 10:21:20.609372 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:21:20.609346 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-64c5997d6f-drqht" Sep 17 10:21:20.609372 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:21:20.609363 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-64c5997d6f-drqht" Sep 17 10:21:20.609372 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:21:20.609376 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-64c5997d6f-drqht" Sep 17 10:21:20.611022 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:21:20.610983 2581 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-64c5997d6f-drqht" podUID="f25b7469-ba0c-4359-b965-e96df8ed1830" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.28:8443: connect: connection refused" Sep 17 10:21:20.611160 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:21:20.611032 2581 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-64c5997d6f-drqht" podUID="f25b7469-ba0c-4359-b965-e96df8ed1830" containerName="online" probeResult="failure" output="Get \"https://10.134.0.28:6567/health\": dial tcp 10.134.0.28:6567: connect: connection refused" Sep 17 10:21:20.611160 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:21:20.610990 2581 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-64c5997d6f-drqht" podUID="f25b7469-ba0c-4359-b965-e96df8ed1830" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.28:8816: connect: connection refused" Sep 17 10:21:23.609553 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:21:23.609503 2581 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-64c5997d6f-drqht" podUID="f25b7469-ba0c-4359-b965-e96df8ed1830" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.28:8443: connect: connection refused" Sep 17 10:21:23.609981 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:21:23.609503 2581 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-64c5997d6f-drqht" podUID="f25b7469-ba0c-4359-b965-e96df8ed1830" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.28:8816: connect: connection refused" Sep 17 10:21:23.609981 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:21:23.609606 2581 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-64c5997d6f-drqht" podUID="f25b7469-ba0c-4359-b965-e96df8ed1830" containerName="online" probeResult="failure" output="Get \"https://10.134.0.28:6567/health\": dial tcp 10.134.0.28:6567: connect: connection refused" Sep 17 10:21:26.609483 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:21:26.609429 2581 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-64c5997d6f-drqht" podUID="f25b7469-ba0c-4359-b965-e96df8ed1830" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.28:8443: connect: connection refused" Sep 17 10:21:26.610076 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:21:26.610051 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-64c5997d6f-drqht" Sep 17 10:21:26.610917 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:21:26.610895 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-64c5997d6f-drqht" Sep 17 10:21:26.699601 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:21:26.699572 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-64c5997d6f-drqht" Sep 17 10:21:27.710917 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:21:27.710887 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-64c5997d6f-drqht" Sep 17 10:21:29.609654 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:21:29.609620 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-64c5997d6f-drqht" Sep 17 10:21:29.697975 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:21:29.697942 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-64c5997d6f-drqht" Sep 17 10:22:31.895282 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:22:31.895246 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-64c5997d6f-drqht"] Sep 17 10:22:31.895851 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:22:31.895532 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-64c5997d6f-drqht" podUID="f25b7469-ba0c-4359-b965-e96df8ed1830" containerName="online" containerID="cri-o://1d9e6a0b904d4f23a306ef983582d0affed2162fc870f05b008c14ed497c9324" gracePeriod=30 Sep 17 10:22:31.895851 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:22:31.895702 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-64c5997d6f-drqht" podUID="f25b7469-ba0c-4359-b965-e96df8ed1830" containerName="offline" containerID="cri-o://955a29a3f5c91e519956aa85737e75d52a05ecfe2f629b405f1218155591f475" gracePeriod=30 Sep 17 10:22:31.895851 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:22:31.895696 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-64c5997d6f-drqht" podUID="f25b7469-ba0c-4359-b965-e96df8ed1830" containerName="ui" containerID="cri-o://a4d378190937813479b68abe166e786b87bb994de3bce55b336dcbb11a13f09d" gracePeriod=30 Sep 17 10:22:32.045995 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:22:32.045959 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-7899657bdc-hqsrp"] Sep 17 10:22:32.047712 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:22:32.046814 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-hqsrp" podUID="7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7" containerName="registry" containerID="cri-o://0ba4a2e418a199c3250d5478785cc4e048e043dc86c1acd9e9522e622d831151" gracePeriod=30 Sep 17 10:22:32.047712 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:22:32.047367 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-hqsrp" podUID="7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7" containerName="ui" containerID="cri-o://2ef6de94729047a8a977a531e5d6429eb69784b33f3e297f3ac60aaaa891d569" gracePeriod=30 Sep 17 10:22:32.047712 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:22:32.047413 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-hqsrp" podUID="7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7" containerName="online" containerID="cri-o://8c9d9489ae7db7ad8e25da6c0ff64446c22d85a7dd5a3c5a299a8028183e80ec" gracePeriod=30 Sep 17 10:22:32.047712 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:22:32.047612 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-hqsrp" podUID="7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7" containerName="offline" containerID="cri-o://4cfc9c280a15f49bcdc3ffa354031afbcd439b042cdfef4270f557ead88f6141" gracePeriod=30 Sep 17 10:22:32.959869 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:22:32.959833 2581 generic.go:356] "Generic (PLEG): container finished" podID="f25b7469-ba0c-4359-b965-e96df8ed1830" containerID="a4d378190937813479b68abe166e786b87bb994de3bce55b336dcbb11a13f09d" exitCode=0 Sep 17 10:22:32.959869 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:22:32.959860 2581 generic.go:356] "Generic (PLEG): container finished" podID="f25b7469-ba0c-4359-b965-e96df8ed1830" containerID="1d9e6a0b904d4f23a306ef983582d0affed2162fc870f05b008c14ed497c9324" exitCode=0 Sep 17 10:22:32.960305 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:22:32.959915 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-64c5997d6f-drqht" event={"ID":"f25b7469-ba0c-4359-b965-e96df8ed1830","Type":"ContainerDied","Data":"a4d378190937813479b68abe166e786b87bb994de3bce55b336dcbb11a13f09d"} Sep 17 10:22:32.960305 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:22:32.959968 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-64c5997d6f-drqht" event={"ID":"f25b7469-ba0c-4359-b965-e96df8ed1830","Type":"ContainerDied","Data":"1d9e6a0b904d4f23a306ef983582d0affed2162fc870f05b008c14ed497c9324"} Sep 17 10:22:32.962773 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:22:32.962750 2581 generic.go:356] "Generic (PLEG): container finished" podID="7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7" containerID="2ef6de94729047a8a977a531e5d6429eb69784b33f3e297f3ac60aaaa891d569" exitCode=0 Sep 17 10:22:32.962773 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:22:32.962769 2581 generic.go:356] "Generic (PLEG): container finished" podID="7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7" containerID="8c9d9489ae7db7ad8e25da6c0ff64446c22d85a7dd5a3c5a299a8028183e80ec" exitCode=0 Sep 17 10:22:32.962934 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:22:32.962801 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-hqsrp" event={"ID":"7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7","Type":"ContainerDied","Data":"2ef6de94729047a8a977a531e5d6429eb69784b33f3e297f3ac60aaaa891d569"} Sep 17 10:22:32.962934 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:22:32.962836 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-hqsrp" event={"ID":"7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7","Type":"ContainerDied","Data":"8c9d9489ae7db7ad8e25da6c0ff64446c22d85a7dd5a3c5a299a8028183e80ec"} Sep 17 10:22:36.531942 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:22:36.531894 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-hqsrp" podUID="7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.27:8443: connect: connection refused" Sep 17 10:22:37.256626 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:22:37.256582 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-hqsrp" podUID="7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7" containerName="online" probeResult="failure" output="Get \"https://10.134.0.27:6567/health\": dial tcp 10.134.0.27:6567: connect: connection refused" Sep 17 10:22:37.690096 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:22:37.690013 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-64c5997d6f-drqht" podUID="f25b7469-ba0c-4359-b965-e96df8ed1830" containerName="online" probeResult="failure" output="Get \"https://10.134.0.28:6567/health\": dial tcp 10.134.0.28:6567: connect: connection refused" Sep 17 10:22:39.698194 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:22:39.698112 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-64c5997d6f-drqht" podUID="f25b7469-ba0c-4359-b965-e96df8ed1830" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.28:8443: connect: connection refused" Sep 17 10:22:46.531622 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:22:46.531571 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-hqsrp" podUID="7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.27:8443: connect: connection refused" Sep 17 10:22:47.256495 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:22:47.256445 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-hqsrp" podUID="7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7" containerName="online" probeResult="failure" output="Get \"https://10.134.0.27:6567/health\": dial tcp 10.134.0.27:6567: connect: connection refused" Sep 17 10:22:47.690888 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:22:47.690769 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-64c5997d6f-drqht" podUID="f25b7469-ba0c-4359-b965-e96df8ed1830" containerName="online" probeResult="failure" output="Get \"https://10.134.0.28:6567/health\": dial tcp 10.134.0.28:6567: connect: connection refused" Sep 17 10:22:49.698407 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:22:49.698361 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-64c5997d6f-drqht" podUID="f25b7469-ba0c-4359-b965-e96df8ed1830" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.28:8443: connect: connection refused" Sep 17 10:22:56.531353 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:22:56.531294 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-hqsrp" podUID="7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.27:8443: connect: connection refused" Sep 17 10:22:56.531902 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:22:56.531461 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-hqsrp" Sep 17 10:22:57.256794 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:22:57.256741 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-hqsrp" podUID="7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7" containerName="online" probeResult="failure" output="Get \"https://10.134.0.27:6567/health\": dial tcp 10.134.0.27:6567: connect: connection refused" Sep 17 10:22:57.256985 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:22:57.256897 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-hqsrp" Sep 17 10:22:57.690263 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:22:57.690186 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-64c5997d6f-drqht" podUID="f25b7469-ba0c-4359-b965-e96df8ed1830" containerName="online" probeResult="failure" output="Get \"https://10.134.0.28:6567/health\": dial tcp 10.134.0.28:6567: connect: connection refused" Sep 17 10:22:57.690576 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:22:57.690291 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-64c5997d6f-drqht" Sep 17 10:22:59.697881 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:22:59.697839 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-64c5997d6f-drqht" podUID="f25b7469-ba0c-4359-b965-e96df8ed1830" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.28:8443: connect: connection refused" Sep 17 10:22:59.698249 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:22:59.697951 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-64c5997d6f-drqht" Sep 17 10:23:02.087395 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:02.087364 2581 generic.go:356] "Generic (PLEG): container finished" podID="f25b7469-ba0c-4359-b965-e96df8ed1830" containerID="955a29a3f5c91e519956aa85737e75d52a05ecfe2f629b405f1218155591f475" exitCode=137 Sep 17 10:23:02.087716 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:02.087438 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-64c5997d6f-drqht" event={"ID":"f25b7469-ba0c-4359-b965-e96df8ed1830","Type":"ContainerDied","Data":"955a29a3f5c91e519956aa85737e75d52a05ecfe2f629b405f1218155591f475"} Sep 17 10:23:02.542301 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:02.542272 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-64c5997d6f-drqht" Sep 17 10:23:02.684722 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:02.684690 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r8vjk\" (UniqueName: \"kubernetes.io/projected/f25b7469-ba0c-4359-b965-e96df8ed1830-kube-api-access-r8vjk\") pod \"f25b7469-ba0c-4359-b965-e96df8ed1830\" (UID: \"f25b7469-ba0c-4359-b965-e96df8ed1830\") " Sep 17 10:23:02.684879 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:02.684741 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/f25b7469-ba0c-4359-b965-e96df8ed1830-ui-tls\") pod \"f25b7469-ba0c-4359-b965-e96df8ed1830\" (UID: \"f25b7469-ba0c-4359-b965-e96df8ed1830\") " Sep 17 10:23:02.684879 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:02.684799 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/f25b7469-ba0c-4359-b965-e96df8ed1830-registry-tls\") pod \"f25b7469-ba0c-4359-b965-e96df8ed1830\" (UID: \"f25b7469-ba0c-4359-b965-e96df8ed1830\") " Sep 17 10:23:02.684879 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:02.684834 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/f25b7469-ba0c-4359-b965-e96df8ed1830-offline-tls\") pod \"f25b7469-ba0c-4359-b965-e96df8ed1830\" (UID: \"f25b7469-ba0c-4359-b965-e96df8ed1830\") " Sep 17 10:23:02.685049 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:02.684884 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/f25b7469-ba0c-4359-b965-e96df8ed1830-online-tls\") pod \"f25b7469-ba0c-4359-b965-e96df8ed1830\" (UID: \"f25b7469-ba0c-4359-b965-e96df8ed1830\") " Sep 17 10:23:02.685049 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:02.684907 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/f25b7469-ba0c-4359-b965-e96df8ed1830-feast-data\") pod \"f25b7469-ba0c-4359-b965-e96df8ed1830\" (UID: \"f25b7469-ba0c-4359-b965-e96df8ed1830\") " Sep 17 10:23:02.685268 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:02.685220 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f25b7469-ba0c-4359-b965-e96df8ed1830-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "f25b7469-ba0c-4359-b965-e96df8ed1830" (UID: "f25b7469-ba0c-4359-b965-e96df8ed1830"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 10:23:02.685663 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:02.685621 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f25b7469-ba0c-4359-b965-e96df8ed1830-feast-data" (OuterVolumeSpecName: "feast-data") pod "f25b7469-ba0c-4359-b965-e96df8ed1830" (UID: "f25b7469-ba0c-4359-b965-e96df8ed1830"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 10:23:02.687100 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:02.687074 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f25b7469-ba0c-4359-b965-e96df8ed1830-kube-api-access-r8vjk" (OuterVolumeSpecName: "kube-api-access-r8vjk") pod "f25b7469-ba0c-4359-b965-e96df8ed1830" (UID: "f25b7469-ba0c-4359-b965-e96df8ed1830"). InnerVolumeSpecName "kube-api-access-r8vjk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 10:23:02.687240 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:02.687218 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f25b7469-ba0c-4359-b965-e96df8ed1830-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "f25b7469-ba0c-4359-b965-e96df8ed1830" (UID: "f25b7469-ba0c-4359-b965-e96df8ed1830"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 10:23:02.687335 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:02.687312 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f25b7469-ba0c-4359-b965-e96df8ed1830-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "f25b7469-ba0c-4359-b965-e96df8ed1830" (UID: "f25b7469-ba0c-4359-b965-e96df8ed1830"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 10:23:02.687442 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:02.687325 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f25b7469-ba0c-4359-b965-e96df8ed1830-online-tls" (OuterVolumeSpecName: "online-tls") pod "f25b7469-ba0c-4359-b965-e96df8ed1830" (UID: "f25b7469-ba0c-4359-b965-e96df8ed1830"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 10:23:02.691958 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:02.691939 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-hqsrp" Sep 17 10:23:02.786225 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:02.786193 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7-online-tls\") pod \"7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7\" (UID: \"7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7\") " Sep 17 10:23:02.786405 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:02.786253 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7-ui-tls\") pod \"7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7\" (UID: \"7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7\") " Sep 17 10:23:02.786405 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:02.786279 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7-registry-tls\") pod \"7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7\" (UID: \"7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7\") " Sep 17 10:23:02.786405 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:02.786296 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7-offline-tls\") pod \"7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7\" (UID: \"7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7\") " Sep 17 10:23:02.786405 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:02.786349 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7-feast-data\") pod \"7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7\" (UID: \"7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7\") " Sep 17 10:23:02.786405 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:02.786380 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hr9zt\" (UniqueName: \"kubernetes.io/projected/7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7-kube-api-access-hr9zt\") pod \"7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7\" (UID: \"7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7\") " Sep 17 10:23:02.786611 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:02.786599 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8vjk\" (UniqueName: \"kubernetes.io/projected/f25b7469-ba0c-4359-b965-e96df8ed1830-kube-api-access-r8vjk\") on node \"ip-10-0-137-203.ec2.internal\" DevicePath \"\"" Sep 17 10:23:02.786664 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:02.786618 2581 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/f25b7469-ba0c-4359-b965-e96df8ed1830-ui-tls\") on node \"ip-10-0-137-203.ec2.internal\" DevicePath \"\"" Sep 17 10:23:02.786664 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:02.786628 2581 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/f25b7469-ba0c-4359-b965-e96df8ed1830-registry-tls\") on node \"ip-10-0-137-203.ec2.internal\" DevicePath \"\"" Sep 17 10:23:02.786664 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:02.786637 2581 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/f25b7469-ba0c-4359-b965-e96df8ed1830-offline-tls\") on node \"ip-10-0-137-203.ec2.internal\" DevicePath \"\"" Sep 17 10:23:02.786664 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:02.786645 2581 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/f25b7469-ba0c-4359-b965-e96df8ed1830-online-tls\") on node \"ip-10-0-137-203.ec2.internal\" DevicePath \"\"" Sep 17 10:23:02.786664 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:02.786652 2581 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/f25b7469-ba0c-4359-b965-e96df8ed1830-feast-data\") on node \"ip-10-0-137-203.ec2.internal\" DevicePath \"\"" Sep 17 10:23:02.787281 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:02.787252 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7-feast-data" (OuterVolumeSpecName: "feast-data") pod "7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7" (UID: "7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 10:23:02.788724 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:02.788695 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7" (UID: "7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 10:23:02.788842 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:02.788749 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7-online-tls" (OuterVolumeSpecName: "online-tls") pod "7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7" (UID: "7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 10:23:02.789049 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:02.789026 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7-kube-api-access-hr9zt" (OuterVolumeSpecName: "kube-api-access-hr9zt") pod "7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7" (UID: "7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7"). InnerVolumeSpecName "kube-api-access-hr9zt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 10:23:02.789049 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:02.789042 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7" (UID: "7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 10:23:02.789186 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:02.789158 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7" (UID: "7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 10:23:02.887724 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:02.887673 2581 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7-feast-data\") on node \"ip-10-0-137-203.ec2.internal\" DevicePath \"\"" Sep 17 10:23:02.887724 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:02.887718 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hr9zt\" (UniqueName: \"kubernetes.io/projected/7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7-kube-api-access-hr9zt\") on node \"ip-10-0-137-203.ec2.internal\" DevicePath \"\"" Sep 17 10:23:02.887724 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:02.887729 2581 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7-online-tls\") on node \"ip-10-0-137-203.ec2.internal\" DevicePath \"\"" Sep 17 10:23:02.887724 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:02.887738 2581 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7-ui-tls\") on node \"ip-10-0-137-203.ec2.internal\" DevicePath \"\"" Sep 17 10:23:02.888002 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:02.887746 2581 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7-registry-tls\") on node \"ip-10-0-137-203.ec2.internal\" DevicePath \"\"" Sep 17 10:23:02.888002 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:02.887754 2581 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7-offline-tls\") on node \"ip-10-0-137-203.ec2.internal\" DevicePath \"\"" Sep 17 10:23:03.093733 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:03.093706 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-64c5997d6f-drqht" Sep 17 10:23:03.094198 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:03.093738 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-64c5997d6f-drqht" event={"ID":"f25b7469-ba0c-4359-b965-e96df8ed1830","Type":"ContainerDied","Data":"036a4d5a78d0cf2998a499f3c7fbf84bc46b132a612b356e2cbe04fc756ded92"} Sep 17 10:23:03.094198 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:03.093803 2581 scope.go:117] "RemoveContainer" containerID="a4d378190937813479b68abe166e786b87bb994de3bce55b336dcbb11a13f09d" Sep 17 10:23:03.097113 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:03.097085 2581 generic.go:356] "Generic (PLEG): container finished" podID="7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7" containerID="4cfc9c280a15f49bcdc3ffa354031afbcd439b042cdfef4270f557ead88f6141" exitCode=137 Sep 17 10:23:03.097113 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:03.097112 2581 generic.go:356] "Generic (PLEG): container finished" podID="7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7" containerID="0ba4a2e418a199c3250d5478785cc4e048e043dc86c1acd9e9522e622d831151" exitCode=137 Sep 17 10:23:03.097242 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:03.097131 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-hqsrp" event={"ID":"7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7","Type":"ContainerDied","Data":"4cfc9c280a15f49bcdc3ffa354031afbcd439b042cdfef4270f557ead88f6141"} Sep 17 10:23:03.097242 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:03.097145 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-hqsrp" Sep 17 10:23:03.097242 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:03.097172 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-hqsrp" event={"ID":"7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7","Type":"ContainerDied","Data":"0ba4a2e418a199c3250d5478785cc4e048e043dc86c1acd9e9522e622d831151"} Sep 17 10:23:03.097242 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:03.097193 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7899657bdc-hqsrp" event={"ID":"7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7","Type":"ContainerDied","Data":"bf1a3f0cff0db61bf998e8725e5f1ea316728c3889dc7ecab4c31c77ce7643cb"} Sep 17 10:23:03.104704 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:03.104618 2581 scope.go:117] "RemoveContainer" containerID="955a29a3f5c91e519956aa85737e75d52a05ecfe2f629b405f1218155591f475" Sep 17 10:23:03.113029 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:03.113013 2581 scope.go:117] "RemoveContainer" containerID="1d9e6a0b904d4f23a306ef983582d0affed2162fc870f05b008c14ed497c9324" Sep 17 10:23:03.120126 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:03.120109 2581 scope.go:117] "RemoveContainer" containerID="c722d7aa3758f8faef54ef28ff30eb21dd4523c6456229b87db37f4722113fb8" Sep 17 10:23:03.123233 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:03.123211 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-64c5997d6f-drqht"] Sep 17 10:23:03.127081 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:03.127058 2581 scope.go:117] "RemoveContainer" containerID="64510e71fe9d6f9e5e1d2bc23a43f7a7d84428cf393f590871fb29b192512762" Sep 17 10:23:03.130413 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:03.130392 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-64c5997d6f-drqht"] Sep 17 10:23:03.134320 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:03.134304 2581 scope.go:117] "RemoveContainer" containerID="2ef6de94729047a8a977a531e5d6429eb69784b33f3e297f3ac60aaaa891d569" Sep 17 10:23:03.140887 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:03.140871 2581 scope.go:117] "RemoveContainer" containerID="4cfc9c280a15f49bcdc3ffa354031afbcd439b042cdfef4270f557ead88f6141" Sep 17 10:23:03.148325 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:03.148304 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-7899657bdc-hqsrp"] Sep 17 10:23:03.148581 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:03.148558 2581 scope.go:117] "RemoveContainer" containerID="8c9d9489ae7db7ad8e25da6c0ff64446c22d85a7dd5a3c5a299a8028183e80ec" Sep 17 10:23:03.156515 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:03.156416 2581 scope.go:117] "RemoveContainer" containerID="0ba4a2e418a199c3250d5478785cc4e048e043dc86c1acd9e9522e622d831151" Sep 17 10:23:03.157899 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:03.157881 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-7899657bdc-hqsrp"] Sep 17 10:23:03.163336 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:03.163319 2581 scope.go:117] "RemoveContainer" containerID="99c285ee3ed8161b981b5d92a8f6d11a01ea2d2102391ff66f9f0e68199c3b00" Sep 17 10:23:03.170150 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:03.170136 2581 scope.go:117] "RemoveContainer" containerID="578bffd0279f5b36cffe0e30361e009922da309f2ca00700a7335cb44b9f4cdb" Sep 17 10:23:03.176688 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:03.176672 2581 scope.go:117] "RemoveContainer" containerID="2ef6de94729047a8a977a531e5d6429eb69784b33f3e297f3ac60aaaa891d569" Sep 17 10:23:03.177045 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:23:03.177025 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ef6de94729047a8a977a531e5d6429eb69784b33f3e297f3ac60aaaa891d569\": container with ID starting with 2ef6de94729047a8a977a531e5d6429eb69784b33f3e297f3ac60aaaa891d569 not found: ID does not exist" containerID="2ef6de94729047a8a977a531e5d6429eb69784b33f3e297f3ac60aaaa891d569" Sep 17 10:23:03.177100 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:03.177053 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ef6de94729047a8a977a531e5d6429eb69784b33f3e297f3ac60aaaa891d569"} err="failed to get container status \"2ef6de94729047a8a977a531e5d6429eb69784b33f3e297f3ac60aaaa891d569\": rpc error: code = NotFound desc = could not find container \"2ef6de94729047a8a977a531e5d6429eb69784b33f3e297f3ac60aaaa891d569\": container with ID starting with 2ef6de94729047a8a977a531e5d6429eb69784b33f3e297f3ac60aaaa891d569 not found: ID does not exist" Sep 17 10:23:03.177100 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:03.177071 2581 scope.go:117] "RemoveContainer" containerID="4cfc9c280a15f49bcdc3ffa354031afbcd439b042cdfef4270f557ead88f6141" Sep 17 10:23:03.177303 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:23:03.177289 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4cfc9c280a15f49bcdc3ffa354031afbcd439b042cdfef4270f557ead88f6141\": container with ID starting with 4cfc9c280a15f49bcdc3ffa354031afbcd439b042cdfef4270f557ead88f6141 not found: ID does not exist" containerID="4cfc9c280a15f49bcdc3ffa354031afbcd439b042cdfef4270f557ead88f6141" Sep 17 10:23:03.177347 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:03.177307 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4cfc9c280a15f49bcdc3ffa354031afbcd439b042cdfef4270f557ead88f6141"} err="failed to get container status \"4cfc9c280a15f49bcdc3ffa354031afbcd439b042cdfef4270f557ead88f6141\": rpc error: code = NotFound desc = could not find container \"4cfc9c280a15f49bcdc3ffa354031afbcd439b042cdfef4270f557ead88f6141\": container with ID starting with 4cfc9c280a15f49bcdc3ffa354031afbcd439b042cdfef4270f557ead88f6141 not found: ID does not exist" Sep 17 10:23:03.177347 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:03.177320 2581 scope.go:117] "RemoveContainer" containerID="8c9d9489ae7db7ad8e25da6c0ff64446c22d85a7dd5a3c5a299a8028183e80ec" Sep 17 10:23:03.177536 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:23:03.177517 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c9d9489ae7db7ad8e25da6c0ff64446c22d85a7dd5a3c5a299a8028183e80ec\": container with ID starting with 8c9d9489ae7db7ad8e25da6c0ff64446c22d85a7dd5a3c5a299a8028183e80ec not found: ID does not exist" containerID="8c9d9489ae7db7ad8e25da6c0ff64446c22d85a7dd5a3c5a299a8028183e80ec" Sep 17 10:23:03.177577 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:03.177543 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c9d9489ae7db7ad8e25da6c0ff64446c22d85a7dd5a3c5a299a8028183e80ec"} err="failed to get container status \"8c9d9489ae7db7ad8e25da6c0ff64446c22d85a7dd5a3c5a299a8028183e80ec\": rpc error: code = NotFound desc = could not find container \"8c9d9489ae7db7ad8e25da6c0ff64446c22d85a7dd5a3c5a299a8028183e80ec\": container with ID starting with 8c9d9489ae7db7ad8e25da6c0ff64446c22d85a7dd5a3c5a299a8028183e80ec not found: ID does not exist" Sep 17 10:23:03.177577 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:03.177560 2581 scope.go:117] "RemoveContainer" containerID="0ba4a2e418a199c3250d5478785cc4e048e043dc86c1acd9e9522e622d831151" Sep 17 10:23:03.177827 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:23:03.177799 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ba4a2e418a199c3250d5478785cc4e048e043dc86c1acd9e9522e622d831151\": container with ID starting with 0ba4a2e418a199c3250d5478785cc4e048e043dc86c1acd9e9522e622d831151 not found: ID does not exist" containerID="0ba4a2e418a199c3250d5478785cc4e048e043dc86c1acd9e9522e622d831151" Sep 17 10:23:03.177951 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:03.177828 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ba4a2e418a199c3250d5478785cc4e048e043dc86c1acd9e9522e622d831151"} err="failed to get container status \"0ba4a2e418a199c3250d5478785cc4e048e043dc86c1acd9e9522e622d831151\": rpc error: code = NotFound desc = could not find container \"0ba4a2e418a199c3250d5478785cc4e048e043dc86c1acd9e9522e622d831151\": container with ID starting with 0ba4a2e418a199c3250d5478785cc4e048e043dc86c1acd9e9522e622d831151 not found: ID does not exist" Sep 17 10:23:03.177951 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:03.177842 2581 scope.go:117] "RemoveContainer" containerID="99c285ee3ed8161b981b5d92a8f6d11a01ea2d2102391ff66f9f0e68199c3b00" Sep 17 10:23:03.178084 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:23:03.178067 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99c285ee3ed8161b981b5d92a8f6d11a01ea2d2102391ff66f9f0e68199c3b00\": container with ID starting with 99c285ee3ed8161b981b5d92a8f6d11a01ea2d2102391ff66f9f0e68199c3b00 not found: ID does not exist" containerID="99c285ee3ed8161b981b5d92a8f6d11a01ea2d2102391ff66f9f0e68199c3b00" Sep 17 10:23:03.178121 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:03.178090 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99c285ee3ed8161b981b5d92a8f6d11a01ea2d2102391ff66f9f0e68199c3b00"} err="failed to get container status \"99c285ee3ed8161b981b5d92a8f6d11a01ea2d2102391ff66f9f0e68199c3b00\": rpc error: code = NotFound desc = could not find container \"99c285ee3ed8161b981b5d92a8f6d11a01ea2d2102391ff66f9f0e68199c3b00\": container with ID starting with 99c285ee3ed8161b981b5d92a8f6d11a01ea2d2102391ff66f9f0e68199c3b00 not found: ID does not exist" Sep 17 10:23:03.178121 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:03.178105 2581 scope.go:117] "RemoveContainer" containerID="578bffd0279f5b36cffe0e30361e009922da309f2ca00700a7335cb44b9f4cdb" Sep 17 10:23:03.178326 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:23:03.178310 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"578bffd0279f5b36cffe0e30361e009922da309f2ca00700a7335cb44b9f4cdb\": container with ID starting with 578bffd0279f5b36cffe0e30361e009922da309f2ca00700a7335cb44b9f4cdb not found: ID does not exist" containerID="578bffd0279f5b36cffe0e30361e009922da309f2ca00700a7335cb44b9f4cdb" Sep 17 10:23:03.178378 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:03.178329 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"578bffd0279f5b36cffe0e30361e009922da309f2ca00700a7335cb44b9f4cdb"} err="failed to get container status \"578bffd0279f5b36cffe0e30361e009922da309f2ca00700a7335cb44b9f4cdb\": rpc error: code = NotFound desc = could not find container \"578bffd0279f5b36cffe0e30361e009922da309f2ca00700a7335cb44b9f4cdb\": container with ID starting with 578bffd0279f5b36cffe0e30361e009922da309f2ca00700a7335cb44b9f4cdb not found: ID does not exist" Sep 17 10:23:03.178378 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:03.178343 2581 scope.go:117] "RemoveContainer" containerID="2ef6de94729047a8a977a531e5d6429eb69784b33f3e297f3ac60aaaa891d569" Sep 17 10:23:03.178550 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:03.178534 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ef6de94729047a8a977a531e5d6429eb69784b33f3e297f3ac60aaaa891d569"} err="failed to get container status \"2ef6de94729047a8a977a531e5d6429eb69784b33f3e297f3ac60aaaa891d569\": rpc error: code = NotFound desc = could not find container \"2ef6de94729047a8a977a531e5d6429eb69784b33f3e297f3ac60aaaa891d569\": container with ID starting with 2ef6de94729047a8a977a531e5d6429eb69784b33f3e297f3ac60aaaa891d569 not found: ID does not exist" Sep 17 10:23:03.178590 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:03.178550 2581 scope.go:117] "RemoveContainer" containerID="4cfc9c280a15f49bcdc3ffa354031afbcd439b042cdfef4270f557ead88f6141" Sep 17 10:23:03.178803 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:03.178769 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4cfc9c280a15f49bcdc3ffa354031afbcd439b042cdfef4270f557ead88f6141"} err="failed to get container status \"4cfc9c280a15f49bcdc3ffa354031afbcd439b042cdfef4270f557ead88f6141\": rpc error: code = NotFound desc = could not find container \"4cfc9c280a15f49bcdc3ffa354031afbcd439b042cdfef4270f557ead88f6141\": container with ID starting with 4cfc9c280a15f49bcdc3ffa354031afbcd439b042cdfef4270f557ead88f6141 not found: ID does not exist" Sep 17 10:23:03.178891 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:03.178805 2581 scope.go:117] "RemoveContainer" containerID="8c9d9489ae7db7ad8e25da6c0ff64446c22d85a7dd5a3c5a299a8028183e80ec" Sep 17 10:23:03.179040 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:03.179020 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c9d9489ae7db7ad8e25da6c0ff64446c22d85a7dd5a3c5a299a8028183e80ec"} err="failed to get container status \"8c9d9489ae7db7ad8e25da6c0ff64446c22d85a7dd5a3c5a299a8028183e80ec\": rpc error: code = NotFound desc = could not find container \"8c9d9489ae7db7ad8e25da6c0ff64446c22d85a7dd5a3c5a299a8028183e80ec\": container with ID starting with 8c9d9489ae7db7ad8e25da6c0ff64446c22d85a7dd5a3c5a299a8028183e80ec not found: ID does not exist" Sep 17 10:23:03.179096 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:03.179041 2581 scope.go:117] "RemoveContainer" containerID="0ba4a2e418a199c3250d5478785cc4e048e043dc86c1acd9e9522e622d831151" Sep 17 10:23:03.179259 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:03.179243 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ba4a2e418a199c3250d5478785cc4e048e043dc86c1acd9e9522e622d831151"} err="failed to get container status \"0ba4a2e418a199c3250d5478785cc4e048e043dc86c1acd9e9522e622d831151\": rpc error: code = NotFound desc = could not find container \"0ba4a2e418a199c3250d5478785cc4e048e043dc86c1acd9e9522e622d831151\": container with ID starting with 0ba4a2e418a199c3250d5478785cc4e048e043dc86c1acd9e9522e622d831151 not found: ID does not exist" Sep 17 10:23:03.179296 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:03.179259 2581 scope.go:117] "RemoveContainer" containerID="99c285ee3ed8161b981b5d92a8f6d11a01ea2d2102391ff66f9f0e68199c3b00" Sep 17 10:23:03.179457 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:03.179440 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99c285ee3ed8161b981b5d92a8f6d11a01ea2d2102391ff66f9f0e68199c3b00"} err="failed to get container status \"99c285ee3ed8161b981b5d92a8f6d11a01ea2d2102391ff66f9f0e68199c3b00\": rpc error: code = NotFound desc = could not find container \"99c285ee3ed8161b981b5d92a8f6d11a01ea2d2102391ff66f9f0e68199c3b00\": container with ID starting with 99c285ee3ed8161b981b5d92a8f6d11a01ea2d2102391ff66f9f0e68199c3b00 not found: ID does not exist" Sep 17 10:23:03.179457 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:03.179455 2581 scope.go:117] "RemoveContainer" containerID="578bffd0279f5b36cffe0e30361e009922da309f2ca00700a7335cb44b9f4cdb" Sep 17 10:23:03.179660 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:03.179644 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"578bffd0279f5b36cffe0e30361e009922da309f2ca00700a7335cb44b9f4cdb"} err="failed to get container status \"578bffd0279f5b36cffe0e30361e009922da309f2ca00700a7335cb44b9f4cdb\": rpc error: code = NotFound desc = could not find container \"578bffd0279f5b36cffe0e30361e009922da309f2ca00700a7335cb44b9f4cdb\": container with ID starting with 578bffd0279f5b36cffe0e30361e009922da309f2ca00700a7335cb44b9f4cdb not found: ID does not exist" Sep 17 10:23:03.943341 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:03.943303 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7" path="/var/lib/kubelet/pods/7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7/volumes" Sep 17 10:23:03.944019 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:03.943998 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f25b7469-ba0c-4359-b965-e96df8ed1830" path="/var/lib/kubelet/pods/f25b7469-ba0c-4359-b965-e96df8ed1830/volumes" Sep 17 10:23:14.961350 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:14.961313 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/postgres-864846d4f5-b84kl"] Sep 17 10:23:14.961725 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:14.961609 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7" containerName="offline" Sep 17 10:23:14.961725 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:14.961620 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7" containerName="offline" Sep 17 10:23:14.961725 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:14.961628 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f25b7469-ba0c-4359-b965-e96df8ed1830" containerName="offline" Sep 17 10:23:14.961725 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:14.961634 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f25b7469-ba0c-4359-b965-e96df8ed1830" containerName="offline" Sep 17 10:23:14.961725 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:14.961646 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f25b7469-ba0c-4359-b965-e96df8ed1830" containerName="feast-apply" Sep 17 10:23:14.961725 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:14.961652 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f25b7469-ba0c-4359-b965-e96df8ed1830" containerName="feast-apply" Sep 17 10:23:14.961725 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:14.961660 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f25b7469-ba0c-4359-b965-e96df8ed1830" containerName="ui" Sep 17 10:23:14.961725 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:14.961665 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f25b7469-ba0c-4359-b965-e96df8ed1830" containerName="ui" Sep 17 10:23:14.961725 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:14.961671 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f25b7469-ba0c-4359-b965-e96df8ed1830" containerName="feast-init" Sep 17 10:23:14.961725 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:14.961676 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f25b7469-ba0c-4359-b965-e96df8ed1830" containerName="feast-init" Sep 17 10:23:14.961725 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:14.961685 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7" containerName="feast-apply" Sep 17 10:23:14.961725 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:14.961690 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7" containerName="feast-apply" Sep 17 10:23:14.961725 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:14.961694 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7" containerName="registry" Sep 17 10:23:14.961725 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:14.961699 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7" containerName="registry" Sep 17 10:23:14.961725 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:14.961710 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7" containerName="ui" Sep 17 10:23:14.961725 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:14.961714 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7" containerName="ui" Sep 17 10:23:14.961725 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:14.961722 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7" containerName="feast-init" Sep 17 10:23:14.961725 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:14.961727 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7" containerName="feast-init" Sep 17 10:23:14.961725 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:14.961732 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7" containerName="online" Sep 17 10:23:14.962286 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:14.961738 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7" containerName="online" Sep 17 10:23:14.962286 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:14.961743 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f25b7469-ba0c-4359-b965-e96df8ed1830" containerName="online" Sep 17 10:23:14.962286 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:14.961748 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f25b7469-ba0c-4359-b965-e96df8ed1830" containerName="online" Sep 17 10:23:14.962286 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:14.961816 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7" containerName="ui" Sep 17 10:23:14.962286 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:14.961823 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f25b7469-ba0c-4359-b965-e96df8ed1830" containerName="offline" Sep 17 10:23:14.962286 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:14.961828 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7" containerName="online" Sep 17 10:23:14.962286 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:14.961834 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7" containerName="offline" Sep 17 10:23:14.962286 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:14.961841 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e10a1f6-18c1-4fab-ae59-bdcacf4ffba7" containerName="registry" Sep 17 10:23:14.962286 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:14.961846 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f25b7469-ba0c-4359-b965-e96df8ed1830" containerName="online" Sep 17 10:23:14.962286 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:14.961850 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f25b7469-ba0c-4359-b965-e96df8ed1830" containerName="ui" Sep 17 10:23:14.964667 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:14.964644 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-864846d4f5-b84kl" Sep 17 10:23:14.967247 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:14.967210 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"postgres-secret\"" Sep 17 10:23:14.967397 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:14.967328 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"default-dockercfg-t7t7t\"" Sep 17 10:23:14.967397 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:14.967380 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\"" Sep 17 10:23:14.968399 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:14.968379 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\"" Sep 17 10:23:14.971654 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:14.971626 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/postgres-864846d4f5-b84kl"] Sep 17 10:23:15.084515 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:15.084474 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2gtc\" (UniqueName: \"kubernetes.io/projected/c7293908-2b25-4082-95ae-16f444683300-kube-api-access-m2gtc\") pod \"postgres-864846d4f5-b84kl\" (UID: \"c7293908-2b25-4082-95ae-16f444683300\") " pod="test-ns-feast/postgres-864846d4f5-b84kl" Sep 17 10:23:15.084724 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:15.084641 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/c7293908-2b25-4082-95ae-16f444683300-postgresdata\") pod \"postgres-864846d4f5-b84kl\" (UID: \"c7293908-2b25-4082-95ae-16f444683300\") " pod="test-ns-feast/postgres-864846d4f5-b84kl" Sep 17 10:23:15.185304 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:15.185262 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/c7293908-2b25-4082-95ae-16f444683300-postgresdata\") pod \"postgres-864846d4f5-b84kl\" (UID: \"c7293908-2b25-4082-95ae-16f444683300\") " pod="test-ns-feast/postgres-864846d4f5-b84kl" Sep 17 10:23:15.185475 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:15.185326 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2gtc\" (UniqueName: \"kubernetes.io/projected/c7293908-2b25-4082-95ae-16f444683300-kube-api-access-m2gtc\") pod \"postgres-864846d4f5-b84kl\" (UID: \"c7293908-2b25-4082-95ae-16f444683300\") " pod="test-ns-feast/postgres-864846d4f5-b84kl" Sep 17 10:23:15.185653 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:15.185629 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/c7293908-2b25-4082-95ae-16f444683300-postgresdata\") pod \"postgres-864846d4f5-b84kl\" (UID: \"c7293908-2b25-4082-95ae-16f444683300\") " pod="test-ns-feast/postgres-864846d4f5-b84kl" Sep 17 10:23:15.195747 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:15.195716 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2gtc\" (UniqueName: \"kubernetes.io/projected/c7293908-2b25-4082-95ae-16f444683300-kube-api-access-m2gtc\") pod \"postgres-864846d4f5-b84kl\" (UID: \"c7293908-2b25-4082-95ae-16f444683300\") " pod="test-ns-feast/postgres-864846d4f5-b84kl" Sep 17 10:23:15.275567 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:15.275517 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-864846d4f5-b84kl" Sep 17 10:23:15.547887 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:15.547863 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/postgres-864846d4f5-b84kl"] Sep 17 10:23:15.550283 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:23:15.550251 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc7293908_2b25_4082_95ae_16f444683300.slice/crio-5ee925e6c6dfc54695a4757da870f0388fe254b60928ca909c990a6b5e460033 WatchSource:0}: Error finding container 5ee925e6c6dfc54695a4757da870f0388fe254b60928ca909c990a6b5e460033: Status 404 returned error can't find the container with id 5ee925e6c6dfc54695a4757da870f0388fe254b60928ca909c990a6b5e460033 Sep 17 10:23:15.552594 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:15.552572 2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 17 10:23:16.149058 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:16.149022 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-864846d4f5-b84kl" event={"ID":"c7293908-2b25-4082-95ae-16f444683300","Type":"ContainerStarted","Data":"5ee925e6c6dfc54695a4757da870f0388fe254b60928ca909c990a6b5e460033"} Sep 17 10:23:22.176755 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:22.176718 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-864846d4f5-b84kl" event={"ID":"c7293908-2b25-4082-95ae-16f444683300","Type":"ContainerStarted","Data":"2842e042a265358f82f4b96a388591251092bd8ab3ff3038da09867940459798"} Sep 17 10:23:22.194326 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:22.194277 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/postgres-864846d4f5-b84kl" podStartSLOduration=2.539409744 podStartE2EDuration="8.194263092s" totalImagesPullingTime="5.654853348s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 10:23:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 10:23:22.19293746 +0000 UTC m=+756.741494572" watchObservedRunningTime="2026-09-17 10:23:22.194263092 +0000 UTC m=+756.742820203" Sep 17 10:23:25.796716 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:25.796683 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-credit-scoring-69f6b6c8cb-2r2bk"] Sep 17 10:23:25.803388 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:25.803364 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-69f6b6c8cb-2r2bk" Sep 17 10:23:25.806299 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:25.806272 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-online-tls\"" Sep 17 10:23:25.806425 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:25.806293 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-offline-tls\"" Sep 17 10:23:25.806425 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:25.806322 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-intra-comm\"" Sep 17 10:23:25.806425 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:25.806284 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-registry-tls\"" Sep 17 10:23:25.806425 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:25.806301 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-dockercfg-72mvf\"" Sep 17 10:23:25.812234 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:25.812212 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-credit-scoring-69f6b6c8cb-2r2bk"] Sep 17 10:23:25.874069 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:25.874026 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/62039600-b028-47bf-8e07-d5a9e1a3ec63-feast-data\") pod \"feast-credit-scoring-69f6b6c8cb-2r2bk\" (UID: \"62039600-b028-47bf-8e07-d5a9e1a3ec63\") " pod="test-ns-feast/feast-credit-scoring-69f6b6c8cb-2r2bk" Sep 17 10:23:25.874069 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:25.874070 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/62039600-b028-47bf-8e07-d5a9e1a3ec63-offline-tls\") pod \"feast-credit-scoring-69f6b6c8cb-2r2bk\" (UID: \"62039600-b028-47bf-8e07-d5a9e1a3ec63\") " pod="test-ns-feast/feast-credit-scoring-69f6b6c8cb-2r2bk" Sep 17 10:23:25.874305 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:25.874092 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/62039600-b028-47bf-8e07-d5a9e1a3ec63-registry-tls\") pod \"feast-credit-scoring-69f6b6c8cb-2r2bk\" (UID: \"62039600-b028-47bf-8e07-d5a9e1a3ec63\") " pod="test-ns-feast/feast-credit-scoring-69f6b6c8cb-2r2bk" Sep 17 10:23:25.874305 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:25.874173 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/62039600-b028-47bf-8e07-d5a9e1a3ec63-online-tls\") pod \"feast-credit-scoring-69f6b6c8cb-2r2bk\" (UID: \"62039600-b028-47bf-8e07-d5a9e1a3ec63\") " pod="test-ns-feast/feast-credit-scoring-69f6b6c8cb-2r2bk" Sep 17 10:23:25.874305 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:25.874209 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fvh5\" (UniqueName: \"kubernetes.io/projected/62039600-b028-47bf-8e07-d5a9e1a3ec63-kube-api-access-4fvh5\") pod \"feast-credit-scoring-69f6b6c8cb-2r2bk\" (UID: \"62039600-b028-47bf-8e07-d5a9e1a3ec63\") " pod="test-ns-feast/feast-credit-scoring-69f6b6c8cb-2r2bk" Sep 17 10:23:25.975197 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:25.975157 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/62039600-b028-47bf-8e07-d5a9e1a3ec63-feast-data\") pod \"feast-credit-scoring-69f6b6c8cb-2r2bk\" (UID: \"62039600-b028-47bf-8e07-d5a9e1a3ec63\") " pod="test-ns-feast/feast-credit-scoring-69f6b6c8cb-2r2bk" Sep 17 10:23:25.975197 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:25.975196 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/62039600-b028-47bf-8e07-d5a9e1a3ec63-offline-tls\") pod \"feast-credit-scoring-69f6b6c8cb-2r2bk\" (UID: \"62039600-b028-47bf-8e07-d5a9e1a3ec63\") " pod="test-ns-feast/feast-credit-scoring-69f6b6c8cb-2r2bk" Sep 17 10:23:25.975405 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:25.975213 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/62039600-b028-47bf-8e07-d5a9e1a3ec63-registry-tls\") pod \"feast-credit-scoring-69f6b6c8cb-2r2bk\" (UID: \"62039600-b028-47bf-8e07-d5a9e1a3ec63\") " pod="test-ns-feast/feast-credit-scoring-69f6b6c8cb-2r2bk" Sep 17 10:23:25.975405 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:23:25.975320 2581 secret.go:189] Couldn't get secret test-ns-feast/feast-credit-scoring-registry-tls: secret "feast-credit-scoring-registry-tls" not found Sep 17 10:23:25.975405 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:25.975352 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/62039600-b028-47bf-8e07-d5a9e1a3ec63-online-tls\") pod \"feast-credit-scoring-69f6b6c8cb-2r2bk\" (UID: \"62039600-b028-47bf-8e07-d5a9e1a3ec63\") " pod="test-ns-feast/feast-credit-scoring-69f6b6c8cb-2r2bk" Sep 17 10:23:25.975405 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:23:25.975373 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62039600-b028-47bf-8e07-d5a9e1a3ec63-registry-tls podName:62039600-b028-47bf-8e07-d5a9e1a3ec63 nodeName:}" failed. No retries permitted until 2026-09-17 10:23:26.475358812 +0000 UTC m=+761.023915916 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/62039600-b028-47bf-8e07-d5a9e1a3ec63-registry-tls") pod "feast-credit-scoring-69f6b6c8cb-2r2bk" (UID: "62039600-b028-47bf-8e07-d5a9e1a3ec63") : secret "feast-credit-scoring-registry-tls" not found Sep 17 10:23:25.975405 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:23:25.975320 2581 secret.go:189] Couldn't get secret test-ns-feast/feast-credit-scoring-offline-tls: secret "feast-credit-scoring-offline-tls" not found Sep 17 10:23:25.975634 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:25.975404 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fvh5\" (UniqueName: \"kubernetes.io/projected/62039600-b028-47bf-8e07-d5a9e1a3ec63-kube-api-access-4fvh5\") pod \"feast-credit-scoring-69f6b6c8cb-2r2bk\" (UID: \"62039600-b028-47bf-8e07-d5a9e1a3ec63\") " pod="test-ns-feast/feast-credit-scoring-69f6b6c8cb-2r2bk" Sep 17 10:23:25.975634 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:23:25.975447 2581 secret.go:189] Couldn't get secret test-ns-feast/feast-credit-scoring-online-tls: secret "feast-credit-scoring-online-tls" not found Sep 17 10:23:25.975634 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:23:25.975471 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62039600-b028-47bf-8e07-d5a9e1a3ec63-offline-tls podName:62039600-b028-47bf-8e07-d5a9e1a3ec63 nodeName:}" failed. No retries permitted until 2026-09-17 10:23:26.475451255 +0000 UTC m=+761.024008355 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/62039600-b028-47bf-8e07-d5a9e1a3ec63-offline-tls") pod "feast-credit-scoring-69f6b6c8cb-2r2bk" (UID: "62039600-b028-47bf-8e07-d5a9e1a3ec63") : secret "feast-credit-scoring-offline-tls" not found Sep 17 10:23:25.975634 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:23:25.975519 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62039600-b028-47bf-8e07-d5a9e1a3ec63-online-tls podName:62039600-b028-47bf-8e07-d5a9e1a3ec63 nodeName:}" failed. No retries permitted until 2026-09-17 10:23:26.47550297 +0000 UTC m=+761.024060061 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/62039600-b028-47bf-8e07-d5a9e1a3ec63-online-tls") pod "feast-credit-scoring-69f6b6c8cb-2r2bk" (UID: "62039600-b028-47bf-8e07-d5a9e1a3ec63") : secret "feast-credit-scoring-online-tls" not found Sep 17 10:23:25.975634 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:25.975590 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/62039600-b028-47bf-8e07-d5a9e1a3ec63-feast-data\") pod \"feast-credit-scoring-69f6b6c8cb-2r2bk\" (UID: \"62039600-b028-47bf-8e07-d5a9e1a3ec63\") " pod="test-ns-feast/feast-credit-scoring-69f6b6c8cb-2r2bk" Sep 17 10:23:25.996084 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:25.996047 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fvh5\" (UniqueName: \"kubernetes.io/projected/62039600-b028-47bf-8e07-d5a9e1a3ec63-kube-api-access-4fvh5\") pod \"feast-credit-scoring-69f6b6c8cb-2r2bk\" (UID: \"62039600-b028-47bf-8e07-d5a9e1a3ec63\") " pod="test-ns-feast/feast-credit-scoring-69f6b6c8cb-2r2bk" Sep 17 10:23:26.480923 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:26.480868 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/62039600-b028-47bf-8e07-d5a9e1a3ec63-offline-tls\") pod \"feast-credit-scoring-69f6b6c8cb-2r2bk\" (UID: \"62039600-b028-47bf-8e07-d5a9e1a3ec63\") " pod="test-ns-feast/feast-credit-scoring-69f6b6c8cb-2r2bk" Sep 17 10:23:26.480923 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:26.480926 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/62039600-b028-47bf-8e07-d5a9e1a3ec63-registry-tls\") pod \"feast-credit-scoring-69f6b6c8cb-2r2bk\" (UID: \"62039600-b028-47bf-8e07-d5a9e1a3ec63\") " pod="test-ns-feast/feast-credit-scoring-69f6b6c8cb-2r2bk" Sep 17 10:23:26.481140 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:26.480959 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/62039600-b028-47bf-8e07-d5a9e1a3ec63-online-tls\") pod \"feast-credit-scoring-69f6b6c8cb-2r2bk\" (UID: \"62039600-b028-47bf-8e07-d5a9e1a3ec63\") " pod="test-ns-feast/feast-credit-scoring-69f6b6c8cb-2r2bk" Sep 17 10:23:26.483483 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:26.483457 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/62039600-b028-47bf-8e07-d5a9e1a3ec63-offline-tls\") pod \"feast-credit-scoring-69f6b6c8cb-2r2bk\" (UID: \"62039600-b028-47bf-8e07-d5a9e1a3ec63\") " pod="test-ns-feast/feast-credit-scoring-69f6b6c8cb-2r2bk" Sep 17 10:23:26.483592 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:26.483507 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/62039600-b028-47bf-8e07-d5a9e1a3ec63-online-tls\") pod \"feast-credit-scoring-69f6b6c8cb-2r2bk\" (UID: \"62039600-b028-47bf-8e07-d5a9e1a3ec63\") " pod="test-ns-feast/feast-credit-scoring-69f6b6c8cb-2r2bk" Sep 17 10:23:26.483592 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:26.483572 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/62039600-b028-47bf-8e07-d5a9e1a3ec63-registry-tls\") pod \"feast-credit-scoring-69f6b6c8cb-2r2bk\" (UID: \"62039600-b028-47bf-8e07-d5a9e1a3ec63\") " pod="test-ns-feast/feast-credit-scoring-69f6b6c8cb-2r2bk" Sep 17 10:23:26.713857 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:26.713823 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-69f6b6c8cb-2r2bk" Sep 17 10:23:27.001142 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:27.001111 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-credit-scoring-69f6b6c8cb-2r2bk"] Sep 17 10:23:27.002088 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:23:27.002053 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62039600_b028_47bf_8e07_d5a9e1a3ec63.slice/crio-d20a685a36f107e91973f8f1fca0d904b55b7e9de2bb20482aa50ad6dbcd55be WatchSource:0}: Error finding container d20a685a36f107e91973f8f1fca0d904b55b7e9de2bb20482aa50ad6dbcd55be: Status 404 returned error can't find the container with id d20a685a36f107e91973f8f1fca0d904b55b7e9de2bb20482aa50ad6dbcd55be Sep 17 10:23:27.197634 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:27.197595 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-69f6b6c8cb-2r2bk" event={"ID":"62039600-b028-47bf-8e07-d5a9e1a3ec63","Type":"ContainerStarted","Data":"326e4dd9845b92fb406324aa8d48c276adb3090a65fcd76b292494ecbf508dcf"} Sep 17 10:23:27.197854 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:27.197649 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-69f6b6c8cb-2r2bk" event={"ID":"62039600-b028-47bf-8e07-d5a9e1a3ec63","Type":"ContainerStarted","Data":"d20a685a36f107e91973f8f1fca0d904b55b7e9de2bb20482aa50ad6dbcd55be"} Sep 17 10:23:28.202642 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:28.202607 2581 generic.go:356] "Generic (PLEG): container finished" podID="62039600-b028-47bf-8e07-d5a9e1a3ec63" containerID="326e4dd9845b92fb406324aa8d48c276adb3090a65fcd76b292494ecbf508dcf" exitCode=0 Sep 17 10:23:28.203095 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:28.202704 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-69f6b6c8cb-2r2bk" event={"ID":"62039600-b028-47bf-8e07-d5a9e1a3ec63","Type":"ContainerDied","Data":"326e4dd9845b92fb406324aa8d48c276adb3090a65fcd76b292494ecbf508dcf"} Sep 17 10:23:29.208538 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:29.208496 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-69f6b6c8cb-2r2bk" event={"ID":"62039600-b028-47bf-8e07-d5a9e1a3ec63","Type":"ContainerStarted","Data":"4e729b5cc422e8bd380376ea47b2ac967ffd9c98a2910f3ced2106bda54be20e"} Sep 17 10:23:35.241316 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:35.241276 2581 generic.go:356] "Generic (PLEG): container finished" podID="62039600-b028-47bf-8e07-d5a9e1a3ec63" containerID="4e729b5cc422e8bd380376ea47b2ac967ffd9c98a2910f3ced2106bda54be20e" exitCode=0 Sep 17 10:23:35.241683 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:35.241347 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-69f6b6c8cb-2r2bk" event={"ID":"62039600-b028-47bf-8e07-d5a9e1a3ec63","Type":"ContainerDied","Data":"4e729b5cc422e8bd380376ea47b2ac967ffd9c98a2910f3ced2106bda54be20e"} Sep 17 10:23:36.250130 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:36.250091 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-69f6b6c8cb-2r2bk" event={"ID":"62039600-b028-47bf-8e07-d5a9e1a3ec63","Type":"ContainerStarted","Data":"3947c8a2f98ce1fb2efb8b5f3f7b6bee51ea5c1bf55e2ca240b0c5495e3c7b40"} Sep 17 10:23:36.250595 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:36.250168 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-69f6b6c8cb-2r2bk" event={"ID":"62039600-b028-47bf-8e07-d5a9e1a3ec63","Type":"ContainerStarted","Data":"d55d3da2b99cb953889379070a5aea03c0331f14afceef34a64e9006b4661bd5"} Sep 17 10:23:36.250595 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:36.250194 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-69f6b6c8cb-2r2bk" event={"ID":"62039600-b028-47bf-8e07-d5a9e1a3ec63","Type":"ContainerStarted","Data":"61e3457aba55d6c2e91d758e921c9de70161620727f3b3a60d656dfc255f17fc"} Sep 17 10:23:36.277903 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:36.277835 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-credit-scoring-69f6b6c8cb-2r2bk" podStartSLOduration=11.277810701 podStartE2EDuration="11.277810701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 10:23:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 10:23:36.27598813 +0000 UTC m=+770.824545245" watchObservedRunningTime="2026-09-17 10:23:36.277810701 +0000 UTC m=+770.826367810" Sep 17 10:23:36.714973 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:36.714873 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-69f6b6c8cb-2r2bk" Sep 17 10:23:36.714973 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:36.714971 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-69f6b6c8cb-2r2bk" Sep 17 10:23:36.715164 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:36.714997 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-69f6b6c8cb-2r2bk" Sep 17 10:23:38.714312 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:38.714274 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-credit-scoring-69f6b6c8cb-2r2bk" Sep 17 10:23:38.714806 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:38.714404 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-credit-scoring-69f6b6c8cb-2r2bk" Sep 17 10:23:38.714806 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:38.714437 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-credit-scoring-69f6b6c8cb-2r2bk" Sep 17 10:23:38.716433 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:38.716395 2581 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-69f6b6c8cb-2r2bk" podUID="62039600-b028-47bf-8e07-d5a9e1a3ec63" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.30:8816: connect: connection refused" Sep 17 10:23:38.716433 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:38.716425 2581 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-69f6b6c8cb-2r2bk" podUID="62039600-b028-47bf-8e07-d5a9e1a3ec63" containerName="online" probeResult="failure" output="Get \"https://10.134.0.30:6567/health\": dial tcp 10.134.0.30:6567: connect: connection refused" Sep 17 10:23:38.716616 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:38.716399 2581 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-69f6b6c8cb-2r2bk" podUID="62039600-b028-47bf-8e07-d5a9e1a3ec63" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.30:6571: connect: connection refused" Sep 17 10:23:41.714671 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:41.714626 2581 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-69f6b6c8cb-2r2bk" podUID="62039600-b028-47bf-8e07-d5a9e1a3ec63" containerName="online" probeResult="failure" output="Get \"https://10.134.0.30:6567/health\": dial tcp 10.134.0.30:6567: connect: connection refused" Sep 17 10:23:41.715472 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:41.715434 2581 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-69f6b6c8cb-2r2bk" podUID="62039600-b028-47bf-8e07-d5a9e1a3ec63" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.30:8816: connect: connection refused" Sep 17 10:23:41.715645 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:41.715372 2581 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-69f6b6c8cb-2r2bk" podUID="62039600-b028-47bf-8e07-d5a9e1a3ec63" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.30:6571: connect: connection refused" Sep 17 10:23:44.714810 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:44.714729 2581 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-69f6b6c8cb-2r2bk" podUID="62039600-b028-47bf-8e07-d5a9e1a3ec63" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.30:6571: connect: connection refused" Sep 17 10:23:44.715950 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:44.715922 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-credit-scoring-69f6b6c8cb-2r2bk" Sep 17 10:23:44.716848 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:44.716826 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-credit-scoring-69f6b6c8cb-2r2bk" Sep 17 10:23:44.721011 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:44.720987 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-credit-scoring-69f6b6c8cb-2r2bk" Sep 17 10:23:45.312207 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:45.312177 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-credit-scoring-69f6b6c8cb-2r2bk" Sep 17 10:23:47.715012 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:47.714956 2581 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-69f6b6c8cb-2r2bk" podUID="62039600-b028-47bf-8e07-d5a9e1a3ec63" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.30:6571: connect: connection refused" Sep 17 10:23:50.715077 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:50.715039 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-credit-scoring-69f6b6c8cb-2r2bk" Sep 17 10:23:50.715849 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:23:50.715828 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-credit-scoring-69f6b6c8cb-2r2bk" Sep 17 10:24:54.422677 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:24:54.422644 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-credit-scoring-69f6b6c8cb-2r2bk"] Sep 17 10:24:54.423197 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:24:54.422933 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-credit-scoring-69f6b6c8cb-2r2bk" podUID="62039600-b028-47bf-8e07-d5a9e1a3ec63" containerName="online" containerID="cri-o://d55d3da2b99cb953889379070a5aea03c0331f14afceef34a64e9006b4661bd5" gracePeriod=30 Sep 17 10:24:54.423197 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:24:54.422966 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-credit-scoring-69f6b6c8cb-2r2bk" podUID="62039600-b028-47bf-8e07-d5a9e1a3ec63" containerName="offline" containerID="cri-o://3947c8a2f98ce1fb2efb8b5f3f7b6bee51ea5c1bf55e2ca240b0c5495e3c7b40" gracePeriod=30 Sep 17 10:24:54.423197 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:24:54.422933 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-credit-scoring-69f6b6c8cb-2r2bk" podUID="62039600-b028-47bf-8e07-d5a9e1a3ec63" containerName="registry" containerID="cri-o://61e3457aba55d6c2e91d758e921c9de70161620727f3b3a60d656dfc255f17fc" gracePeriod=30 Sep 17 10:24:54.443551 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:24:54.443521 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/postgres-864846d4f5-b84kl"] Sep 17 10:24:54.443716 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:24:54.443669 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/postgres-864846d4f5-b84kl" podUID="c7293908-2b25-4082-95ae-16f444683300" containerName="postgres" containerID="cri-o://2842e042a265358f82f4b96a388591251092bd8ab3ff3038da09867940459798" gracePeriod=30 Sep 17 10:24:54.599726 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:24:54.599693 2581 generic.go:356] "Generic (PLEG): container finished" podID="c7293908-2b25-4082-95ae-16f444683300" containerID="2842e042a265358f82f4b96a388591251092bd8ab3ff3038da09867940459798" exitCode=0 Sep 17 10:24:54.599908 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:24:54.599796 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-864846d4f5-b84kl" event={"ID":"c7293908-2b25-4082-95ae-16f444683300","Type":"ContainerDied","Data":"2842e042a265358f82f4b96a388591251092bd8ab3ff3038da09867940459798"} Sep 17 10:24:54.682667 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:24:54.682606 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-864846d4f5-b84kl" Sep 17 10:24:54.815096 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:24:54.815050 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m2gtc\" (UniqueName: \"kubernetes.io/projected/c7293908-2b25-4082-95ae-16f444683300-kube-api-access-m2gtc\") pod \"c7293908-2b25-4082-95ae-16f444683300\" (UID: \"c7293908-2b25-4082-95ae-16f444683300\") " Sep 17 10:24:54.815297 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:24:54.815149 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/c7293908-2b25-4082-95ae-16f444683300-postgresdata\") pod \"c7293908-2b25-4082-95ae-16f444683300\" (UID: \"c7293908-2b25-4082-95ae-16f444683300\") " Sep 17 10:24:54.817242 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:24:54.817198 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7293908-2b25-4082-95ae-16f444683300-kube-api-access-m2gtc" (OuterVolumeSpecName: "kube-api-access-m2gtc") pod "c7293908-2b25-4082-95ae-16f444683300" (UID: "c7293908-2b25-4082-95ae-16f444683300"). InnerVolumeSpecName "kube-api-access-m2gtc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 10:24:54.846310 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:24:54.846253 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7293908-2b25-4082-95ae-16f444683300-postgresdata" (OuterVolumeSpecName: "postgresdata") pod "c7293908-2b25-4082-95ae-16f444683300" (UID: "c7293908-2b25-4082-95ae-16f444683300"). InnerVolumeSpecName "postgresdata". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 10:24:54.916711 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:24:54.916664 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2gtc\" (UniqueName: \"kubernetes.io/projected/c7293908-2b25-4082-95ae-16f444683300-kube-api-access-m2gtc\") on node \"ip-10-0-137-203.ec2.internal\" DevicePath \"\"" Sep 17 10:24:54.916711 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:24:54.916707 2581 reconciler_common.go:299] "Volume detached for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/c7293908-2b25-4082-95ae-16f444683300-postgresdata\") on node \"ip-10-0-137-203.ec2.internal\" DevicePath \"\"" Sep 17 10:24:55.309173 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:24:55.309125 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-69f6b6c8cb-2r2bk" podUID="62039600-b028-47bf-8e07-d5a9e1a3ec63" containerName="online" probeResult="failure" output="Get \"https://10.134.0.30:6567/health\": dial tcp 10.134.0.30:6567: connect: connection refused" Sep 17 10:24:55.605178 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:24:55.605100 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-864846d4f5-b84kl" Sep 17 10:24:55.605609 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:24:55.605099 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-864846d4f5-b84kl" event={"ID":"c7293908-2b25-4082-95ae-16f444683300","Type":"ContainerDied","Data":"5ee925e6c6dfc54695a4757da870f0388fe254b60928ca909c990a6b5e460033"} Sep 17 10:24:55.605609 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:24:55.605232 2581 scope.go:117] "RemoveContainer" containerID="2842e042a265358f82f4b96a388591251092bd8ab3ff3038da09867940459798" Sep 17 10:24:55.608093 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:24:55.608066 2581 generic.go:356] "Generic (PLEG): container finished" podID="62039600-b028-47bf-8e07-d5a9e1a3ec63" containerID="d55d3da2b99cb953889379070a5aea03c0331f14afceef34a64e9006b4661bd5" exitCode=0 Sep 17 10:24:55.608192 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:24:55.608133 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-69f6b6c8cb-2r2bk" event={"ID":"62039600-b028-47bf-8e07-d5a9e1a3ec63","Type":"ContainerDied","Data":"d55d3da2b99cb953889379070a5aea03c0331f14afceef34a64e9006b4661bd5"} Sep 17 10:24:55.631707 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:24:55.631684 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/postgres-864846d4f5-b84kl"] Sep 17 10:24:55.637821 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:24:55.637801 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/postgres-864846d4f5-b84kl"] Sep 17 10:24:55.943602 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:24:55.943524 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c7293908-2b25-4082-95ae-16f444683300" path="/var/lib/kubelet/pods/c7293908-2b25-4082-95ae-16f444683300/volumes" Sep 17 10:25:05.309136 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:25:05.309090 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-69f6b6c8cb-2r2bk" podUID="62039600-b028-47bf-8e07-d5a9e1a3ec63" containerName="online" probeResult="failure" output="Get \"https://10.134.0.30:6567/health\": dial tcp 10.134.0.30:6567: connect: connection refused" Sep 17 10:25:15.309107 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:25:15.309067 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-69f6b6c8cb-2r2bk" podUID="62039600-b028-47bf-8e07-d5a9e1a3ec63" containerName="online" probeResult="failure" output="Get \"https://10.134.0.30:6567/health\": dial tcp 10.134.0.30:6567: connect: connection refused" Sep 17 10:25:15.309487 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:25:15.309219 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-69f6b6c8cb-2r2bk" Sep 17 10:25:24.716754 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:25:24.716649 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-69f6b6c8cb-2r2bk" podUID="62039600-b028-47bf-8e07-d5a9e1a3ec63" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.30:8816: connect: connection refused" Sep 17 10:25:24.724851 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:25:24.724820 2581 generic.go:356] "Generic (PLEG): container finished" podID="62039600-b028-47bf-8e07-d5a9e1a3ec63" containerID="3947c8a2f98ce1fb2efb8b5f3f7b6bee51ea5c1bf55e2ca240b0c5495e3c7b40" exitCode=137 Sep 17 10:25:24.724851 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:25:24.724846 2581 generic.go:356] "Generic (PLEG): container finished" podID="62039600-b028-47bf-8e07-d5a9e1a3ec63" containerID="61e3457aba55d6c2e91d758e921c9de70161620727f3b3a60d656dfc255f17fc" exitCode=137 Sep 17 10:25:24.725040 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:25:24.724893 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-69f6b6c8cb-2r2bk" event={"ID":"62039600-b028-47bf-8e07-d5a9e1a3ec63","Type":"ContainerDied","Data":"3947c8a2f98ce1fb2efb8b5f3f7b6bee51ea5c1bf55e2ca240b0c5495e3c7b40"} Sep 17 10:25:24.725040 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:25:24.724966 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-69f6b6c8cb-2r2bk" event={"ID":"62039600-b028-47bf-8e07-d5a9e1a3ec63","Type":"ContainerDied","Data":"61e3457aba55d6c2e91d758e921c9de70161620727f3b3a60d656dfc255f17fc"} Sep 17 10:25:25.066393 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:25:25.066368 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-69f6b6c8cb-2r2bk" Sep 17 10:25:25.157627 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:25:25.157595 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/62039600-b028-47bf-8e07-d5a9e1a3ec63-offline-tls\") pod \"62039600-b028-47bf-8e07-d5a9e1a3ec63\" (UID: \"62039600-b028-47bf-8e07-d5a9e1a3ec63\") " Sep 17 10:25:25.157773 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:25:25.157654 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/62039600-b028-47bf-8e07-d5a9e1a3ec63-registry-tls\") pod \"62039600-b028-47bf-8e07-d5a9e1a3ec63\" (UID: \"62039600-b028-47bf-8e07-d5a9e1a3ec63\") " Sep 17 10:25:25.157773 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:25:25.157673 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4fvh5\" (UniqueName: \"kubernetes.io/projected/62039600-b028-47bf-8e07-d5a9e1a3ec63-kube-api-access-4fvh5\") pod \"62039600-b028-47bf-8e07-d5a9e1a3ec63\" (UID: \"62039600-b028-47bf-8e07-d5a9e1a3ec63\") " Sep 17 10:25:25.157773 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:25:25.157710 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/62039600-b028-47bf-8e07-d5a9e1a3ec63-feast-data\") pod \"62039600-b028-47bf-8e07-d5a9e1a3ec63\" (UID: \"62039600-b028-47bf-8e07-d5a9e1a3ec63\") " Sep 17 10:25:25.157773 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:25:25.157726 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/62039600-b028-47bf-8e07-d5a9e1a3ec63-online-tls\") pod \"62039600-b028-47bf-8e07-d5a9e1a3ec63\" (UID: \"62039600-b028-47bf-8e07-d5a9e1a3ec63\") " Sep 17 10:25:25.159956 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:25:25.159928 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62039600-b028-47bf-8e07-d5a9e1a3ec63-kube-api-access-4fvh5" (OuterVolumeSpecName: "kube-api-access-4fvh5") pod "62039600-b028-47bf-8e07-d5a9e1a3ec63" (UID: "62039600-b028-47bf-8e07-d5a9e1a3ec63"). InnerVolumeSpecName "kube-api-access-4fvh5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 10:25:25.160055 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:25:25.159931 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62039600-b028-47bf-8e07-d5a9e1a3ec63-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "62039600-b028-47bf-8e07-d5a9e1a3ec63" (UID: "62039600-b028-47bf-8e07-d5a9e1a3ec63"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 10:25:25.160055 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:25:25.159920 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62039600-b028-47bf-8e07-d5a9e1a3ec63-online-tls" (OuterVolumeSpecName: "online-tls") pod "62039600-b028-47bf-8e07-d5a9e1a3ec63" (UID: "62039600-b028-47bf-8e07-d5a9e1a3ec63"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 10:25:25.160055 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:25:25.160009 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62039600-b028-47bf-8e07-d5a9e1a3ec63-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "62039600-b028-47bf-8e07-d5a9e1a3ec63" (UID: "62039600-b028-47bf-8e07-d5a9e1a3ec63"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 10:25:25.166099 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:25:25.166078 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62039600-b028-47bf-8e07-d5a9e1a3ec63-feast-data" (OuterVolumeSpecName: "feast-data") pod "62039600-b028-47bf-8e07-d5a9e1a3ec63" (UID: "62039600-b028-47bf-8e07-d5a9e1a3ec63"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 10:25:25.258980 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:25:25.258899 2581 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/62039600-b028-47bf-8e07-d5a9e1a3ec63-registry-tls\") on node \"ip-10-0-137-203.ec2.internal\" DevicePath \"\"" Sep 17 10:25:25.258980 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:25:25.258927 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fvh5\" (UniqueName: \"kubernetes.io/projected/62039600-b028-47bf-8e07-d5a9e1a3ec63-kube-api-access-4fvh5\") on node \"ip-10-0-137-203.ec2.internal\" DevicePath \"\"" Sep 17 10:25:25.258980 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:25:25.258940 2581 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/62039600-b028-47bf-8e07-d5a9e1a3ec63-feast-data\") on node \"ip-10-0-137-203.ec2.internal\" DevicePath \"\"" Sep 17 10:25:25.258980 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:25:25.258951 2581 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/62039600-b028-47bf-8e07-d5a9e1a3ec63-online-tls\") on node \"ip-10-0-137-203.ec2.internal\" DevicePath \"\"" Sep 17 10:25:25.258980 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:25:25.258961 2581 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/62039600-b028-47bf-8e07-d5a9e1a3ec63-offline-tls\") on node \"ip-10-0-137-203.ec2.internal\" DevicePath \"\"" Sep 17 10:25:25.731238 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:25:25.731209 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-69f6b6c8cb-2r2bk" event={"ID":"62039600-b028-47bf-8e07-d5a9e1a3ec63","Type":"ContainerDied","Data":"d20a685a36f107e91973f8f1fca0d904b55b7e9de2bb20482aa50ad6dbcd55be"} Sep 17 10:25:25.731723 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:25:25.731271 2581 scope.go:117] "RemoveContainer" containerID="3947c8a2f98ce1fb2efb8b5f3f7b6bee51ea5c1bf55e2ca240b0c5495e3c7b40" Sep 17 10:25:25.731723 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:25:25.731220 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-69f6b6c8cb-2r2bk" Sep 17 10:25:25.739639 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:25:25.739626 2581 scope.go:117] "RemoveContainer" containerID="d55d3da2b99cb953889379070a5aea03c0331f14afceef34a64e9006b4661bd5" Sep 17 10:25:25.747073 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:25:25.747057 2581 scope.go:117] "RemoveContainer" containerID="61e3457aba55d6c2e91d758e921c9de70161620727f3b3a60d656dfc255f17fc" Sep 17 10:25:25.753861 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:25:25.753839 2581 scope.go:117] "RemoveContainer" containerID="4e729b5cc422e8bd380376ea47b2ac967ffd9c98a2910f3ced2106bda54be20e" Sep 17 10:25:25.755264 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:25:25.755211 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-credit-scoring-69f6b6c8cb-2r2bk"] Sep 17 10:25:25.760715 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:25:25.760691 2581 scope.go:117] "RemoveContainer" containerID="326e4dd9845b92fb406324aa8d48c276adb3090a65fcd76b292494ecbf508dcf" Sep 17 10:25:25.762740 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:25:25.762721 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-credit-scoring-69f6b6c8cb-2r2bk"] Sep 17 10:25:25.943476 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:25:25.943435 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="62039600-b028-47bf-8e07-d5a9e1a3ec63" path="/var/lib/kubelet/pods/62039600-b028-47bf-8e07-d5a9e1a3ec63/volumes" Sep 17 10:25:45.880733 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:25:45.880662 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-b9wnf_762d3e10-7b7d-4ce9-9773-90a4fe41b474/console-operator/2.log" Sep 17 10:25:45.886241 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:25:45.886218 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-b9wnf_762d3e10-7b7d-4ce9-9773-90a4fe41b474/console-operator/2.log" Sep 17 10:27:57.048595 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:27:57.048563 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-f27pz"] Sep 17 10:27:57.049130 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:27:57.048932 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62039600-b028-47bf-8e07-d5a9e1a3ec63" containerName="registry" Sep 17 10:27:57.049130 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:27:57.048946 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="62039600-b028-47bf-8e07-d5a9e1a3ec63" containerName="registry" Sep 17 10:27:57.049130 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:27:57.048957 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62039600-b028-47bf-8e07-d5a9e1a3ec63" containerName="online" Sep 17 10:27:57.049130 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:27:57.048963 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="62039600-b028-47bf-8e07-d5a9e1a3ec63" containerName="online" Sep 17 10:27:57.049130 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:27:57.048970 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62039600-b028-47bf-8e07-d5a9e1a3ec63" containerName="feast-init" Sep 17 10:27:57.049130 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:27:57.048975 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="62039600-b028-47bf-8e07-d5a9e1a3ec63" containerName="feast-init" Sep 17 10:27:57.049130 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:27:57.048986 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62039600-b028-47bf-8e07-d5a9e1a3ec63" containerName="feast-apply" Sep 17 10:27:57.049130 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:27:57.048991 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="62039600-b028-47bf-8e07-d5a9e1a3ec63" containerName="feast-apply" Sep 17 10:27:57.049130 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:27:57.048999 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62039600-b028-47bf-8e07-d5a9e1a3ec63" containerName="offline" Sep 17 10:27:57.049130 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:27:57.049006 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="62039600-b028-47bf-8e07-d5a9e1a3ec63" containerName="offline" Sep 17 10:27:57.049130 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:27:57.049014 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7293908-2b25-4082-95ae-16f444683300" containerName="postgres" Sep 17 10:27:57.049130 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:27:57.049019 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7293908-2b25-4082-95ae-16f444683300" containerName="postgres" Sep 17 10:27:57.049130 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:27:57.049072 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="62039600-b028-47bf-8e07-d5a9e1a3ec63" containerName="online" Sep 17 10:27:57.049130 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:27:57.049078 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="62039600-b028-47bf-8e07-d5a9e1a3ec63" containerName="offline" Sep 17 10:27:57.049130 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:27:57.049085 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="62039600-b028-47bf-8e07-d5a9e1a3ec63" containerName="registry" Sep 17 10:27:57.049130 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:27:57.049091 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7293908-2b25-4082-95ae-16f444683300" containerName="postgres" Sep 17 10:27:57.051634 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:27:57.051616 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-69bbd798b6-f27pz" Sep 17 10:27:57.054626 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:27:57.054596 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"kube-root-ca.crt\"" Sep 17 10:27:57.054750 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:27:57.054644 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"default-dockercfg-6cknx\"" Sep 17 10:27:57.054750 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:27:57.054652 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"openshift-service-ca.crt\"" Sep 17 10:27:57.065619 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:27:57.065594 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-f27pz"] Sep 17 10:27:57.133860 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:27:57.133823 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccntm\" (UniqueName: \"kubernetes.io/projected/6d5f7d26-e44d-4cfb-8557-0f22457d1f81-kube-api-access-ccntm\") pod \"redis-69bbd798b6-f27pz\" (UID: \"6d5f7d26-e44d-4cfb-8557-0f22457d1f81\") " pod="test-ns-feast-rest/redis-69bbd798b6-f27pz" Sep 17 10:27:57.234457 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:27:57.234425 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccntm\" (UniqueName: \"kubernetes.io/projected/6d5f7d26-e44d-4cfb-8557-0f22457d1f81-kube-api-access-ccntm\") pod \"redis-69bbd798b6-f27pz\" (UID: \"6d5f7d26-e44d-4cfb-8557-0f22457d1f81\") " pod="test-ns-feast-rest/redis-69bbd798b6-f27pz" Sep 17 10:27:57.244202 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:27:57.244174 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccntm\" (UniqueName: \"kubernetes.io/projected/6d5f7d26-e44d-4cfb-8557-0f22457d1f81-kube-api-access-ccntm\") pod \"redis-69bbd798b6-f27pz\" (UID: \"6d5f7d26-e44d-4cfb-8557-0f22457d1f81\") " pod="test-ns-feast-rest/redis-69bbd798b6-f27pz" Sep 17 10:27:57.360691 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:27:57.360612 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-69bbd798b6-f27pz" Sep 17 10:27:57.629243 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:27:57.629216 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-f27pz"] Sep 17 10:27:57.631806 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:27:57.631760 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6d5f7d26_e44d_4cfb_8557_0f22457d1f81.slice/crio-ac1bbffe051e9b62fb49e980f5b8107a3114b8e3a75e7762bc04a04a83438274 WatchSource:0}: Error finding container ac1bbffe051e9b62fb49e980f5b8107a3114b8e3a75e7762bc04a04a83438274: Status 404 returned error can't find the container with id ac1bbffe051e9b62fb49e980f5b8107a3114b8e3a75e7762bc04a04a83438274 Sep 17 10:27:58.302949 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:27:58.302918 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-69bbd798b6-f27pz" event={"ID":"6d5f7d26-e44d-4cfb-8557-0f22457d1f81","Type":"ContainerStarted","Data":"ac1bbffe051e9b62fb49e980f5b8107a3114b8e3a75e7762bc04a04a83438274"} Sep 17 10:28:00.313065 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:28:00.313027 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-69bbd798b6-f27pz" event={"ID":"6d5f7d26-e44d-4cfb-8557-0f22457d1f81","Type":"ContainerStarted","Data":"628abd3a5941d937f09c3c69fc320aabad54239e62503f5203a1242a51a97f78"} Sep 17 10:28:00.333368 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:28:00.333317 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast-rest/redis-69bbd798b6-f27pz" podStartSLOduration=1.573627984 podStartE2EDuration="3.333303003s" totalImagesPullingTime="1.759675019s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 10:27:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 10:28:00.331795868 +0000 UTC m=+1034.880352971" watchObservedRunningTime="2026-09-17 10:28:00.333303003 +0000 UTC m=+1034.881860111" Sep 17 10:28:02.479897 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:28:02.479859 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-xkz72"] Sep 17 10:28:02.483743 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:28:02.483720 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-864846d4f5-xkz72" Sep 17 10:28:02.486585 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:28:02.486561 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"postgres-secret\"" Sep 17 10:28:02.490386 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:28:02.490360 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-xkz72"] Sep 17 10:28:02.583398 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:28:02.583359 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6cv2\" (UniqueName: \"kubernetes.io/projected/edf43eab-2a20-4f46-b694-2fd0cc8dcefb-kube-api-access-f6cv2\") pod \"postgres-864846d4f5-xkz72\" (UID: \"edf43eab-2a20-4f46-b694-2fd0cc8dcefb\") " pod="test-ns-feast-rest/postgres-864846d4f5-xkz72" Sep 17 10:28:02.583398 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:28:02.583400 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/edf43eab-2a20-4f46-b694-2fd0cc8dcefb-postgresdata\") pod \"postgres-864846d4f5-xkz72\" (UID: \"edf43eab-2a20-4f46-b694-2fd0cc8dcefb\") " pod="test-ns-feast-rest/postgres-864846d4f5-xkz72" Sep 17 10:28:02.684657 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:28:02.684625 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6cv2\" (UniqueName: \"kubernetes.io/projected/edf43eab-2a20-4f46-b694-2fd0cc8dcefb-kube-api-access-f6cv2\") pod \"postgres-864846d4f5-xkz72\" (UID: \"edf43eab-2a20-4f46-b694-2fd0cc8dcefb\") " pod="test-ns-feast-rest/postgres-864846d4f5-xkz72" Sep 17 10:28:02.684775 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:28:02.684664 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/edf43eab-2a20-4f46-b694-2fd0cc8dcefb-postgresdata\") pod \"postgres-864846d4f5-xkz72\" (UID: \"edf43eab-2a20-4f46-b694-2fd0cc8dcefb\") " pod="test-ns-feast-rest/postgres-864846d4f5-xkz72" Sep 17 10:28:02.685051 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:28:02.685032 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/edf43eab-2a20-4f46-b694-2fd0cc8dcefb-postgresdata\") pod \"postgres-864846d4f5-xkz72\" (UID: \"edf43eab-2a20-4f46-b694-2fd0cc8dcefb\") " pod="test-ns-feast-rest/postgres-864846d4f5-xkz72" Sep 17 10:28:02.700196 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:28:02.700168 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6cv2\" (UniqueName: \"kubernetes.io/projected/edf43eab-2a20-4f46-b694-2fd0cc8dcefb-kube-api-access-f6cv2\") pod \"postgres-864846d4f5-xkz72\" (UID: \"edf43eab-2a20-4f46-b694-2fd0cc8dcefb\") " pod="test-ns-feast-rest/postgres-864846d4f5-xkz72" Sep 17 10:28:02.794098 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:28:02.794038 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-864846d4f5-xkz72" Sep 17 10:28:03.077893 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:28:03.077868 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-xkz72"] Sep 17 10:28:03.080390 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:28:03.080360 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podedf43eab_2a20_4f46_b694_2fd0cc8dcefb.slice/crio-4fd3129b7b70db7c76dfec268b952b01d050b5ecc17596c57b79f2d8eb0213b5 WatchSource:0}: Error finding container 4fd3129b7b70db7c76dfec268b952b01d050b5ecc17596c57b79f2d8eb0213b5: Status 404 returned error can't find the container with id 4fd3129b7b70db7c76dfec268b952b01d050b5ecc17596c57b79f2d8eb0213b5 Sep 17 10:28:03.327837 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:28:03.327754 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-864846d4f5-xkz72" event={"ID":"edf43eab-2a20-4f46-b694-2fd0cc8dcefb","Type":"ContainerStarted","Data":"3f4915e859c2b34d9c88f2f89bde63d51323af674a8325a86ca8717505ac7ddd"} Sep 17 10:28:03.327837 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:28:03.327807 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-864846d4f5-xkz72" event={"ID":"edf43eab-2a20-4f46-b694-2fd0cc8dcefb","Type":"ContainerStarted","Data":"4fd3129b7b70db7c76dfec268b952b01d050b5ecc17596c57b79f2d8eb0213b5"} Sep 17 10:28:03.344897 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:28:03.344834 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast-rest/postgres-864846d4f5-xkz72" podStartSLOduration=1.344816845 podStartE2EDuration="1.344816845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 10:28:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 10:28:03.344752488 +0000 UTC m=+1037.893309599" watchObservedRunningTime="2026-09-17 10:28:03.344816845 +0000 UTC m=+1037.893373957" Sep 17 10:28:08.041743 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:28:08.041706 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-5796b44ff7-55v9h"] Sep 17 10:28:08.047874 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:28:08.047854 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-5796b44ff7-55v9h" Sep 17 10:28:08.050492 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:28:08.050468 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-online-tls\"" Sep 17 10:28:08.050634 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:28:08.050509 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-intra-comm\"" Sep 17 10:28:08.050767 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:28:08.050751 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-registry-tls\"" Sep 17 10:28:08.050888 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:28:08.050856 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-dockercfg-q6jcg\"" Sep 17 10:28:08.056210 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:28:08.056179 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-5796b44ff7-55v9h"] Sep 17 10:28:08.129615 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:28:08.129577 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/2cde7ec7-fb5f-4f2f-b660-fb9099eb1f1d-online-tls\") pod \"feast-credit-scoring-5796b44ff7-55v9h\" (UID: \"2cde7ec7-fb5f-4f2f-b660-fb9099eb1f1d\") " pod="test-ns-feast-rest/feast-credit-scoring-5796b44ff7-55v9h" Sep 17 10:28:08.129753 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:28:08.129641 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9bfn\" (UniqueName: \"kubernetes.io/projected/2cde7ec7-fb5f-4f2f-b660-fb9099eb1f1d-kube-api-access-v9bfn\") pod \"feast-credit-scoring-5796b44ff7-55v9h\" (UID: \"2cde7ec7-fb5f-4f2f-b660-fb9099eb1f1d\") " pod="test-ns-feast-rest/feast-credit-scoring-5796b44ff7-55v9h" Sep 17 10:28:08.129753 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:28:08.129681 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/2cde7ec7-fb5f-4f2f-b660-fb9099eb1f1d-feast-data\") pod \"feast-credit-scoring-5796b44ff7-55v9h\" (UID: \"2cde7ec7-fb5f-4f2f-b660-fb9099eb1f1d\") " pod="test-ns-feast-rest/feast-credit-scoring-5796b44ff7-55v9h" Sep 17 10:28:08.129753 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:28:08.129702 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/2cde7ec7-fb5f-4f2f-b660-fb9099eb1f1d-registry-tls\") pod \"feast-credit-scoring-5796b44ff7-55v9h\" (UID: \"2cde7ec7-fb5f-4f2f-b660-fb9099eb1f1d\") " pod="test-ns-feast-rest/feast-credit-scoring-5796b44ff7-55v9h" Sep 17 10:28:08.230877 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:28:08.230838 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/2cde7ec7-fb5f-4f2f-b660-fb9099eb1f1d-online-tls\") pod \"feast-credit-scoring-5796b44ff7-55v9h\" (UID: \"2cde7ec7-fb5f-4f2f-b660-fb9099eb1f1d\") " pod="test-ns-feast-rest/feast-credit-scoring-5796b44ff7-55v9h" Sep 17 10:28:08.231055 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:28:08.230895 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9bfn\" (UniqueName: \"kubernetes.io/projected/2cde7ec7-fb5f-4f2f-b660-fb9099eb1f1d-kube-api-access-v9bfn\") pod \"feast-credit-scoring-5796b44ff7-55v9h\" (UID: \"2cde7ec7-fb5f-4f2f-b660-fb9099eb1f1d\") " pod="test-ns-feast-rest/feast-credit-scoring-5796b44ff7-55v9h" Sep 17 10:28:08.231055 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:28:08.230916 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/2cde7ec7-fb5f-4f2f-b660-fb9099eb1f1d-feast-data\") pod \"feast-credit-scoring-5796b44ff7-55v9h\" (UID: \"2cde7ec7-fb5f-4f2f-b660-fb9099eb1f1d\") " pod="test-ns-feast-rest/feast-credit-scoring-5796b44ff7-55v9h" Sep 17 10:28:08.231055 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:28:08.230936 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/2cde7ec7-fb5f-4f2f-b660-fb9099eb1f1d-registry-tls\") pod \"feast-credit-scoring-5796b44ff7-55v9h\" (UID: \"2cde7ec7-fb5f-4f2f-b660-fb9099eb1f1d\") " pod="test-ns-feast-rest/feast-credit-scoring-5796b44ff7-55v9h" Sep 17 10:28:08.231232 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:28:08.231145 2581 secret.go:189] Couldn't get secret test-ns-feast-rest/feast-credit-scoring-registry-tls: secret "feast-credit-scoring-registry-tls" not found Sep 17 10:28:08.231282 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:28:08.231234 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2cde7ec7-fb5f-4f2f-b660-fb9099eb1f1d-registry-tls podName:2cde7ec7-fb5f-4f2f-b660-fb9099eb1f1d nodeName:}" failed. No retries permitted until 2026-09-17 10:28:08.731211156 +0000 UTC m=+1043.279768260 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/2cde7ec7-fb5f-4f2f-b660-fb9099eb1f1d-registry-tls") pod "feast-credit-scoring-5796b44ff7-55v9h" (UID: "2cde7ec7-fb5f-4f2f-b660-fb9099eb1f1d") : secret "feast-credit-scoring-registry-tls" not found Sep 17 10:28:08.231342 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:28:08.231334 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/2cde7ec7-fb5f-4f2f-b660-fb9099eb1f1d-feast-data\") pod \"feast-credit-scoring-5796b44ff7-55v9h\" (UID: \"2cde7ec7-fb5f-4f2f-b660-fb9099eb1f1d\") " pod="test-ns-feast-rest/feast-credit-scoring-5796b44ff7-55v9h" Sep 17 10:28:08.233311 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:28:08.233288 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/2cde7ec7-fb5f-4f2f-b660-fb9099eb1f1d-online-tls\") pod \"feast-credit-scoring-5796b44ff7-55v9h\" (UID: \"2cde7ec7-fb5f-4f2f-b660-fb9099eb1f1d\") " pod="test-ns-feast-rest/feast-credit-scoring-5796b44ff7-55v9h" Sep 17 10:28:08.240328 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:28:08.240305 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9bfn\" (UniqueName: \"kubernetes.io/projected/2cde7ec7-fb5f-4f2f-b660-fb9099eb1f1d-kube-api-access-v9bfn\") pod \"feast-credit-scoring-5796b44ff7-55v9h\" (UID: \"2cde7ec7-fb5f-4f2f-b660-fb9099eb1f1d\") " pod="test-ns-feast-rest/feast-credit-scoring-5796b44ff7-55v9h" Sep 17 10:28:08.735369 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:28:08.735307 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/2cde7ec7-fb5f-4f2f-b660-fb9099eb1f1d-registry-tls\") pod \"feast-credit-scoring-5796b44ff7-55v9h\" (UID: \"2cde7ec7-fb5f-4f2f-b660-fb9099eb1f1d\") " pod="test-ns-feast-rest/feast-credit-scoring-5796b44ff7-55v9h" Sep 17 10:28:08.737740 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:28:08.737713 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/2cde7ec7-fb5f-4f2f-b660-fb9099eb1f1d-registry-tls\") pod \"feast-credit-scoring-5796b44ff7-55v9h\" (UID: \"2cde7ec7-fb5f-4f2f-b660-fb9099eb1f1d\") " pod="test-ns-feast-rest/feast-credit-scoring-5796b44ff7-55v9h" Sep 17 10:28:08.958281 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:28:08.958244 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-5796b44ff7-55v9h" Sep 17 10:28:09.236203 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:28:09.236174 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-5796b44ff7-55v9h"] Sep 17 10:28:09.236579 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:28:09.236527 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cde7ec7_fb5f_4f2f_b660_fb9099eb1f1d.slice/crio-ad881c36b1bc67694f2acd07d5635bcf57bef34629873145e516cfdb08d1ef56 WatchSource:0}: Error finding container ad881c36b1bc67694f2acd07d5635bcf57bef34629873145e516cfdb08d1ef56: Status 404 returned error can't find the container with id ad881c36b1bc67694f2acd07d5635bcf57bef34629873145e516cfdb08d1ef56 Sep 17 10:28:09.352230 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:28:09.352191 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-5796b44ff7-55v9h" event={"ID":"2cde7ec7-fb5f-4f2f-b660-fb9099eb1f1d","Type":"ContainerStarted","Data":"dcb5eeed3349e76a8c2b5386e564f8bf91fefa830255f395e9f668c92be3ab09"} Sep 17 10:28:09.352433 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:28:09.352243 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-5796b44ff7-55v9h" event={"ID":"2cde7ec7-fb5f-4f2f-b660-fb9099eb1f1d","Type":"ContainerStarted","Data":"ad881c36b1bc67694f2acd07d5635bcf57bef34629873145e516cfdb08d1ef56"} Sep 17 10:28:10.358229 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:28:10.358195 2581 generic.go:356] "Generic (PLEG): container finished" podID="2cde7ec7-fb5f-4f2f-b660-fb9099eb1f1d" containerID="dcb5eeed3349e76a8c2b5386e564f8bf91fefa830255f395e9f668c92be3ab09" exitCode=0 Sep 17 10:28:10.358595 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:28:10.358259 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-5796b44ff7-55v9h" event={"ID":"2cde7ec7-fb5f-4f2f-b660-fb9099eb1f1d","Type":"ContainerDied","Data":"dcb5eeed3349e76a8c2b5386e564f8bf91fefa830255f395e9f668c92be3ab09"} Sep 17 10:28:11.364639 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:28:11.364592 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-5796b44ff7-55v9h" event={"ID":"2cde7ec7-fb5f-4f2f-b660-fb9099eb1f1d","Type":"ContainerStarted","Data":"96264d0b755d11b69c9cf6b5fdb890c1a173512fe9a0048577c35abc36c169ba"} Sep 17 10:28:18.399907 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:28:18.399876 2581 generic.go:356] "Generic (PLEG): container finished" podID="2cde7ec7-fb5f-4f2f-b660-fb9099eb1f1d" containerID="96264d0b755d11b69c9cf6b5fdb890c1a173512fe9a0048577c35abc36c169ba" exitCode=0 Sep 17 10:28:18.400288 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:28:18.399932 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-5796b44ff7-55v9h" event={"ID":"2cde7ec7-fb5f-4f2f-b660-fb9099eb1f1d","Type":"ContainerDied","Data":"96264d0b755d11b69c9cf6b5fdb890c1a173512fe9a0048577c35abc36c169ba"} Sep 17 10:28:19.412136 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:28:19.412095 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-5796b44ff7-55v9h" event={"ID":"2cde7ec7-fb5f-4f2f-b660-fb9099eb1f1d","Type":"ContainerStarted","Data":"29b554c1a64dcf65ca8c498a1cade999404fb3cbb31c564ad48511852da6b94f"} Sep 17 10:28:19.412477 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:28:19.412172 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-5796b44ff7-55v9h" event={"ID":"2cde7ec7-fb5f-4f2f-b660-fb9099eb1f1d","Type":"ContainerStarted","Data":"171ac6eb85463cfdf7ae854fea5e537c41b23592b13deacb133f10029a2d2adf"} Sep 17 10:28:19.437034 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:28:19.436982 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast-rest/feast-credit-scoring-5796b44ff7-55v9h" podStartSLOduration=11.436965594 podStartE2EDuration="11.436965594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 10:28:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 10:28:19.436249074 +0000 UTC m=+1053.984806187" watchObservedRunningTime="2026-09-17 10:28:19.436965594 +0000 UTC m=+1053.985522705" Sep 17 10:28:20.959321 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:28:20.959280 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-credit-scoring-5796b44ff7-55v9h" Sep 17 10:28:20.959707 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:28:20.959353 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-credit-scoring-5796b44ff7-55v9h" Sep 17 10:28:20.961071 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:28:20.961039 2581 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-5796b44ff7-55v9h" podUID="2cde7ec7-fb5f-4f2f-b660-fb9099eb1f1d" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.33:6571: connect: connection refused" Sep 17 10:28:20.961178 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:28:20.961075 2581 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-5796b44ff7-55v9h" podUID="2cde7ec7-fb5f-4f2f-b660-fb9099eb1f1d" containerName="online" probeResult="failure" output="Get \"https://10.134.0.33:6567/health\": dial tcp 10.134.0.33:6567: connect: connection refused" Sep 17 10:28:23.958732 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:28:23.958689 2581 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-5796b44ff7-55v9h" podUID="2cde7ec7-fb5f-4f2f-b660-fb9099eb1f1d" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.33:6571: connect: connection refused" Sep 17 10:28:23.958732 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:28:23.958706 2581 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-5796b44ff7-55v9h" podUID="2cde7ec7-fb5f-4f2f-b660-fb9099eb1f1d" containerName="online" probeResult="failure" output="Get \"https://10.134.0.33:6567/health\": dial tcp 10.134.0.33:6567: connect: connection refused" Sep 17 10:28:26.959169 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:28:26.959113 2581 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-5796b44ff7-55v9h" podUID="2cde7ec7-fb5f-4f2f-b660-fb9099eb1f1d" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.33:6571: connect: connection refused" Sep 17 10:28:26.965515 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:28:26.965483 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-credit-scoring-5796b44ff7-55v9h" Sep 17 10:28:26.965915 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:28:26.965891 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-credit-scoring-5796b44ff7-55v9h" Sep 17 10:28:26.968971 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:28:26.968951 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-credit-scoring-5796b44ff7-55v9h" Sep 17 10:28:28.959161 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:28:28.959118 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-credit-scoring-5796b44ff7-55v9h" Sep 17 10:28:29.958716 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:28:29.958676 2581 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-5796b44ff7-55v9h" podUID="2cde7ec7-fb5f-4f2f-b660-fb9099eb1f1d" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.33:6571: connect: connection refused" Sep 17 10:28:32.959495 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:28:32.959459 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-credit-scoring-5796b44ff7-55v9h" Sep 17 10:28:32.960241 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:28:32.960221 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-credit-scoring-5796b44ff7-55v9h" Sep 17 10:28:33.881622 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:28:33.881582 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-66c7d475f4-npsl9"] Sep 17 10:28:33.886311 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:28:33.886286 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-66c7d475f4-npsl9" Sep 17 10:28:33.889034 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:28:33.889001 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-intra-comm\"" Sep 17 10:28:33.889144 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:28:33.889130 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-dockercfg-6n96m\"" Sep 17 10:28:33.889192 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:28:33.889126 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-online-tls\"" Sep 17 10:28:33.890302 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:28:33.890282 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-registry-tls\"" Sep 17 10:28:33.895759 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:28:33.895733 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-66c7d475f4-npsl9"] Sep 17 10:28:33.954153 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:28:33.954120 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/ec179b30-fa6c-4c70-881a-18b96573a475-feast-data\") pod \"feast-driver-ranking-66c7d475f4-npsl9\" (UID: \"ec179b30-fa6c-4c70-881a-18b96573a475\") " pod="test-ns-feast-rest/feast-driver-ranking-66c7d475f4-npsl9" Sep 17 10:28:33.954317 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:28:33.954161 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/ec179b30-fa6c-4c70-881a-18b96573a475-registry-tls\") pod \"feast-driver-ranking-66c7d475f4-npsl9\" (UID: \"ec179b30-fa6c-4c70-881a-18b96573a475\") " pod="test-ns-feast-rest/feast-driver-ranking-66c7d475f4-npsl9" Sep 17 10:28:33.954317 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:28:33.954186 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfdgc\" (UniqueName: \"kubernetes.io/projected/ec179b30-fa6c-4c70-881a-18b96573a475-kube-api-access-cfdgc\") pod \"feast-driver-ranking-66c7d475f4-npsl9\" (UID: \"ec179b30-fa6c-4c70-881a-18b96573a475\") " pod="test-ns-feast-rest/feast-driver-ranking-66c7d475f4-npsl9" Sep 17 10:28:33.954317 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:28:33.954279 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/ec179b30-fa6c-4c70-881a-18b96573a475-online-tls\") pod \"feast-driver-ranking-66c7d475f4-npsl9\" (UID: \"ec179b30-fa6c-4c70-881a-18b96573a475\") " pod="test-ns-feast-rest/feast-driver-ranking-66c7d475f4-npsl9" Sep 17 10:28:34.055575 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:28:34.055527 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/ec179b30-fa6c-4c70-881a-18b96573a475-registry-tls\") pod \"feast-driver-ranking-66c7d475f4-npsl9\" (UID: \"ec179b30-fa6c-4c70-881a-18b96573a475\") " pod="test-ns-feast-rest/feast-driver-ranking-66c7d475f4-npsl9" Sep 17 10:28:34.055575 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:28:34.055580 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfdgc\" (UniqueName: \"kubernetes.io/projected/ec179b30-fa6c-4c70-881a-18b96573a475-kube-api-access-cfdgc\") pod \"feast-driver-ranking-66c7d475f4-npsl9\" (UID: \"ec179b30-fa6c-4c70-881a-18b96573a475\") " pod="test-ns-feast-rest/feast-driver-ranking-66c7d475f4-npsl9" Sep 17 10:28:34.056034 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:28:34.055774 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/ec179b30-fa6c-4c70-881a-18b96573a475-online-tls\") pod \"feast-driver-ranking-66c7d475f4-npsl9\" (UID: \"ec179b30-fa6c-4c70-881a-18b96573a475\") " pod="test-ns-feast-rest/feast-driver-ranking-66c7d475f4-npsl9" Sep 17 10:28:34.056034 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:28:34.056016 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/ec179b30-fa6c-4c70-881a-18b96573a475-feast-data\") pod \"feast-driver-ranking-66c7d475f4-npsl9\" (UID: \"ec179b30-fa6c-4c70-881a-18b96573a475\") " pod="test-ns-feast-rest/feast-driver-ranking-66c7d475f4-npsl9" Sep 17 10:28:34.056368 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:28:34.056350 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/ec179b30-fa6c-4c70-881a-18b96573a475-feast-data\") pod \"feast-driver-ranking-66c7d475f4-npsl9\" (UID: \"ec179b30-fa6c-4c70-881a-18b96573a475\") " pod="test-ns-feast-rest/feast-driver-ranking-66c7d475f4-npsl9" Sep 17 10:28:34.058080 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:28:34.058061 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/ec179b30-fa6c-4c70-881a-18b96573a475-online-tls\") pod \"feast-driver-ranking-66c7d475f4-npsl9\" (UID: \"ec179b30-fa6c-4c70-881a-18b96573a475\") " pod="test-ns-feast-rest/feast-driver-ranking-66c7d475f4-npsl9" Sep 17 10:28:34.058187 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:28:34.058170 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/ec179b30-fa6c-4c70-881a-18b96573a475-registry-tls\") pod \"feast-driver-ranking-66c7d475f4-npsl9\" (UID: \"ec179b30-fa6c-4c70-881a-18b96573a475\") " pod="test-ns-feast-rest/feast-driver-ranking-66c7d475f4-npsl9" Sep 17 10:28:34.066650 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:28:34.066629 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfdgc\" (UniqueName: \"kubernetes.io/projected/ec179b30-fa6c-4c70-881a-18b96573a475-kube-api-access-cfdgc\") pod \"feast-driver-ranking-66c7d475f4-npsl9\" (UID: \"ec179b30-fa6c-4c70-881a-18b96573a475\") " pod="test-ns-feast-rest/feast-driver-ranking-66c7d475f4-npsl9" Sep 17 10:28:34.196548 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:28:34.196451 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-66c7d475f4-npsl9" Sep 17 10:28:34.470601 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:28:34.470575 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-66c7d475f4-npsl9"] Sep 17 10:28:34.472667 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:28:34.472641 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec179b30_fa6c_4c70_881a_18b96573a475.slice/crio-f47b16b8d0e4b38fc04b7b8e2838a7f5c824c44bacd8562e6ad92ae86a1fafc4 WatchSource:0}: Error finding container f47b16b8d0e4b38fc04b7b8e2838a7f5c824c44bacd8562e6ad92ae86a1fafc4: Status 404 returned error can't find the container with id f47b16b8d0e4b38fc04b7b8e2838a7f5c824c44bacd8562e6ad92ae86a1fafc4 Sep 17 10:28:34.479735 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:28:34.479714 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-66c7d475f4-npsl9" event={"ID":"ec179b30-fa6c-4c70-881a-18b96573a475","Type":"ContainerStarted","Data":"f47b16b8d0e4b38fc04b7b8e2838a7f5c824c44bacd8562e6ad92ae86a1fafc4"} Sep 17 10:28:35.485851 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:28:35.485815 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-66c7d475f4-npsl9" event={"ID":"ec179b30-fa6c-4c70-881a-18b96573a475","Type":"ContainerStarted","Data":"fba06518aa62f05ff65ca4f7f6e5d43d52f189e74d00b5660f2e9e35f262417a"} Sep 17 10:28:39.505666 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:28:39.505631 2581 generic.go:356] "Generic (PLEG): container finished" podID="ec179b30-fa6c-4c70-881a-18b96573a475" containerID="fba06518aa62f05ff65ca4f7f6e5d43d52f189e74d00b5660f2e9e35f262417a" exitCode=0 Sep 17 10:28:39.506065 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:28:39.505707 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-66c7d475f4-npsl9" event={"ID":"ec179b30-fa6c-4c70-881a-18b96573a475","Type":"ContainerDied","Data":"fba06518aa62f05ff65ca4f7f6e5d43d52f189e74d00b5660f2e9e35f262417a"} Sep 17 10:28:40.512426 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:28:40.512394 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-66c7d475f4-npsl9" event={"ID":"ec179b30-fa6c-4c70-881a-18b96573a475","Type":"ContainerStarted","Data":"d71c67e8e1e87f6c90af7af030d68f922226f6aecbc6ff095f03a63e90a0380d"} Sep 17 10:28:46.538067 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:28:46.538031 2581 generic.go:356] "Generic (PLEG): container finished" podID="ec179b30-fa6c-4c70-881a-18b96573a475" containerID="d71c67e8e1e87f6c90af7af030d68f922226f6aecbc6ff095f03a63e90a0380d" exitCode=0 Sep 17 10:28:46.538481 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:28:46.538104 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-66c7d475f4-npsl9" event={"ID":"ec179b30-fa6c-4c70-881a-18b96573a475","Type":"ContainerDied","Data":"d71c67e8e1e87f6c90af7af030d68f922226f6aecbc6ff095f03a63e90a0380d"} Sep 17 10:28:46.541712 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:28:46.541691 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-intra-comm\"" Sep 17 10:28:47.546365 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:28:47.546332 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-66c7d475f4-npsl9" event={"ID":"ec179b30-fa6c-4c70-881a-18b96573a475","Type":"ContainerStarted","Data":"cf1f626b1fb04162bac8c7120fe53ddedda713f66b8b11f16d64b49e0a7c5f1d"} Sep 17 10:28:47.546756 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:28:47.546418 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-66c7d475f4-npsl9" event={"ID":"ec179b30-fa6c-4c70-881a-18b96573a475","Type":"ContainerStarted","Data":"c9d005ab04a7a59eb91748abf0d73b1d737a3548f77e50fc71025ecd522c5c29"} Sep 17 10:28:47.571907 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:28:47.571856 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast-rest/feast-driver-ranking-66c7d475f4-npsl9" podStartSLOduration=14.571841414 podStartE2EDuration="14.571841414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 10:28:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 10:28:47.570695324 +0000 UTC m=+1082.119252439" watchObservedRunningTime="2026-09-17 10:28:47.571841414 +0000 UTC m=+1082.120398524" Sep 17 10:28:49.197523 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:28:49.197476 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-driver-ranking-66c7d475f4-npsl9" Sep 17 10:28:49.197980 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:28:49.197561 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-driver-ranking-66c7d475f4-npsl9" Sep 17 10:28:49.199160 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:28:49.199133 2581 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-66c7d475f4-npsl9" podUID="ec179b30-fa6c-4c70-881a-18b96573a475" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.34:6571: connect: connection refused" Sep 17 10:28:49.199247 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:28:49.199185 2581 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-66c7d475f4-npsl9" podUID="ec179b30-fa6c-4c70-881a-18b96573a475" containerName="online" probeResult="failure" output="Get \"https://10.134.0.34:6567/health\": dial tcp 10.134.0.34:6567: connect: connection refused" Sep 17 10:28:52.197451 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:28:52.197405 2581 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-66c7d475f4-npsl9" podUID="ec179b30-fa6c-4c70-881a-18b96573a475" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.34:6571: connect: connection refused" Sep 17 10:28:52.197946 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:28:52.197464 2581 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-66c7d475f4-npsl9" podUID="ec179b30-fa6c-4c70-881a-18b96573a475" containerName="online" probeResult="failure" output="Get \"https://10.134.0.34:6567/health\": dial tcp 10.134.0.34:6567: connect: connection refused" Sep 17 10:28:54.196642 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:28:54.196601 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-driver-ranking-66c7d475f4-npsl9" Sep 17 10:28:54.197033 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:28:54.196671 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-driver-ranking-66c7d475f4-npsl9" Sep 17 10:28:55.197617 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:28:55.197576 2581 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-66c7d475f4-npsl9" podUID="ec179b30-fa6c-4c70-881a-18b96573a475" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.34:6571: connect: connection refused" Sep 17 10:28:55.204206 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:28:55.204173 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-driver-ranking-66c7d475f4-npsl9" Sep 17 10:28:55.207592 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:28:55.207574 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-driver-ranking-66c7d475f4-npsl9" Sep 17 10:28:58.197756 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:28:58.197710 2581 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-66c7d475f4-npsl9" podUID="ec179b30-fa6c-4c70-881a-18b96573a475" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.34:6571: connect: connection refused" Sep 17 10:29:01.198024 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:29:01.197994 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-driver-ranking-66c7d475f4-npsl9" Sep 17 10:29:01.198774 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:29:01.198751 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-driver-ranking-66c7d475f4-npsl9" Sep 17 10:30:21.982001 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:30:21.981955 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-5796b44ff7-55v9h"] Sep 17 10:30:21.982537 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:30:21.982290 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast-rest/feast-credit-scoring-5796b44ff7-55v9h" podUID="2cde7ec7-fb5f-4f2f-b660-fb9099eb1f1d" containerName="registry" containerID="cri-o://171ac6eb85463cfdf7ae854fea5e537c41b23592b13deacb133f10029a2d2adf" gracePeriod=30 Sep 17 10:30:21.982537 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:30:21.982321 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast-rest/feast-credit-scoring-5796b44ff7-55v9h" podUID="2cde7ec7-fb5f-4f2f-b660-fb9099eb1f1d" containerName="online" containerID="cri-o://29b554c1a64dcf65ca8c498a1cade999404fb3cbb31c564ad48511852da6b94f" gracePeriod=30 Sep 17 10:30:21.985432 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:30:21.985406 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-66c7d475f4-npsl9"] Sep 17 10:30:21.985641 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:30:21.985614 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast-rest/feast-driver-ranking-66c7d475f4-npsl9" podUID="ec179b30-fa6c-4c70-881a-18b96573a475" containerName="registry" containerID="cri-o://c9d005ab04a7a59eb91748abf0d73b1d737a3548f77e50fc71025ecd522c5c29" gracePeriod=30 Sep 17 10:30:21.985769 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:30:21.985640 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast-rest/feast-driver-ranking-66c7d475f4-npsl9" podUID="ec179b30-fa6c-4c70-881a-18b96573a475" containerName="online" containerID="cri-o://cf1f626b1fb04162bac8c7120fe53ddedda713f66b8b11f16d64b49e0a7c5f1d" gracePeriod=30 Sep 17 10:30:22.965732 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:30:22.965702 2581 generic.go:356] "Generic (PLEG): container finished" podID="ec179b30-fa6c-4c70-881a-18b96573a475" containerID="cf1f626b1fb04162bac8c7120fe53ddedda713f66b8b11f16d64b49e0a7c5f1d" exitCode=0 Sep 17 10:30:22.965904 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:30:22.965789 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-66c7d475f4-npsl9" event={"ID":"ec179b30-fa6c-4c70-881a-18b96573a475","Type":"ContainerDied","Data":"cf1f626b1fb04162bac8c7120fe53ddedda713f66b8b11f16d64b49e0a7c5f1d"} Sep 17 10:30:22.968048 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:30:22.968028 2581 generic.go:356] "Generic (PLEG): container finished" podID="2cde7ec7-fb5f-4f2f-b660-fb9099eb1f1d" containerID="29b554c1a64dcf65ca8c498a1cade999404fb3cbb31c564ad48511852da6b94f" exitCode=0 Sep 17 10:30:22.968140 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:30:22.968057 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-5796b44ff7-55v9h" event={"ID":"2cde7ec7-fb5f-4f2f-b660-fb9099eb1f1d","Type":"ContainerDied","Data":"29b554c1a64dcf65ca8c498a1cade999404fb3cbb31c564ad48511852da6b94f"} Sep 17 10:30:24.210899 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:30:24.210861 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-xkz72"] Sep 17 10:30:24.211332 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:30:24.211073 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast-rest/postgres-864846d4f5-xkz72" podUID="edf43eab-2a20-4f46-b694-2fd0cc8dcefb" containerName="postgres" containerID="cri-o://3f4915e859c2b34d9c88f2f89bde63d51323af674a8325a86ca8717505ac7ddd" gracePeriod=30 Sep 17 10:30:24.216863 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:30:24.216827 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-f27pz"] Sep 17 10:30:24.217271 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:30:24.217207 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast-rest/redis-69bbd798b6-f27pz" podUID="6d5f7d26-e44d-4cfb-8557-0f22457d1f81" containerName="redis" containerID="cri-o://628abd3a5941d937f09c3c69fc320aabad54239e62503f5203a1242a51a97f78" gracePeriod=30 Sep 17 10:30:24.479677 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:30:24.479653 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-69bbd798b6-f27pz" Sep 17 10:30:24.482435 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:30:24.482413 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-864846d4f5-xkz72" Sep 17 10:30:24.669261 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:30:24.669226 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ccntm\" (UniqueName: \"kubernetes.io/projected/6d5f7d26-e44d-4cfb-8557-0f22457d1f81-kube-api-access-ccntm\") pod \"6d5f7d26-e44d-4cfb-8557-0f22457d1f81\" (UID: \"6d5f7d26-e44d-4cfb-8557-0f22457d1f81\") " Sep 17 10:30:24.669424 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:30:24.669285 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f6cv2\" (UniqueName: \"kubernetes.io/projected/edf43eab-2a20-4f46-b694-2fd0cc8dcefb-kube-api-access-f6cv2\") pod \"edf43eab-2a20-4f46-b694-2fd0cc8dcefb\" (UID: \"edf43eab-2a20-4f46-b694-2fd0cc8dcefb\") " Sep 17 10:30:24.669424 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:30:24.669315 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/edf43eab-2a20-4f46-b694-2fd0cc8dcefb-postgresdata\") pod \"edf43eab-2a20-4f46-b694-2fd0cc8dcefb\" (UID: \"edf43eab-2a20-4f46-b694-2fd0cc8dcefb\") " Sep 17 10:30:24.671351 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:30:24.671317 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d5f7d26-e44d-4cfb-8557-0f22457d1f81-kube-api-access-ccntm" (OuterVolumeSpecName: "kube-api-access-ccntm") pod "6d5f7d26-e44d-4cfb-8557-0f22457d1f81" (UID: "6d5f7d26-e44d-4cfb-8557-0f22457d1f81"). InnerVolumeSpecName "kube-api-access-ccntm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 10:30:24.671454 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:30:24.671364 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edf43eab-2a20-4f46-b694-2fd0cc8dcefb-kube-api-access-f6cv2" (OuterVolumeSpecName: "kube-api-access-f6cv2") pod "edf43eab-2a20-4f46-b694-2fd0cc8dcefb" (UID: "edf43eab-2a20-4f46-b694-2fd0cc8dcefb"). InnerVolumeSpecName "kube-api-access-f6cv2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 10:30:24.697685 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:30:24.697654 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edf43eab-2a20-4f46-b694-2fd0cc8dcefb-postgresdata" (OuterVolumeSpecName: "postgresdata") pod "edf43eab-2a20-4f46-b694-2fd0cc8dcefb" (UID: "edf43eab-2a20-4f46-b694-2fd0cc8dcefb"). InnerVolumeSpecName "postgresdata". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 10:30:24.770822 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:30:24.770772 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccntm\" (UniqueName: \"kubernetes.io/projected/6d5f7d26-e44d-4cfb-8557-0f22457d1f81-kube-api-access-ccntm\") on node \"ip-10-0-137-203.ec2.internal\" DevicePath \"\"" Sep 17 10:30:24.770822 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:30:24.770822 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6cv2\" (UniqueName: \"kubernetes.io/projected/edf43eab-2a20-4f46-b694-2fd0cc8dcefb-kube-api-access-f6cv2\") on node \"ip-10-0-137-203.ec2.internal\" DevicePath \"\"" Sep 17 10:30:24.770963 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:30:24.770832 2581 reconciler_common.go:299] "Volume detached for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/edf43eab-2a20-4f46-b694-2fd0cc8dcefb-postgresdata\") on node \"ip-10-0-137-203.ec2.internal\" DevicePath \"\"" Sep 17 10:30:24.976254 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:30:24.976217 2581 generic.go:356] "Generic (PLEG): container finished" podID="6d5f7d26-e44d-4cfb-8557-0f22457d1f81" containerID="628abd3a5941d937f09c3c69fc320aabad54239e62503f5203a1242a51a97f78" exitCode=0 Sep 17 10:30:24.976426 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:30:24.976259 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-69bbd798b6-f27pz" Sep 17 10:30:24.976426 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:30:24.976299 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-69bbd798b6-f27pz" event={"ID":"6d5f7d26-e44d-4cfb-8557-0f22457d1f81","Type":"ContainerDied","Data":"628abd3a5941d937f09c3c69fc320aabad54239e62503f5203a1242a51a97f78"} Sep 17 10:30:24.976426 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:30:24.976345 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-69bbd798b6-f27pz" event={"ID":"6d5f7d26-e44d-4cfb-8557-0f22457d1f81","Type":"ContainerDied","Data":"ac1bbffe051e9b62fb49e980f5b8107a3114b8e3a75e7762bc04a04a83438274"} Sep 17 10:30:24.976426 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:30:24.976366 2581 scope.go:117] "RemoveContainer" containerID="628abd3a5941d937f09c3c69fc320aabad54239e62503f5203a1242a51a97f78" Sep 17 10:30:24.977580 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:30:24.977553 2581 generic.go:356] "Generic (PLEG): container finished" podID="edf43eab-2a20-4f46-b694-2fd0cc8dcefb" containerID="3f4915e859c2b34d9c88f2f89bde63d51323af674a8325a86ca8717505ac7ddd" exitCode=0 Sep 17 10:30:24.977665 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:30:24.977598 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-864846d4f5-xkz72" Sep 17 10:30:24.977705 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:30:24.977594 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-864846d4f5-xkz72" event={"ID":"edf43eab-2a20-4f46-b694-2fd0cc8dcefb","Type":"ContainerDied","Data":"3f4915e859c2b34d9c88f2f89bde63d51323af674a8325a86ca8717505ac7ddd"} Sep 17 10:30:24.977741 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:30:24.977710 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-864846d4f5-xkz72" event={"ID":"edf43eab-2a20-4f46-b694-2fd0cc8dcefb","Type":"ContainerDied","Data":"4fd3129b7b70db7c76dfec268b952b01d050b5ecc17596c57b79f2d8eb0213b5"} Sep 17 10:30:24.985673 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:30:24.985657 2581 scope.go:117] "RemoveContainer" containerID="628abd3a5941d937f09c3c69fc320aabad54239e62503f5203a1242a51a97f78" Sep 17 10:30:24.985965 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:30:24.985946 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"628abd3a5941d937f09c3c69fc320aabad54239e62503f5203a1242a51a97f78\": container with ID starting with 628abd3a5941d937f09c3c69fc320aabad54239e62503f5203a1242a51a97f78 not found: ID does not exist" containerID="628abd3a5941d937f09c3c69fc320aabad54239e62503f5203a1242a51a97f78" Sep 17 10:30:24.986014 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:30:24.985974 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"628abd3a5941d937f09c3c69fc320aabad54239e62503f5203a1242a51a97f78"} err="failed to get container status \"628abd3a5941d937f09c3c69fc320aabad54239e62503f5203a1242a51a97f78\": rpc error: code = NotFound desc = could not find container \"628abd3a5941d937f09c3c69fc320aabad54239e62503f5203a1242a51a97f78\": container with ID starting with 628abd3a5941d937f09c3c69fc320aabad54239e62503f5203a1242a51a97f78 not found: ID does not exist" Sep 17 10:30:24.986014 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:30:24.985991 2581 scope.go:117] "RemoveContainer" containerID="3f4915e859c2b34d9c88f2f89bde63d51323af674a8325a86ca8717505ac7ddd" Sep 17 10:30:24.993329 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:30:24.993314 2581 scope.go:117] "RemoveContainer" containerID="3f4915e859c2b34d9c88f2f89bde63d51323af674a8325a86ca8717505ac7ddd" Sep 17 10:30:24.993587 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:30:24.993571 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f4915e859c2b34d9c88f2f89bde63d51323af674a8325a86ca8717505ac7ddd\": container with ID starting with 3f4915e859c2b34d9c88f2f89bde63d51323af674a8325a86ca8717505ac7ddd not found: ID does not exist" containerID="3f4915e859c2b34d9c88f2f89bde63d51323af674a8325a86ca8717505ac7ddd" Sep 17 10:30:24.993634 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:30:24.993593 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f4915e859c2b34d9c88f2f89bde63d51323af674a8325a86ca8717505ac7ddd"} err="failed to get container status \"3f4915e859c2b34d9c88f2f89bde63d51323af674a8325a86ca8717505ac7ddd\": rpc error: code = NotFound desc = could not find container \"3f4915e859c2b34d9c88f2f89bde63d51323af674a8325a86ca8717505ac7ddd\": container with ID starting with 3f4915e859c2b34d9c88f2f89bde63d51323af674a8325a86ca8717505ac7ddd not found: ID does not exist" Sep 17 10:30:24.998934 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:30:24.998910 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-f27pz"] Sep 17 10:30:25.002825 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:30:25.002804 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-f27pz"] Sep 17 10:30:25.014041 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:30:25.014015 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-xkz72"] Sep 17 10:30:25.016010 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:30:25.015987 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-xkz72"] Sep 17 10:30:25.205407 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:30:25.205323 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-driver-ranking-66c7d475f4-npsl9" podUID="ec179b30-fa6c-4c70-881a-18b96573a475" containerName="online" probeResult="failure" output="Get \"https://10.134.0.34:6567/health\": dial tcp 10.134.0.34:6567: connect: connection refused" Sep 17 10:30:25.943378 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:30:25.943347 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d5f7d26-e44d-4cfb-8557-0f22457d1f81" path="/var/lib/kubelet/pods/6d5f7d26-e44d-4cfb-8557-0f22457d1f81/volumes" Sep 17 10:30:25.943723 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:30:25.943668 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="edf43eab-2a20-4f46-b694-2fd0cc8dcefb" path="/var/lib/kubelet/pods/edf43eab-2a20-4f46-b694-2fd0cc8dcefb/volumes" Sep 17 10:30:26.966677 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:30:26.966633 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-credit-scoring-5796b44ff7-55v9h" podUID="2cde7ec7-fb5f-4f2f-b660-fb9099eb1f1d" containerName="online" probeResult="failure" output="Get \"https://10.134.0.33:6567/health\": dial tcp 10.134.0.33:6567: connect: connection refused" Sep 17 10:30:35.205671 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:30:35.205637 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-driver-ranking-66c7d475f4-npsl9" podUID="ec179b30-fa6c-4c70-881a-18b96573a475" containerName="online" probeResult="failure" output="Get \"https://10.134.0.34:6567/health\": dial tcp 10.134.0.34:6567: connect: connection refused" Sep 17 10:30:36.966845 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:30:36.966806 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-credit-scoring-5796b44ff7-55v9h" podUID="2cde7ec7-fb5f-4f2f-b660-fb9099eb1f1d" containerName="online" probeResult="failure" output="Get \"https://10.134.0.33:6567/health\": dial tcp 10.134.0.33:6567: connect: connection refused" Sep 17 10:30:45.205680 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:30:45.205626 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-driver-ranking-66c7d475f4-npsl9" podUID="ec179b30-fa6c-4c70-881a-18b96573a475" containerName="online" probeResult="failure" output="Get \"https://10.134.0.34:6567/health\": dial tcp 10.134.0.34:6567: connect: connection refused" Sep 17 10:30:45.206075 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:30:45.205766 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-driver-ranking-66c7d475f4-npsl9" Sep 17 10:30:45.911117 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:30:45.911083 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-b9wnf_762d3e10-7b7d-4ce9-9773-90a4fe41b474/console-operator/2.log" Sep 17 10:30:45.912933 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:30:45.912910 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-b9wnf_762d3e10-7b7d-4ce9-9773-90a4fe41b474/console-operator/2.log" Sep 17 10:30:46.966605 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:30:46.966558 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-credit-scoring-5796b44ff7-55v9h" podUID="2cde7ec7-fb5f-4f2f-b660-fb9099eb1f1d" containerName="online" probeResult="failure" output="Get \"https://10.134.0.33:6567/health\": dial tcp 10.134.0.33:6567: connect: connection refused" Sep 17 10:30:46.966998 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:30:46.966696 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-credit-scoring-5796b44ff7-55v9h" Sep 17 10:30:52.043893 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:30:52.043500 2581 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cde7ec7_fb5f_4f2f_b660_fb9099eb1f1d.slice/crio-171ac6eb85463cfdf7ae854fea5e537c41b23592b13deacb133f10029a2d2adf.scope\": RecentStats: unable to find data in memory cache]" Sep 17 10:30:52.085495 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:30:52.085467 2581 generic.go:356] "Generic (PLEG): container finished" podID="ec179b30-fa6c-4c70-881a-18b96573a475" containerID="c9d005ab04a7a59eb91748abf0d73b1d737a3548f77e50fc71025ecd522c5c29" exitCode=137 Sep 17 10:30:52.085657 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:30:52.085540 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-66c7d475f4-npsl9" event={"ID":"ec179b30-fa6c-4c70-881a-18b96573a475","Type":"ContainerDied","Data":"c9d005ab04a7a59eb91748abf0d73b1d737a3548f77e50fc71025ecd522c5c29"} Sep 17 10:30:52.088065 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:30:52.088041 2581 generic.go:356] "Generic (PLEG): container finished" podID="2cde7ec7-fb5f-4f2f-b660-fb9099eb1f1d" containerID="171ac6eb85463cfdf7ae854fea5e537c41b23592b13deacb133f10029a2d2adf" exitCode=137 Sep 17 10:30:52.088188 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:30:52.088089 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-5796b44ff7-55v9h" event={"ID":"2cde7ec7-fb5f-4f2f-b660-fb9099eb1f1d","Type":"ContainerDied","Data":"171ac6eb85463cfdf7ae854fea5e537c41b23592b13deacb133f10029a2d2adf"} Sep 17 10:30:52.642960 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:30:52.642935 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-66c7d475f4-npsl9" Sep 17 10:30:52.645707 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:30:52.645690 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-5796b44ff7-55v9h" Sep 17 10:30:52.686992 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:30:52.686963 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/ec179b30-fa6c-4c70-881a-18b96573a475-online-tls\") pod \"ec179b30-fa6c-4c70-881a-18b96573a475\" (UID: \"ec179b30-fa6c-4c70-881a-18b96573a475\") " Sep 17 10:30:52.687172 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:30:52.687016 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/ec179b30-fa6c-4c70-881a-18b96573a475-feast-data\") pod \"ec179b30-fa6c-4c70-881a-18b96573a475\" (UID: \"ec179b30-fa6c-4c70-881a-18b96573a475\") " Sep 17 10:30:52.687172 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:30:52.687039 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v9bfn\" (UniqueName: \"kubernetes.io/projected/2cde7ec7-fb5f-4f2f-b660-fb9099eb1f1d-kube-api-access-v9bfn\") pod \"2cde7ec7-fb5f-4f2f-b660-fb9099eb1f1d\" (UID: \"2cde7ec7-fb5f-4f2f-b660-fb9099eb1f1d\") " Sep 17 10:30:52.687172 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:30:52.687103 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cfdgc\" (UniqueName: \"kubernetes.io/projected/ec179b30-fa6c-4c70-881a-18b96573a475-kube-api-access-cfdgc\") pod \"ec179b30-fa6c-4c70-881a-18b96573a475\" (UID: \"ec179b30-fa6c-4c70-881a-18b96573a475\") " Sep 17 10:30:52.687172 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:30:52.687138 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/2cde7ec7-fb5f-4f2f-b660-fb9099eb1f1d-feast-data\") pod \"2cde7ec7-fb5f-4f2f-b660-fb9099eb1f1d\" (UID: \"2cde7ec7-fb5f-4f2f-b660-fb9099eb1f1d\") " Sep 17 10:30:52.687399 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:30:52.687292 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/ec179b30-fa6c-4c70-881a-18b96573a475-registry-tls\") pod \"ec179b30-fa6c-4c70-881a-18b96573a475\" (UID: \"ec179b30-fa6c-4c70-881a-18b96573a475\") " Sep 17 10:30:52.687399 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:30:52.687335 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/2cde7ec7-fb5f-4f2f-b660-fb9099eb1f1d-online-tls\") pod \"2cde7ec7-fb5f-4f2f-b660-fb9099eb1f1d\" (UID: \"2cde7ec7-fb5f-4f2f-b660-fb9099eb1f1d\") " Sep 17 10:30:52.687399 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:30:52.687358 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/2cde7ec7-fb5f-4f2f-b660-fb9099eb1f1d-registry-tls\") pod \"2cde7ec7-fb5f-4f2f-b660-fb9099eb1f1d\" (UID: \"2cde7ec7-fb5f-4f2f-b660-fb9099eb1f1d\") " Sep 17 10:30:52.687738 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:30:52.687562 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec179b30-fa6c-4c70-881a-18b96573a475-feast-data" (OuterVolumeSpecName: "feast-data") pod "ec179b30-fa6c-4c70-881a-18b96573a475" (UID: "ec179b30-fa6c-4c70-881a-18b96573a475"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 10:30:52.687738 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:30:52.687644 2581 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/ec179b30-fa6c-4c70-881a-18b96573a475-feast-data\") on node \"ip-10-0-137-203.ec2.internal\" DevicePath \"\"" Sep 17 10:30:52.690063 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:30:52.690020 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec179b30-fa6c-4c70-881a-18b96573a475-kube-api-access-cfdgc" (OuterVolumeSpecName: "kube-api-access-cfdgc") pod "ec179b30-fa6c-4c70-881a-18b96573a475" (UID: "ec179b30-fa6c-4c70-881a-18b96573a475"). InnerVolumeSpecName "kube-api-access-cfdgc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 10:30:52.690188 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:30:52.690108 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec179b30-fa6c-4c70-881a-18b96573a475-online-tls" (OuterVolumeSpecName: "online-tls") pod "ec179b30-fa6c-4c70-881a-18b96573a475" (UID: "ec179b30-fa6c-4c70-881a-18b96573a475"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 10:30:52.690278 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:30:52.690254 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cde7ec7-fb5f-4f2f-b660-fb9099eb1f1d-kube-api-access-v9bfn" (OuterVolumeSpecName: "kube-api-access-v9bfn") pod "2cde7ec7-fb5f-4f2f-b660-fb9099eb1f1d" (UID: "2cde7ec7-fb5f-4f2f-b660-fb9099eb1f1d"). InnerVolumeSpecName "kube-api-access-v9bfn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 10:30:52.690622 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:30:52.690599 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec179b30-fa6c-4c70-881a-18b96573a475-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "ec179b30-fa6c-4c70-881a-18b96573a475" (UID: "ec179b30-fa6c-4c70-881a-18b96573a475"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 10:30:52.690932 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:30:52.690911 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2cde7ec7-fb5f-4f2f-b660-fb9099eb1f1d-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "2cde7ec7-fb5f-4f2f-b660-fb9099eb1f1d" (UID: "2cde7ec7-fb5f-4f2f-b660-fb9099eb1f1d"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 10:30:52.691005 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:30:52.690935 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2cde7ec7-fb5f-4f2f-b660-fb9099eb1f1d-online-tls" (OuterVolumeSpecName: "online-tls") pod "2cde7ec7-fb5f-4f2f-b660-fb9099eb1f1d" (UID: "2cde7ec7-fb5f-4f2f-b660-fb9099eb1f1d"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 10:30:52.697179 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:30:52.697158 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cde7ec7-fb5f-4f2f-b660-fb9099eb1f1d-feast-data" (OuterVolumeSpecName: "feast-data") pod "2cde7ec7-fb5f-4f2f-b660-fb9099eb1f1d" (UID: "2cde7ec7-fb5f-4f2f-b660-fb9099eb1f1d"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 10:30:52.789046 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:30:52.788958 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfdgc\" (UniqueName: \"kubernetes.io/projected/ec179b30-fa6c-4c70-881a-18b96573a475-kube-api-access-cfdgc\") on node \"ip-10-0-137-203.ec2.internal\" DevicePath \"\"" Sep 17 10:30:52.789046 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:30:52.788991 2581 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/2cde7ec7-fb5f-4f2f-b660-fb9099eb1f1d-feast-data\") on node \"ip-10-0-137-203.ec2.internal\" DevicePath \"\"" Sep 17 10:30:52.789046 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:30:52.789001 2581 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/ec179b30-fa6c-4c70-881a-18b96573a475-registry-tls\") on node \"ip-10-0-137-203.ec2.internal\" DevicePath \"\"" Sep 17 10:30:52.789046 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:30:52.789009 2581 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/2cde7ec7-fb5f-4f2f-b660-fb9099eb1f1d-online-tls\") on node \"ip-10-0-137-203.ec2.internal\" DevicePath \"\"" Sep 17 10:30:52.789046 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:30:52.789017 2581 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/2cde7ec7-fb5f-4f2f-b660-fb9099eb1f1d-registry-tls\") on node \"ip-10-0-137-203.ec2.internal\" DevicePath \"\"" Sep 17 10:30:52.789046 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:30:52.789025 2581 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/ec179b30-fa6c-4c70-881a-18b96573a475-online-tls\") on node \"ip-10-0-137-203.ec2.internal\" DevicePath \"\"" Sep 17 10:30:52.789046 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:30:52.789033 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9bfn\" (UniqueName: \"kubernetes.io/projected/2cde7ec7-fb5f-4f2f-b660-fb9099eb1f1d-kube-api-access-v9bfn\") on node \"ip-10-0-137-203.ec2.internal\" DevicePath \"\"" Sep 17 10:30:53.094120 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:30:53.094034 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-66c7d475f4-npsl9" Sep 17 10:30:53.094578 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:30:53.094029 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-66c7d475f4-npsl9" event={"ID":"ec179b30-fa6c-4c70-881a-18b96573a475","Type":"ContainerDied","Data":"f47b16b8d0e4b38fc04b7b8e2838a7f5c824c44bacd8562e6ad92ae86a1fafc4"} Sep 17 10:30:53.094578 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:30:53.094182 2581 scope.go:117] "RemoveContainer" containerID="cf1f626b1fb04162bac8c7120fe53ddedda713f66b8b11f16d64b49e0a7c5f1d" Sep 17 10:30:53.096363 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:30:53.096337 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-5796b44ff7-55v9h" event={"ID":"2cde7ec7-fb5f-4f2f-b660-fb9099eb1f1d","Type":"ContainerDied","Data":"ad881c36b1bc67694f2acd07d5635bcf57bef34629873145e516cfdb08d1ef56"} Sep 17 10:30:53.096474 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:30:53.096391 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-5796b44ff7-55v9h" Sep 17 10:30:53.104122 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:30:53.104102 2581 scope.go:117] "RemoveContainer" containerID="c9d005ab04a7a59eb91748abf0d73b1d737a3548f77e50fc71025ecd522c5c29" Sep 17 10:30:53.112188 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:30:53.112169 2581 scope.go:117] "RemoveContainer" containerID="d71c67e8e1e87f6c90af7af030d68f922226f6aecbc6ff095f03a63e90a0380d" Sep 17 10:30:53.120183 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:30:53.120161 2581 scope.go:117] "RemoveContainer" containerID="fba06518aa62f05ff65ca4f7f6e5d43d52f189e74d00b5660f2e9e35f262417a" Sep 17 10:30:53.120509 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:30:53.120485 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-66c7d475f4-npsl9"] Sep 17 10:30:53.123687 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:30:53.123662 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-66c7d475f4-npsl9"] Sep 17 10:30:53.127950 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:30:53.127931 2581 scope.go:117] "RemoveContainer" containerID="29b554c1a64dcf65ca8c498a1cade999404fb3cbb31c564ad48511852da6b94f" Sep 17 10:30:53.135688 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:30:53.135672 2581 scope.go:117] "RemoveContainer" containerID="171ac6eb85463cfdf7ae854fea5e537c41b23592b13deacb133f10029a2d2adf" Sep 17 10:30:53.136210 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:30:53.136188 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-5796b44ff7-55v9h"] Sep 17 10:30:53.140750 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:30:53.140731 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-5796b44ff7-55v9h"] Sep 17 10:30:53.143915 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:30:53.143894 2581 scope.go:117] "RemoveContainer" containerID="96264d0b755d11b69c9cf6b5fdb890c1a173512fe9a0048577c35abc36c169ba" Sep 17 10:30:53.153987 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:30:53.153588 2581 scope.go:117] "RemoveContainer" containerID="dcb5eeed3349e76a8c2b5386e564f8bf91fefa830255f395e9f668c92be3ab09" Sep 17 10:30:53.943897 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:30:53.943865 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2cde7ec7-fb5f-4f2f-b660-fb9099eb1f1d" path="/var/lib/kubelet/pods/2cde7ec7-fb5f-4f2f-b660-fb9099eb1f1d/volumes" Sep 17 10:30:53.944359 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:30:53.944346 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec179b30-fa6c-4c70-881a-18b96573a475" path="/var/lib/kubelet/pods/ec179b30-fa6c-4c70-881a-18b96573a475/volumes" Sep 17 10:31:01.078574 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:31:01.078534 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-h9vhb"] Sep 17 10:31:01.078958 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:31:01.078849 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec179b30-fa6c-4c70-881a-18b96573a475" containerName="feast-init" Sep 17 10:31:01.078958 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:31:01.078862 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec179b30-fa6c-4c70-881a-18b96573a475" containerName="feast-init" Sep 17 10:31:01.078958 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:31:01.078880 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2cde7ec7-fb5f-4f2f-b660-fb9099eb1f1d" containerName="registry" Sep 17 10:31:01.078958 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:31:01.078885 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="2cde7ec7-fb5f-4f2f-b660-fb9099eb1f1d" containerName="registry" Sep 17 10:31:01.078958 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:31:01.078895 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec179b30-fa6c-4c70-881a-18b96573a475" containerName="feast-apply" Sep 17 10:31:01.078958 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:31:01.078904 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec179b30-fa6c-4c70-881a-18b96573a475" containerName="feast-apply" Sep 17 10:31:01.078958 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:31:01.078914 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec179b30-fa6c-4c70-881a-18b96573a475" containerName="registry" Sep 17 10:31:01.078958 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:31:01.078920 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec179b30-fa6c-4c70-881a-18b96573a475" containerName="registry" Sep 17 10:31:01.078958 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:31:01.078926 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec179b30-fa6c-4c70-881a-18b96573a475" containerName="online" Sep 17 10:31:01.078958 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:31:01.078933 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec179b30-fa6c-4c70-881a-18b96573a475" containerName="online" Sep 17 10:31:01.078958 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:31:01.078944 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="edf43eab-2a20-4f46-b694-2fd0cc8dcefb" containerName="postgres" Sep 17 10:31:01.078958 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:31:01.078949 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="edf43eab-2a20-4f46-b694-2fd0cc8dcefb" containerName="postgres" Sep 17 10:31:01.078958 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:31:01.078955 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2cde7ec7-fb5f-4f2f-b660-fb9099eb1f1d" containerName="feast-init" Sep 17 10:31:01.078958 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:31:01.078960 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="2cde7ec7-fb5f-4f2f-b660-fb9099eb1f1d" containerName="feast-init" Sep 17 10:31:01.079421 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:31:01.078971 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d5f7d26-e44d-4cfb-8557-0f22457d1f81" containerName="redis" Sep 17 10:31:01.079421 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:31:01.078979 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d5f7d26-e44d-4cfb-8557-0f22457d1f81" containerName="redis" Sep 17 10:31:01.079421 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:31:01.078987 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2cde7ec7-fb5f-4f2f-b660-fb9099eb1f1d" containerName="online" Sep 17 10:31:01.079421 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:31:01.078993 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="2cde7ec7-fb5f-4f2f-b660-fb9099eb1f1d" containerName="online" Sep 17 10:31:01.079421 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:31:01.079001 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2cde7ec7-fb5f-4f2f-b660-fb9099eb1f1d" containerName="feast-apply" Sep 17 10:31:01.079421 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:31:01.079006 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="2cde7ec7-fb5f-4f2f-b660-fb9099eb1f1d" containerName="feast-apply" Sep 17 10:31:01.079421 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:31:01.079062 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="edf43eab-2a20-4f46-b694-2fd0cc8dcefb" containerName="postgres" Sep 17 10:31:01.079421 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:31:01.079069 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec179b30-fa6c-4c70-881a-18b96573a475" containerName="online" Sep 17 10:31:01.079421 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:31:01.079084 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d5f7d26-e44d-4cfb-8557-0f22457d1f81" containerName="redis" Sep 17 10:31:01.079421 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:31:01.079091 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="2cde7ec7-fb5f-4f2f-b660-fb9099eb1f1d" containerName="registry" Sep 17 10:31:01.079421 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:31:01.079096 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="2cde7ec7-fb5f-4f2f-b660-fb9099eb1f1d" containerName="online" Sep 17 10:31:01.079421 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:31:01.079101 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec179b30-fa6c-4c70-881a-18b96573a475" containerName="registry" Sep 17 10:31:01.084856 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:31:01.084838 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h9vhb" Sep 17 10:31:01.090492 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:31:01.090468 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-online-tls\"" Sep 17 10:31:01.090492 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:31:01.090486 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-offline-tls\"" Sep 17 10:31:01.090749 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:31:01.090732 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\"" Sep 17 10:31:01.090852 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:31:01.090756 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-registry-tls\"" Sep 17 10:31:01.091237 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:31:01.090995 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-s8cpg\"" Sep 17 10:31:01.091850 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:31:01.091830 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-ui-tls\"" Sep 17 10:31:01.091956 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:31:01.091878 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\"" Sep 17 10:31:01.099457 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:31:01.099433 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-h9vhb"] Sep 17 10:31:01.155988 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:31:01.155945 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/e7c593f1-ac96-40da-94b7-93fcff96f6d4-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-h9vhb\" (UID: \"e7c593f1-ac96-40da-94b7-93fcff96f6d4\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h9vhb" Sep 17 10:31:01.156160 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:31:01.156003 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/e7c593f1-ac96-40da-94b7-93fcff96f6d4-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-h9vhb\" (UID: \"e7c593f1-ac96-40da-94b7-93fcff96f6d4\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h9vhb" Sep 17 10:31:01.156160 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:31:01.156031 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/e7c593f1-ac96-40da-94b7-93fcff96f6d4-feast-data\") pod \"feast-simple-feast-setup-57576b8fcf-h9vhb\" (UID: \"e7c593f1-ac96-40da-94b7-93fcff96f6d4\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h9vhb" Sep 17 10:31:01.156160 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:31:01.156074 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/e7c593f1-ac96-40da-94b7-93fcff96f6d4-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-h9vhb\" (UID: \"e7c593f1-ac96-40da-94b7-93fcff96f6d4\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h9vhb" Sep 17 10:31:01.156160 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:31:01.156130 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/e7c593f1-ac96-40da-94b7-93fcff96f6d4-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-h9vhb\" (UID: \"e7c593f1-ac96-40da-94b7-93fcff96f6d4\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h9vhb" Sep 17 10:31:01.156296 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:31:01.156192 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8g6p8\" (UniqueName: \"kubernetes.io/projected/e7c593f1-ac96-40da-94b7-93fcff96f6d4-kube-api-access-8g6p8\") pod \"feast-simple-feast-setup-57576b8fcf-h9vhb\" (UID: \"e7c593f1-ac96-40da-94b7-93fcff96f6d4\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h9vhb" Sep 17 10:31:01.257332 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:31:01.257294 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/e7c593f1-ac96-40da-94b7-93fcff96f6d4-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-h9vhb\" (UID: \"e7c593f1-ac96-40da-94b7-93fcff96f6d4\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h9vhb" Sep 17 10:31:01.257332 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:31:01.257337 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/e7c593f1-ac96-40da-94b7-93fcff96f6d4-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-h9vhb\" (UID: \"e7c593f1-ac96-40da-94b7-93fcff96f6d4\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h9vhb" Sep 17 10:31:01.257611 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:31:01.257357 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/e7c593f1-ac96-40da-94b7-93fcff96f6d4-feast-data\") pod \"feast-simple-feast-setup-57576b8fcf-h9vhb\" (UID: \"e7c593f1-ac96-40da-94b7-93fcff96f6d4\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h9vhb" Sep 17 10:31:01.257611 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:31:01.257412 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/e7c593f1-ac96-40da-94b7-93fcff96f6d4-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-h9vhb\" (UID: \"e7c593f1-ac96-40da-94b7-93fcff96f6d4\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h9vhb" Sep 17 10:31:01.257611 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:31:01.257436 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/e7c593f1-ac96-40da-94b7-93fcff96f6d4-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-h9vhb\" (UID: \"e7c593f1-ac96-40da-94b7-93fcff96f6d4\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h9vhb" Sep 17 10:31:01.257611 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:31:01.257462 2581 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-online-tls: secret "feast-simple-feast-setup-online-tls" not found Sep 17 10:31:01.257611 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:31:01.257501 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8g6p8\" (UniqueName: \"kubernetes.io/projected/e7c593f1-ac96-40da-94b7-93fcff96f6d4-kube-api-access-8g6p8\") pod \"feast-simple-feast-setup-57576b8fcf-h9vhb\" (UID: \"e7c593f1-ac96-40da-94b7-93fcff96f6d4\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h9vhb" Sep 17 10:31:01.257611 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:31:01.257539 2581 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-registry-tls: secret "feast-simple-feast-setup-registry-tls" not found Sep 17 10:31:01.257611 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:31:01.257540 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7c593f1-ac96-40da-94b7-93fcff96f6d4-online-tls podName:e7c593f1-ac96-40da-94b7-93fcff96f6d4 nodeName:}" failed. No retries permitted until 2026-09-17 10:31:01.757520573 +0000 UTC m=+1216.306077675 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/e7c593f1-ac96-40da-94b7-93fcff96f6d4-online-tls") pod "feast-simple-feast-setup-57576b8fcf-h9vhb" (UID: "e7c593f1-ac96-40da-94b7-93fcff96f6d4") : secret "feast-simple-feast-setup-online-tls" not found Sep 17 10:31:01.257985 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:31:01.257623 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7c593f1-ac96-40da-94b7-93fcff96f6d4-registry-tls podName:e7c593f1-ac96-40da-94b7-93fcff96f6d4 nodeName:}" failed. No retries permitted until 2026-09-17 10:31:01.757608648 +0000 UTC m=+1216.306165737 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/e7c593f1-ac96-40da-94b7-93fcff96f6d4-registry-tls") pod "feast-simple-feast-setup-57576b8fcf-h9vhb" (UID: "e7c593f1-ac96-40da-94b7-93fcff96f6d4") : secret "feast-simple-feast-setup-registry-tls" not found Sep 17 10:31:01.257985 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:31:01.257632 2581 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-offline-tls: secret "feast-simple-feast-setup-offline-tls" not found Sep 17 10:31:01.257985 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:31:01.257697 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7c593f1-ac96-40da-94b7-93fcff96f6d4-offline-tls podName:e7c593f1-ac96-40da-94b7-93fcff96f6d4 nodeName:}" failed. No retries permitted until 2026-09-17 10:31:01.757678834 +0000 UTC m=+1216.306235929 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/e7c593f1-ac96-40da-94b7-93fcff96f6d4-offline-tls") pod "feast-simple-feast-setup-57576b8fcf-h9vhb" (UID: "e7c593f1-ac96-40da-94b7-93fcff96f6d4") : secret "feast-simple-feast-setup-offline-tls" not found Sep 17 10:31:01.257985 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:31:01.257859 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/e7c593f1-ac96-40da-94b7-93fcff96f6d4-feast-data\") pod \"feast-simple-feast-setup-57576b8fcf-h9vhb\" (UID: \"e7c593f1-ac96-40da-94b7-93fcff96f6d4\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h9vhb" Sep 17 10:31:01.259930 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:31:01.259909 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/e7c593f1-ac96-40da-94b7-93fcff96f6d4-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-h9vhb\" (UID: \"e7c593f1-ac96-40da-94b7-93fcff96f6d4\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h9vhb" Sep 17 10:31:01.269915 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:31:01.269878 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8g6p8\" (UniqueName: \"kubernetes.io/projected/e7c593f1-ac96-40da-94b7-93fcff96f6d4-kube-api-access-8g6p8\") pod \"feast-simple-feast-setup-57576b8fcf-h9vhb\" (UID: \"e7c593f1-ac96-40da-94b7-93fcff96f6d4\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h9vhb" Sep 17 10:31:01.762470 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:31:01.762431 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/e7c593f1-ac96-40da-94b7-93fcff96f6d4-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-h9vhb\" (UID: \"e7c593f1-ac96-40da-94b7-93fcff96f6d4\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h9vhb" Sep 17 10:31:01.762647 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:31:01.762498 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/e7c593f1-ac96-40da-94b7-93fcff96f6d4-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-h9vhb\" (UID: \"e7c593f1-ac96-40da-94b7-93fcff96f6d4\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h9vhb" Sep 17 10:31:01.762647 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:31:01.762523 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/e7c593f1-ac96-40da-94b7-93fcff96f6d4-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-h9vhb\" (UID: \"e7c593f1-ac96-40da-94b7-93fcff96f6d4\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h9vhb" Sep 17 10:31:01.764902 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:31:01.764873 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/e7c593f1-ac96-40da-94b7-93fcff96f6d4-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-h9vhb\" (UID: \"e7c593f1-ac96-40da-94b7-93fcff96f6d4\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h9vhb" Sep 17 10:31:01.764902 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:31:01.764890 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/e7c593f1-ac96-40da-94b7-93fcff96f6d4-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-h9vhb\" (UID: \"e7c593f1-ac96-40da-94b7-93fcff96f6d4\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h9vhb" Sep 17 10:31:01.765050 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:31:01.765003 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/e7c593f1-ac96-40da-94b7-93fcff96f6d4-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-h9vhb\" (UID: \"e7c593f1-ac96-40da-94b7-93fcff96f6d4\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h9vhb" Sep 17 10:31:01.994350 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:31:01.994312 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h9vhb" Sep 17 10:31:02.270755 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:31:02.270728 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-h9vhb"] Sep 17 10:31:02.271415 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:31:02.271391 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode7c593f1_ac96_40da_94b7_93fcff96f6d4.slice/crio-a486b551a0778d334735d2a5920752b2f68fcc2f08a473f31fdc7fb083c9001f WatchSource:0}: Error finding container a486b551a0778d334735d2a5920752b2f68fcc2f08a473f31fdc7fb083c9001f: Status 404 returned error can't find the container with id a486b551a0778d334735d2a5920752b2f68fcc2f08a473f31fdc7fb083c9001f Sep 17 10:31:02.273371 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:31:02.273357 2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 17 10:31:03.137197 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:31:03.137160 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h9vhb" event={"ID":"e7c593f1-ac96-40da-94b7-93fcff96f6d4","Type":"ContainerStarted","Data":"a486b551a0778d334735d2a5920752b2f68fcc2f08a473f31fdc7fb083c9001f"} Sep 17 10:31:16.208278 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:31:16.208247 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h9vhb" event={"ID":"e7c593f1-ac96-40da-94b7-93fcff96f6d4","Type":"ContainerStarted","Data":"bf38c33bfa1b7c869de457f0b1db490207c7442fa661826310937c63c04edbfb"} Sep 17 10:31:21.230526 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:31:21.230488 2581 generic.go:356] "Generic (PLEG): container finished" podID="e7c593f1-ac96-40da-94b7-93fcff96f6d4" containerID="bf38c33bfa1b7c869de457f0b1db490207c7442fa661826310937c63c04edbfb" exitCode=0 Sep 17 10:31:21.230915 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:31:21.230565 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h9vhb" event={"ID":"e7c593f1-ac96-40da-94b7-93fcff96f6d4","Type":"ContainerDied","Data":"bf38c33bfa1b7c869de457f0b1db490207c7442fa661826310937c63c04edbfb"} Sep 17 10:31:22.237037 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:31:22.236994 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h9vhb" event={"ID":"e7c593f1-ac96-40da-94b7-93fcff96f6d4","Type":"ContainerStarted","Data":"1f10f33e468abf1a4269fa0e971962b7465c3ea332b02ce954cf0c5ea9b36b14"} Sep 17 10:31:28.264557 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:31:28.264521 2581 generic.go:356] "Generic (PLEG): container finished" podID="e7c593f1-ac96-40da-94b7-93fcff96f6d4" containerID="1f10f33e468abf1a4269fa0e971962b7465c3ea332b02ce954cf0c5ea9b36b14" exitCode=0 Sep 17 10:31:28.264983 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:31:28.264594 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h9vhb" event={"ID":"e7c593f1-ac96-40da-94b7-93fcff96f6d4","Type":"ContainerDied","Data":"1f10f33e468abf1a4269fa0e971962b7465c3ea332b02ce954cf0c5ea9b36b14"} Sep 17 10:31:29.275244 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:31:29.275200 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h9vhb" event={"ID":"e7c593f1-ac96-40da-94b7-93fcff96f6d4","Type":"ContainerStarted","Data":"696471500c405bdb0ed6e30f020f16a7c0776fc613fdb212a1331a0425888749"} Sep 17 10:31:29.275860 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:31:29.275271 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h9vhb" event={"ID":"e7c593f1-ac96-40da-94b7-93fcff96f6d4","Type":"ContainerStarted","Data":"472b77264ca92b7bc17ebd5aacb9fcaec1505fff57a1a0a929800f97ea27a4c3"} Sep 17 10:31:29.275860 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:31:29.275293 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h9vhb" event={"ID":"e7c593f1-ac96-40da-94b7-93fcff96f6d4","Type":"ContainerStarted","Data":"b132650055509916c24699e9f159c2343b6a27c6f43785e6a9862029f100f0ce"} Sep 17 10:31:29.275860 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:31:29.275311 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h9vhb" event={"ID":"e7c593f1-ac96-40da-94b7-93fcff96f6d4","Type":"ContainerStarted","Data":"dfade198c0a694dde4791930e56a8c82372c8f080823ba65d70620a079d10d2e"} Sep 17 10:31:29.310127 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:31:29.310058 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h9vhb" podStartSLOduration=14.47737415 podStartE2EDuration="28.310037038s" totalImagesPullingTime="13.832662888s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 10:31:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 10:31:29.308678357 +0000 UTC m=+1243.857235468" watchObservedRunningTime="2026-09-17 10:31:29.310037038 +0000 UTC m=+1243.858594150" Sep 17 10:31:31.994444 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:31:31.994400 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h9vhb" Sep 17 10:31:31.995079 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:31:31.995044 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h9vhb" Sep 17 10:31:31.995079 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:31:31.995088 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h9vhb" Sep 17 10:31:31.995278 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:31:31.995107 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h9vhb" Sep 17 10:31:31.995278 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:31:31.995125 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h9vhb" Sep 17 10:31:31.995278 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:31:31.995144 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h9vhb" Sep 17 10:31:31.995278 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:31:31.995163 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h9vhb" Sep 17 10:31:31.995278 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:31:31.995182 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h9vhb" Sep 17 10:31:31.996969 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:31:31.996937 2581 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h9vhb" podUID="e7c593f1-ac96-40da-94b7-93fcff96f6d4" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.35:8443: connect: connection refused" Sep 17 10:31:31.997394 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:31:31.996957 2581 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h9vhb" podUID="e7c593f1-ac96-40da-94b7-93fcff96f6d4" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.35:8816: connect: connection refused" Sep 17 10:31:31.997566 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:31:31.996994 2581 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h9vhb" podUID="e7c593f1-ac96-40da-94b7-93fcff96f6d4" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.35:6571: connect: connection refused" Sep 17 10:31:31.997685 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:31:31.997010 2581 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h9vhb" podUID="e7c593f1-ac96-40da-94b7-93fcff96f6d4" containerName="online" probeResult="failure" output="Get \"https://10.134.0.35:6567/health\": dial tcp 10.134.0.35:6567: connect: connection refused" Sep 17 10:31:34.995451 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:31:34.995400 2581 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h9vhb" podUID="e7c593f1-ac96-40da-94b7-93fcff96f6d4" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.35:8443: connect: connection refused" Sep 17 10:31:34.995451 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:31:34.995436 2581 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h9vhb" podUID="e7c593f1-ac96-40da-94b7-93fcff96f6d4" containerName="online" probeResult="failure" output="Get \"https://10.134.0.35:6567/health\": dial tcp 10.134.0.35:6567: connect: connection refused" Sep 17 10:31:34.996025 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:31:34.995401 2581 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h9vhb" podUID="e7c593f1-ac96-40da-94b7-93fcff96f6d4" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.35:6571: connect: connection refused" Sep 17 10:31:34.996121 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:31:34.996096 2581 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h9vhb" podUID="e7c593f1-ac96-40da-94b7-93fcff96f6d4" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.35:8816: connect: connection refused" Sep 17 10:31:37.995760 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:31:37.995727 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h9vhb" Sep 17 10:31:37.996307 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:31:37.996286 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h9vhb" Sep 17 10:31:37.996426 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:31:37.996414 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h9vhb" Sep 17 10:31:37.997271 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:31:37.997246 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h9vhb" Sep 17 10:31:37.997397 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:31:37.997315 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h9vhb" Sep 17 10:31:37.997397 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:31:37.997366 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h9vhb" Sep 17 10:31:38.002225 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:31:38.002195 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h9vhb" Sep 17 10:31:38.329440 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:31:38.329356 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h9vhb" Sep 17 10:32:40.349730 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:32:40.349695 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-h9vhb"] Sep 17 10:32:40.350303 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:32:40.350026 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h9vhb" podUID="e7c593f1-ac96-40da-94b7-93fcff96f6d4" containerName="registry" containerID="cri-o://dfade198c0a694dde4791930e56a8c82372c8f080823ba65d70620a079d10d2e" gracePeriod=30 Sep 17 10:32:40.350303 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:32:40.350110 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h9vhb" podUID="e7c593f1-ac96-40da-94b7-93fcff96f6d4" containerName="offline" containerID="cri-o://472b77264ca92b7bc17ebd5aacb9fcaec1505fff57a1a0a929800f97ea27a4c3" gracePeriod=30 Sep 17 10:32:40.350303 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:32:40.350210 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h9vhb" podUID="e7c593f1-ac96-40da-94b7-93fcff96f6d4" containerName="online" containerID="cri-o://b132650055509916c24699e9f159c2343b6a27c6f43785e6a9862029f100f0ce" gracePeriod=30 Sep 17 10:32:40.350303 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:32:40.350217 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h9vhb" podUID="e7c593f1-ac96-40da-94b7-93fcff96f6d4" containerName="ui" containerID="cri-o://696471500c405bdb0ed6e30f020f16a7c0776fc613fdb212a1331a0425888749" gracePeriod=30 Sep 17 10:32:40.866131 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:32:40.866086 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-dqn59"] Sep 17 10:32:40.869216 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:32:40.869190 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-dqn59" Sep 17 10:32:40.872099 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:32:40.872080 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-7phtl\"" Sep 17 10:32:40.883425 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:32:40.883401 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-dqn59"] Sep 17 10:32:40.974052 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:32:40.974023 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/e7c9bdd3-0224-4a08-98c1-d0a648f8f70a-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-dqn59\" (UID: \"e7c9bdd3-0224-4a08-98c1-d0a648f8f70a\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-dqn59" Sep 17 10:32:40.974198 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:32:40.974064 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/e7c9bdd3-0224-4a08-98c1-d0a648f8f70a-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-dqn59\" (UID: \"e7c9bdd3-0224-4a08-98c1-d0a648f8f70a\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-dqn59" Sep 17 10:32:40.974198 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:32:40.974082 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77dln\" (UniqueName: \"kubernetes.io/projected/e7c9bdd3-0224-4a08-98c1-d0a648f8f70a-kube-api-access-77dln\") pod \"feast-simple-feast-setup-57576b8fcf-dqn59\" (UID: \"e7c9bdd3-0224-4a08-98c1-d0a648f8f70a\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-dqn59" Sep 17 10:32:40.974198 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:32:40.974160 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/e7c9bdd3-0224-4a08-98c1-d0a648f8f70a-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-dqn59\" (UID: \"e7c9bdd3-0224-4a08-98c1-d0a648f8f70a\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-dqn59" Sep 17 10:32:40.974322 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:32:40.974209 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/e7c9bdd3-0224-4a08-98c1-d0a648f8f70a-feast-data\") pod \"feast-simple-feast-setup-57576b8fcf-dqn59\" (UID: \"e7c9bdd3-0224-4a08-98c1-d0a648f8f70a\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-dqn59" Sep 17 10:32:40.974322 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:32:40.974233 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/e7c9bdd3-0224-4a08-98c1-d0a648f8f70a-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-dqn59\" (UID: \"e7c9bdd3-0224-4a08-98c1-d0a648f8f70a\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-dqn59" Sep 17 10:32:41.075395 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:32:41.075352 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/e7c9bdd3-0224-4a08-98c1-d0a648f8f70a-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-dqn59\" (UID: \"e7c9bdd3-0224-4a08-98c1-d0a648f8f70a\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-dqn59" Sep 17 10:32:41.075588 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:32:41.075411 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/e7c9bdd3-0224-4a08-98c1-d0a648f8f70a-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-dqn59\" (UID: \"e7c9bdd3-0224-4a08-98c1-d0a648f8f70a\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-dqn59" Sep 17 10:32:41.075588 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:32:41.075441 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/e7c9bdd3-0224-4a08-98c1-d0a648f8f70a-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-dqn59\" (UID: \"e7c9bdd3-0224-4a08-98c1-d0a648f8f70a\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-dqn59" Sep 17 10:32:41.075588 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:32:41.075464 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77dln\" (UniqueName: \"kubernetes.io/projected/e7c9bdd3-0224-4a08-98c1-d0a648f8f70a-kube-api-access-77dln\") pod \"feast-simple-feast-setup-57576b8fcf-dqn59\" (UID: \"e7c9bdd3-0224-4a08-98c1-d0a648f8f70a\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-dqn59" Sep 17 10:32:41.075588 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:32:41.075516 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/e7c9bdd3-0224-4a08-98c1-d0a648f8f70a-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-dqn59\" (UID: \"e7c9bdd3-0224-4a08-98c1-d0a648f8f70a\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-dqn59" Sep 17 10:32:41.075588 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:32:41.075557 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/e7c9bdd3-0224-4a08-98c1-d0a648f8f70a-feast-data\") pod \"feast-simple-feast-setup-57576b8fcf-dqn59\" (UID: \"e7c9bdd3-0224-4a08-98c1-d0a648f8f70a\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-dqn59" Sep 17 10:32:41.075917 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:32:41.075600 2581 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-online-tls: secret "feast-simple-feast-setup-online-tls" not found Sep 17 10:32:41.075917 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:32:41.075679 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7c9bdd3-0224-4a08-98c1-d0a648f8f70a-online-tls podName:e7c9bdd3-0224-4a08-98c1-d0a648f8f70a nodeName:}" failed. No retries permitted until 2026-09-17 10:32:41.57566014 +0000 UTC m=+1316.124217230 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/e7c9bdd3-0224-4a08-98c1-d0a648f8f70a-online-tls") pod "feast-simple-feast-setup-57576b8fcf-dqn59" (UID: "e7c9bdd3-0224-4a08-98c1-d0a648f8f70a") : secret "feast-simple-feast-setup-online-tls" not found Sep 17 10:32:41.075917 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:32:41.075684 2581 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-ui-tls: secret "feast-simple-feast-setup-ui-tls" not found Sep 17 10:32:41.075917 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:32:41.075747 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7c9bdd3-0224-4a08-98c1-d0a648f8f70a-ui-tls podName:e7c9bdd3-0224-4a08-98c1-d0a648f8f70a nodeName:}" failed. No retries permitted until 2026-09-17 10:32:41.57573048 +0000 UTC m=+1316.124287587 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "ui-tls" (UniqueName: "kubernetes.io/secret/e7c9bdd3-0224-4a08-98c1-d0a648f8f70a-ui-tls") pod "feast-simple-feast-setup-57576b8fcf-dqn59" (UID: "e7c9bdd3-0224-4a08-98c1-d0a648f8f70a") : secret "feast-simple-feast-setup-ui-tls" not found Sep 17 10:32:41.075917 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:32:41.075603 2581 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-registry-tls: secret "feast-simple-feast-setup-registry-tls" not found Sep 17 10:32:41.075917 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:32:41.075814 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7c9bdd3-0224-4a08-98c1-d0a648f8f70a-registry-tls podName:e7c9bdd3-0224-4a08-98c1-d0a648f8f70a nodeName:}" failed. No retries permitted until 2026-09-17 10:32:41.575775324 +0000 UTC m=+1316.124332413 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/e7c9bdd3-0224-4a08-98c1-d0a648f8f70a-registry-tls") pod "feast-simple-feast-setup-57576b8fcf-dqn59" (UID: "e7c9bdd3-0224-4a08-98c1-d0a648f8f70a") : secret "feast-simple-feast-setup-registry-tls" not found Sep 17 10:32:41.076156 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:32:41.075968 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/e7c9bdd3-0224-4a08-98c1-d0a648f8f70a-feast-data\") pod \"feast-simple-feast-setup-57576b8fcf-dqn59\" (UID: \"e7c9bdd3-0224-4a08-98c1-d0a648f8f70a\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-dqn59" Sep 17 10:32:41.077848 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:32:41.077830 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/e7c9bdd3-0224-4a08-98c1-d0a648f8f70a-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-dqn59\" (UID: \"e7c9bdd3-0224-4a08-98c1-d0a648f8f70a\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-dqn59" Sep 17 10:32:41.085560 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:32:41.085535 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-77dln\" (UniqueName: \"kubernetes.io/projected/e7c9bdd3-0224-4a08-98c1-d0a648f8f70a-kube-api-access-77dln\") pod \"feast-simple-feast-setup-57576b8fcf-dqn59\" (UID: \"e7c9bdd3-0224-4a08-98c1-d0a648f8f70a\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-dqn59" Sep 17 10:32:41.579016 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:32:41.578982 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/e7c9bdd3-0224-4a08-98c1-d0a648f8f70a-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-dqn59\" (UID: \"e7c9bdd3-0224-4a08-98c1-d0a648f8f70a\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-dqn59" Sep 17 10:32:41.579425 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:32:41.579047 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/e7c9bdd3-0224-4a08-98c1-d0a648f8f70a-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-dqn59\" (UID: \"e7c9bdd3-0224-4a08-98c1-d0a648f8f70a\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-dqn59" Sep 17 10:32:41.579425 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:32:41.579071 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/e7c9bdd3-0224-4a08-98c1-d0a648f8f70a-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-dqn59\" (UID: \"e7c9bdd3-0224-4a08-98c1-d0a648f8f70a\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-dqn59" Sep 17 10:32:41.581634 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:32:41.581606 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/e7c9bdd3-0224-4a08-98c1-d0a648f8f70a-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-dqn59\" (UID: \"e7c9bdd3-0224-4a08-98c1-d0a648f8f70a\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-dqn59" Sep 17 10:32:41.581634 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:32:41.581620 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/e7c9bdd3-0224-4a08-98c1-d0a648f8f70a-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-dqn59\" (UID: \"e7c9bdd3-0224-4a08-98c1-d0a648f8f70a\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-dqn59" Sep 17 10:32:41.581829 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:32:41.581682 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/e7c9bdd3-0224-4a08-98c1-d0a648f8f70a-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-dqn59\" (UID: \"e7c9bdd3-0224-4a08-98c1-d0a648f8f70a\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-dqn59" Sep 17 10:32:41.584410 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:32:41.584384 2581 generic.go:356] "Generic (PLEG): container finished" podID="e7c593f1-ac96-40da-94b7-93fcff96f6d4" containerID="696471500c405bdb0ed6e30f020f16a7c0776fc613fdb212a1331a0425888749" exitCode=0 Sep 17 10:32:41.584410 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:32:41.584408 2581 generic.go:356] "Generic (PLEG): container finished" podID="e7c593f1-ac96-40da-94b7-93fcff96f6d4" containerID="b132650055509916c24699e9f159c2343b6a27c6f43785e6a9862029f100f0ce" exitCode=0 Sep 17 10:32:41.584575 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:32:41.584431 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h9vhb" event={"ID":"e7c593f1-ac96-40da-94b7-93fcff96f6d4","Type":"ContainerDied","Data":"696471500c405bdb0ed6e30f020f16a7c0776fc613fdb212a1331a0425888749"} Sep 17 10:32:41.584575 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:32:41.584472 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h9vhb" event={"ID":"e7c593f1-ac96-40da-94b7-93fcff96f6d4","Type":"ContainerDied","Data":"b132650055509916c24699e9f159c2343b6a27c6f43785e6a9862029f100f0ce"} Sep 17 10:32:41.779845 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:32:41.779809 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-dqn59" Sep 17 10:32:42.057197 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:32:42.057172 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-dqn59"] Sep 17 10:32:42.057726 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:32:42.057694 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode7c9bdd3_0224_4a08_98c1_d0a648f8f70a.slice/crio-e9facafb39090e2927fc623d2939a070b9f814fbfffbf7cb645ac9f1f0b56b1f WatchSource:0}: Error finding container e9facafb39090e2927fc623d2939a070b9f814fbfffbf7cb645ac9f1f0b56b1f: Status 404 returned error can't find the container with id e9facafb39090e2927fc623d2939a070b9f814fbfffbf7cb645ac9f1f0b56b1f Sep 17 10:32:42.590666 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:32:42.590627 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-dqn59" event={"ID":"e7c9bdd3-0224-4a08-98c1-d0a648f8f70a","Type":"ContainerStarted","Data":"7d4f33d36d615f0206a33b05be2a8ba8523d842cfc9db802c6099ef0baba91c4"} Sep 17 10:32:42.591062 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:32:42.590682 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-dqn59" event={"ID":"e7c9bdd3-0224-4a08-98c1-d0a648f8f70a","Type":"ContainerStarted","Data":"e9facafb39090e2927fc623d2939a070b9f814fbfffbf7cb645ac9f1f0b56b1f"} Sep 17 10:32:47.612696 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:32:47.612661 2581 generic.go:356] "Generic (PLEG): container finished" podID="e7c9bdd3-0224-4a08-98c1-d0a648f8f70a" containerID="7d4f33d36d615f0206a33b05be2a8ba8523d842cfc9db802c6099ef0baba91c4" exitCode=0 Sep 17 10:32:47.613153 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:32:47.612739 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-dqn59" event={"ID":"e7c9bdd3-0224-4a08-98c1-d0a648f8f70a","Type":"ContainerDied","Data":"7d4f33d36d615f0206a33b05be2a8ba8523d842cfc9db802c6099ef0baba91c4"} Sep 17 10:32:47.997498 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:32:47.997436 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h9vhb" podUID="e7c593f1-ac96-40da-94b7-93fcff96f6d4" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.35:8443: connect: connection refused" Sep 17 10:32:48.326636 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:32:48.326547 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h9vhb" podUID="e7c593f1-ac96-40da-94b7-93fcff96f6d4" containerName="online" probeResult="failure" output="Get \"https://10.134.0.35:6567/health\": dial tcp 10.134.0.35:6567: connect: connection refused" Sep 17 10:32:48.619500 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:32:48.619416 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-dqn59" event={"ID":"e7c9bdd3-0224-4a08-98c1-d0a648f8f70a","Type":"ContainerStarted","Data":"f65f00db289356bacf2cd9adbdf973a0a5806b4274f03d2f60329d055c954073"} Sep 17 10:32:53.642387 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:32:53.642297 2581 generic.go:356] "Generic (PLEG): container finished" podID="e7c9bdd3-0224-4a08-98c1-d0a648f8f70a" containerID="f65f00db289356bacf2cd9adbdf973a0a5806b4274f03d2f60329d055c954073" exitCode=0 Sep 17 10:32:53.642846 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:32:53.642378 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-dqn59" event={"ID":"e7c9bdd3-0224-4a08-98c1-d0a648f8f70a","Type":"ContainerDied","Data":"f65f00db289356bacf2cd9adbdf973a0a5806b4274f03d2f60329d055c954073"} Sep 17 10:32:54.652797 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:32:54.652737 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-dqn59" event={"ID":"e7c9bdd3-0224-4a08-98c1-d0a648f8f70a","Type":"ContainerStarted","Data":"da993cc3c9ece08c90512473d53143cb9a49de4fcbd84230bc2cd2fcf9b28988"} Sep 17 10:32:54.653337 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:32:54.652823 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-dqn59" event={"ID":"e7c9bdd3-0224-4a08-98c1-d0a648f8f70a","Type":"ContainerStarted","Data":"119f23e8169e43350773f0c6b02aa98e75a8cab5db2b96e7480de039485467b0"} Sep 17 10:32:54.653337 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:32:54.652846 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-dqn59" event={"ID":"e7c9bdd3-0224-4a08-98c1-d0a648f8f70a","Type":"ContainerStarted","Data":"dfe8cab6ef9454ae74f7fc17ea20077ea9671f0ca33a148a4c08def24dbeef3a"} Sep 17 10:32:54.653337 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:32:54.652865 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-dqn59" event={"ID":"e7c9bdd3-0224-4a08-98c1-d0a648f8f70a","Type":"ContainerStarted","Data":"3a531941343edd3826bd3f9b22a39350d9e7ee225706dfaa8768cfa28f7b713d"} Sep 17 10:32:54.684600 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:32:54.684537 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-dqn59" podStartSLOduration=14.684514634 podStartE2EDuration="14.684514634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 10:32:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 10:32:54.680740869 +0000 UTC m=+1329.229297982" watchObservedRunningTime="2026-09-17 10:32:54.684514634 +0000 UTC m=+1329.233071746" Sep 17 10:32:56.780530 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:32:56.780492 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-dqn59" Sep 17 10:32:56.781138 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:32:56.781107 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-dqn59" Sep 17 10:32:56.781138 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:32:56.781140 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-dqn59" Sep 17 10:32:56.781321 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:32:56.781157 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-dqn59" Sep 17 10:32:56.782877 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:32:56.782840 2581 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-dqn59" podUID="e7c9bdd3-0224-4a08-98c1-d0a648f8f70a" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.36:8816: connect: connection refused" Sep 17 10:32:56.783104 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:32:56.783068 2581 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-dqn59" podUID="e7c9bdd3-0224-4a08-98c1-d0a648f8f70a" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.36:6571: connect: connection refused" Sep 17 10:32:56.783277 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:32:56.783094 2581 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-dqn59" podUID="e7c9bdd3-0224-4a08-98c1-d0a648f8f70a" containerName="online" probeResult="failure" output="Get \"https://10.134.0.36:6567/health\": dial tcp 10.134.0.36:6567: connect: connection refused" Sep 17 10:32:56.783402 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:32:56.783155 2581 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-dqn59" podUID="e7c9bdd3-0224-4a08-98c1-d0a648f8f70a" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.36:8443: connect: connection refused" Sep 17 10:32:57.997527 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:32:57.997172 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h9vhb" podUID="e7c593f1-ac96-40da-94b7-93fcff96f6d4" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.35:8443: connect: connection refused" Sep 17 10:32:58.326223 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:32:58.326118 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h9vhb" podUID="e7c593f1-ac96-40da-94b7-93fcff96f6d4" containerName="online" probeResult="failure" output="Get \"https://10.134.0.35:6567/health\": dial tcp 10.134.0.35:6567: connect: connection refused" Sep 17 10:32:59.780952 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:32:59.780900 2581 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-dqn59" podUID="e7c9bdd3-0224-4a08-98c1-d0a648f8f70a" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.36:6571: connect: connection refused" Sep 17 10:32:59.781561 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:32:59.780926 2581 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-dqn59" podUID="e7c9bdd3-0224-4a08-98c1-d0a648f8f70a" containerName="online" probeResult="failure" output="Get \"https://10.134.0.36:6567/health\": dial tcp 10.134.0.36:6567: connect: connection refused" Sep 17 10:32:59.781702 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:32:59.781214 2581 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-dqn59" podUID="e7c9bdd3-0224-4a08-98c1-d0a648f8f70a" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.36:8443: connect: connection refused" Sep 17 10:32:59.781702 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:32:59.781291 2581 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-dqn59" podUID="e7c9bdd3-0224-4a08-98c1-d0a648f8f70a" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.36:8816: connect: connection refused" Sep 17 10:33:01.780478 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:33:01.780434 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-dqn59" Sep 17 10:33:01.780478 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:33:01.780488 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-dqn59" Sep 17 10:33:01.780914 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:33:01.780505 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-dqn59" Sep 17 10:33:01.780914 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:33:01.780516 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-dqn59" Sep 17 10:33:02.781442 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:33:02.781404 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-dqn59" Sep 17 10:33:02.781869 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:33:02.781728 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-dqn59" Sep 17 10:33:02.782246 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:33:02.782222 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-dqn59" Sep 17 10:33:02.782358 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:33:02.782287 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-dqn59" Sep 17 10:33:02.782358 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:33:02.782322 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-dqn59" Sep 17 10:33:02.786857 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:33:02.786838 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-dqn59" Sep 17 10:33:03.703637 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:33:03.703605 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-dqn59" Sep 17 10:33:03.706406 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:33:03.706385 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-dqn59" Sep 17 10:33:07.997633 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:33:07.997580 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h9vhb" podUID="e7c593f1-ac96-40da-94b7-93fcff96f6d4" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.35:8443: connect: connection refused" Sep 17 10:33:07.998141 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:33:07.997746 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h9vhb" Sep 17 10:33:08.326773 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:33:08.326691 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h9vhb" podUID="e7c593f1-ac96-40da-94b7-93fcff96f6d4" containerName="online" probeResult="failure" output="Get \"https://10.134.0.35:6567/health\": dial tcp 10.134.0.35:6567: connect: connection refused" Sep 17 10:33:08.326936 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:33:08.326821 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h9vhb" Sep 17 10:33:10.742126 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:33:10.742093 2581 generic.go:356] "Generic (PLEG): container finished" podID="e7c593f1-ac96-40da-94b7-93fcff96f6d4" containerID="472b77264ca92b7bc17ebd5aacb9fcaec1505fff57a1a0a929800f97ea27a4c3" exitCode=137 Sep 17 10:33:10.742126 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:33:10.742119 2581 generic.go:356] "Generic (PLEG): container finished" podID="e7c593f1-ac96-40da-94b7-93fcff96f6d4" containerID="dfade198c0a694dde4791930e56a8c82372c8f080823ba65d70620a079d10d2e" exitCode=137 Sep 17 10:33:10.742556 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:33:10.742160 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h9vhb" event={"ID":"e7c593f1-ac96-40da-94b7-93fcff96f6d4","Type":"ContainerDied","Data":"472b77264ca92b7bc17ebd5aacb9fcaec1505fff57a1a0a929800f97ea27a4c3"} Sep 17 10:33:10.742556 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:33:10.742201 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h9vhb" event={"ID":"e7c593f1-ac96-40da-94b7-93fcff96f6d4","Type":"ContainerDied","Data":"dfade198c0a694dde4791930e56a8c82372c8f080823ba65d70620a079d10d2e"} Sep 17 10:33:10.996692 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:33:10.996637 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h9vhb" Sep 17 10:33:11.144848 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:33:11.144806 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/e7c593f1-ac96-40da-94b7-93fcff96f6d4-feast-data\") pod \"e7c593f1-ac96-40da-94b7-93fcff96f6d4\" (UID: \"e7c593f1-ac96-40da-94b7-93fcff96f6d4\") " Sep 17 10:33:11.145044 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:33:11.144864 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/e7c593f1-ac96-40da-94b7-93fcff96f6d4-registry-tls\") pod \"e7c593f1-ac96-40da-94b7-93fcff96f6d4\" (UID: \"e7c593f1-ac96-40da-94b7-93fcff96f6d4\") " Sep 17 10:33:11.145044 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:33:11.144905 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/e7c593f1-ac96-40da-94b7-93fcff96f6d4-ui-tls\") pod \"e7c593f1-ac96-40da-94b7-93fcff96f6d4\" (UID: \"e7c593f1-ac96-40da-94b7-93fcff96f6d4\") " Sep 17 10:33:11.145044 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:33:11.144924 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/e7c593f1-ac96-40da-94b7-93fcff96f6d4-online-tls\") pod \"e7c593f1-ac96-40da-94b7-93fcff96f6d4\" (UID: \"e7c593f1-ac96-40da-94b7-93fcff96f6d4\") " Sep 17 10:33:11.145044 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:33:11.144942 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8g6p8\" (UniqueName: \"kubernetes.io/projected/e7c593f1-ac96-40da-94b7-93fcff96f6d4-kube-api-access-8g6p8\") pod \"e7c593f1-ac96-40da-94b7-93fcff96f6d4\" (UID: \"e7c593f1-ac96-40da-94b7-93fcff96f6d4\") " Sep 17 10:33:11.145044 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:33:11.145007 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/e7c593f1-ac96-40da-94b7-93fcff96f6d4-offline-tls\") pod \"e7c593f1-ac96-40da-94b7-93fcff96f6d4\" (UID: \"e7c593f1-ac96-40da-94b7-93fcff96f6d4\") " Sep 17 10:33:11.145459 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:33:11.145426 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7c593f1-ac96-40da-94b7-93fcff96f6d4-feast-data" (OuterVolumeSpecName: "feast-data") pod "e7c593f1-ac96-40da-94b7-93fcff96f6d4" (UID: "e7c593f1-ac96-40da-94b7-93fcff96f6d4"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 10:33:11.147365 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:33:11.147335 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7c593f1-ac96-40da-94b7-93fcff96f6d4-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "e7c593f1-ac96-40da-94b7-93fcff96f6d4" (UID: "e7c593f1-ac96-40da-94b7-93fcff96f6d4"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 10:33:11.147365 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:33:11.147348 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7c593f1-ac96-40da-94b7-93fcff96f6d4-kube-api-access-8g6p8" (OuterVolumeSpecName: "kube-api-access-8g6p8") pod "e7c593f1-ac96-40da-94b7-93fcff96f6d4" (UID: "e7c593f1-ac96-40da-94b7-93fcff96f6d4"). InnerVolumeSpecName "kube-api-access-8g6p8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 10:33:11.147505 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:33:11.147428 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7c593f1-ac96-40da-94b7-93fcff96f6d4-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "e7c593f1-ac96-40da-94b7-93fcff96f6d4" (UID: "e7c593f1-ac96-40da-94b7-93fcff96f6d4"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 10:33:11.147545 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:33:11.147531 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7c593f1-ac96-40da-94b7-93fcff96f6d4-online-tls" (OuterVolumeSpecName: "online-tls") pod "e7c593f1-ac96-40da-94b7-93fcff96f6d4" (UID: "e7c593f1-ac96-40da-94b7-93fcff96f6d4"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 10:33:11.147667 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:33:11.147641 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7c593f1-ac96-40da-94b7-93fcff96f6d4-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "e7c593f1-ac96-40da-94b7-93fcff96f6d4" (UID: "e7c593f1-ac96-40da-94b7-93fcff96f6d4"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 10:33:11.245532 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:33:11.245498 2581 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/e7c593f1-ac96-40da-94b7-93fcff96f6d4-offline-tls\") on node \"ip-10-0-137-203.ec2.internal\" DevicePath \"\"" Sep 17 10:33:11.245532 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:33:11.245525 2581 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/e7c593f1-ac96-40da-94b7-93fcff96f6d4-feast-data\") on node \"ip-10-0-137-203.ec2.internal\" DevicePath \"\"" Sep 17 10:33:11.245532 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:33:11.245533 2581 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/e7c593f1-ac96-40da-94b7-93fcff96f6d4-registry-tls\") on node \"ip-10-0-137-203.ec2.internal\" DevicePath \"\"" Sep 17 10:33:11.245532 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:33:11.245542 2581 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/e7c593f1-ac96-40da-94b7-93fcff96f6d4-ui-tls\") on node \"ip-10-0-137-203.ec2.internal\" DevicePath \"\"" Sep 17 10:33:11.245801 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:33:11.245569 2581 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/e7c593f1-ac96-40da-94b7-93fcff96f6d4-online-tls\") on node \"ip-10-0-137-203.ec2.internal\" DevicePath \"\"" Sep 17 10:33:11.245801 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:33:11.245578 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8g6p8\" (UniqueName: \"kubernetes.io/projected/e7c593f1-ac96-40da-94b7-93fcff96f6d4-kube-api-access-8g6p8\") on node \"ip-10-0-137-203.ec2.internal\" DevicePath \"\"" Sep 17 10:33:11.749609 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:33:11.749575 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h9vhb" event={"ID":"e7c593f1-ac96-40da-94b7-93fcff96f6d4","Type":"ContainerDied","Data":"a486b551a0778d334735d2a5920752b2f68fcc2f08a473f31fdc7fb083c9001f"} Sep 17 10:33:11.750039 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:33:11.749630 2581 scope.go:117] "RemoveContainer" containerID="696471500c405bdb0ed6e30f020f16a7c0776fc613fdb212a1331a0425888749" Sep 17 10:33:11.750039 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:33:11.749633 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h9vhb" Sep 17 10:33:11.758400 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:33:11.758343 2581 scope.go:117] "RemoveContainer" containerID="472b77264ca92b7bc17ebd5aacb9fcaec1505fff57a1a0a929800f97ea27a4c3" Sep 17 10:33:11.766845 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:33:11.766824 2581 scope.go:117] "RemoveContainer" containerID="b132650055509916c24699e9f159c2343b6a27c6f43785e6a9862029f100f0ce" Sep 17 10:33:11.774458 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:33:11.774434 2581 scope.go:117] "RemoveContainer" containerID="dfade198c0a694dde4791930e56a8c82372c8f080823ba65d70620a079d10d2e" Sep 17 10:33:11.779463 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:33:11.779437 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-h9vhb"] Sep 17 10:33:11.782389 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:33:11.782371 2581 scope.go:117] "RemoveContainer" containerID="1f10f33e468abf1a4269fa0e971962b7465c3ea332b02ce954cf0c5ea9b36b14" Sep 17 10:33:11.785106 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:33:11.785085 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-h9vhb"] Sep 17 10:33:11.790286 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:33:11.790269 2581 scope.go:117] "RemoveContainer" containerID="bf38c33bfa1b7c869de457f0b1db490207c7442fa661826310937c63c04edbfb" Sep 17 10:33:11.944737 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:33:11.944701 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e7c593f1-ac96-40da-94b7-93fcff96f6d4" path="/var/lib/kubelet/pods/e7c593f1-ac96-40da-94b7-93fcff96f6d4/volumes" Sep 17 10:34:06.021629 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:34:06.021589 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7zc5b"] Sep 17 10:34:06.022204 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:34:06.022048 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7c593f1-ac96-40da-94b7-93fcff96f6d4" containerName="feast-apply" Sep 17 10:34:06.022204 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:34:06.022067 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7c593f1-ac96-40da-94b7-93fcff96f6d4" containerName="feast-apply" Sep 17 10:34:06.022204 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:34:06.022086 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7c593f1-ac96-40da-94b7-93fcff96f6d4" containerName="offline" Sep 17 10:34:06.022204 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:34:06.022094 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7c593f1-ac96-40da-94b7-93fcff96f6d4" containerName="offline" Sep 17 10:34:06.022204 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:34:06.022108 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7c593f1-ac96-40da-94b7-93fcff96f6d4" containerName="ui" Sep 17 10:34:06.022204 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:34:06.022116 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7c593f1-ac96-40da-94b7-93fcff96f6d4" containerName="ui" Sep 17 10:34:06.022204 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:34:06.022133 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7c593f1-ac96-40da-94b7-93fcff96f6d4" containerName="online" Sep 17 10:34:06.022204 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:34:06.022142 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7c593f1-ac96-40da-94b7-93fcff96f6d4" containerName="online" Sep 17 10:34:06.022204 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:34:06.022159 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7c593f1-ac96-40da-94b7-93fcff96f6d4" containerName="registry" Sep 17 10:34:06.022204 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:34:06.022168 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7c593f1-ac96-40da-94b7-93fcff96f6d4" containerName="registry" Sep 17 10:34:06.022204 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:34:06.022188 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7c593f1-ac96-40da-94b7-93fcff96f6d4" containerName="feast-init" Sep 17 10:34:06.022204 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:34:06.022195 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7c593f1-ac96-40da-94b7-93fcff96f6d4" containerName="feast-init" Sep 17 10:34:06.022866 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:34:06.022273 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7c593f1-ac96-40da-94b7-93fcff96f6d4" containerName="online" Sep 17 10:34:06.022866 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:34:06.022284 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7c593f1-ac96-40da-94b7-93fcff96f6d4" containerName="offline" Sep 17 10:34:06.022866 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:34:06.022295 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7c593f1-ac96-40da-94b7-93fcff96f6d4" containerName="ui" Sep 17 10:34:06.022866 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:34:06.022309 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7c593f1-ac96-40da-94b7-93fcff96f6d4" containerName="registry" Sep 17 10:34:06.027139 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:34:06.027118 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7zc5b" Sep 17 10:34:06.031626 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:34:06.031605 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-ui-tls\"" Sep 17 10:34:06.031763 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:34:06.031624 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-offline-tls\"" Sep 17 10:34:06.031763 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:34:06.031607 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-dockercfg-46m42\"" Sep 17 10:34:06.031763 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:34:06.031607 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"openshift-service-ca.crt\"" Sep 17 10:34:06.031977 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:34:06.031916 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-online-tls\"" Sep 17 10:34:06.031977 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:34:06.031973 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-client-ca\"" Sep 17 10:34:06.032065 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:34:06.031975 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"kube-root-ca.crt\"" Sep 17 10:34:06.039517 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:34:06.039493 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7zc5b"] Sep 17 10:34:06.064664 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:34:06.064635 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/4ab70dd1-fa52-4bd5-bff1-73591883601d-feast-data\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-7zc5b\" (UID: \"4ab70dd1-fa52-4bd5-bff1-73591883601d\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7zc5b" Sep 17 10:34:06.064805 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:34:06.064685 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/4ab70dd1-fa52-4bd5-bff1-73591883601d-offline-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-7zc5b\" (UID: \"4ab70dd1-fa52-4bd5-bff1-73591883601d\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7zc5b" Sep 17 10:34:06.064805 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:34:06.064716 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/4ab70dd1-fa52-4bd5-bff1-73591883601d-ui-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-7zc5b\" (UID: \"4ab70dd1-fa52-4bd5-bff1-73591883601d\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7zc5b" Sep 17 10:34:06.064805 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:34:06.064753 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/4ab70dd1-fa52-4bd5-bff1-73591883601d-online-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-7zc5b\" (UID: \"4ab70dd1-fa52-4bd5-bff1-73591883601d\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7zc5b" Sep 17 10:34:06.064805 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:34:06.064798 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsxz9\" (UniqueName: \"kubernetes.io/projected/4ab70dd1-fa52-4bd5-bff1-73591883601d-kube-api-access-lsxz9\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-7zc5b\" (UID: \"4ab70dd1-fa52-4bd5-bff1-73591883601d\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7zc5b" Sep 17 10:34:06.064997 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:34:06.064836 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/4ab70dd1-fa52-4bd5-bff1-73591883601d-registry-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-7zc5b\" (UID: \"4ab70dd1-fa52-4bd5-bff1-73591883601d\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7zc5b" Sep 17 10:34:06.166129 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:34:06.166102 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/4ab70dd1-fa52-4bd5-bff1-73591883601d-offline-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-7zc5b\" (UID: \"4ab70dd1-fa52-4bd5-bff1-73591883601d\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7zc5b" Sep 17 10:34:06.166262 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:34:06.166138 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/4ab70dd1-fa52-4bd5-bff1-73591883601d-ui-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-7zc5b\" (UID: \"4ab70dd1-fa52-4bd5-bff1-73591883601d\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7zc5b" Sep 17 10:34:06.166262 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:34:06.166167 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/4ab70dd1-fa52-4bd5-bff1-73591883601d-online-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-7zc5b\" (UID: \"4ab70dd1-fa52-4bd5-bff1-73591883601d\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7zc5b" Sep 17 10:34:06.166262 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:34:06.166187 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsxz9\" (UniqueName: \"kubernetes.io/projected/4ab70dd1-fa52-4bd5-bff1-73591883601d-kube-api-access-lsxz9\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-7zc5b\" (UID: \"4ab70dd1-fa52-4bd5-bff1-73591883601d\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7zc5b" Sep 17 10:34:06.166262 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:34:06.166208 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/4ab70dd1-fa52-4bd5-bff1-73591883601d-registry-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-7zc5b\" (UID: \"4ab70dd1-fa52-4bd5-bff1-73591883601d\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7zc5b" Sep 17 10:34:06.166262 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:34:06.166247 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/4ab70dd1-fa52-4bd5-bff1-73591883601d-feast-data\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-7zc5b\" (UID: \"4ab70dd1-fa52-4bd5-bff1-73591883601d\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7zc5b" Sep 17 10:34:06.166666 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:34:06.166639 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/4ab70dd1-fa52-4bd5-bff1-73591883601d-feast-data\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-7zc5b\" (UID: \"4ab70dd1-fa52-4bd5-bff1-73591883601d\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7zc5b" Sep 17 10:34:06.167016 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:34:06.166996 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/4ab70dd1-fa52-4bd5-bff1-73591883601d-registry-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-7zc5b\" (UID: \"4ab70dd1-fa52-4bd5-bff1-73591883601d\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7zc5b" Sep 17 10:34:06.168884 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:34:06.168861 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/4ab70dd1-fa52-4bd5-bff1-73591883601d-online-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-7zc5b\" (UID: \"4ab70dd1-fa52-4bd5-bff1-73591883601d\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7zc5b" Sep 17 10:34:06.169158 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:34:06.169139 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/4ab70dd1-fa52-4bd5-bff1-73591883601d-ui-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-7zc5b\" (UID: \"4ab70dd1-fa52-4bd5-bff1-73591883601d\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7zc5b" Sep 17 10:34:06.169158 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:34:06.169153 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/4ab70dd1-fa52-4bd5-bff1-73591883601d-offline-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-7zc5b\" (UID: \"4ab70dd1-fa52-4bd5-bff1-73591883601d\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7zc5b" Sep 17 10:34:06.175379 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:34:06.175359 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsxz9\" (UniqueName: \"kubernetes.io/projected/4ab70dd1-fa52-4bd5-bff1-73591883601d-kube-api-access-lsxz9\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-7zc5b\" (UID: \"4ab70dd1-fa52-4bd5-bff1-73591883601d\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7zc5b" Sep 17 10:34:06.336344 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:34:06.336281 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7zc5b" Sep 17 10:34:06.615154 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:34:06.615125 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7zc5b"] Sep 17 10:34:06.616120 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:34:06.616097 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4ab70dd1_fa52_4bd5_bff1_73591883601d.slice/crio-061339f2d9f6d0498eb69bbcb4f2a9ee22a774366d7991d7dbcd57b1c984cc6f WatchSource:0}: Error finding container 061339f2d9f6d0498eb69bbcb4f2a9ee22a774366d7991d7dbcd57b1c984cc6f: Status 404 returned error can't find the container with id 061339f2d9f6d0498eb69bbcb4f2a9ee22a774366d7991d7dbcd57b1c984cc6f Sep 17 10:34:06.972692 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:34:06.972591 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7zc5b" event={"ID":"4ab70dd1-fa52-4bd5-bff1-73591883601d","Type":"ContainerStarted","Data":"c42d0049732991f9274e79f5a037c60a584f25ac548a184fe130cf268369a119"} Sep 17 10:34:06.972692 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:34:06.972636 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7zc5b" event={"ID":"4ab70dd1-fa52-4bd5-bff1-73591883601d","Type":"ContainerStarted","Data":"061339f2d9f6d0498eb69bbcb4f2a9ee22a774366d7991d7dbcd57b1c984cc6f"} Sep 17 10:34:12.000920 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:34:12.000887 2581 generic.go:356] "Generic (PLEG): container finished" podID="4ab70dd1-fa52-4bd5-bff1-73591883601d" containerID="c42d0049732991f9274e79f5a037c60a584f25ac548a184fe130cf268369a119" exitCode=0 Sep 17 10:34:12.001437 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:34:12.000943 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7zc5b" event={"ID":"4ab70dd1-fa52-4bd5-bff1-73591883601d","Type":"ContainerDied","Data":"c42d0049732991f9274e79f5a037c60a584f25ac548a184fe130cf268369a119"} Sep 17 10:34:13.007523 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:34:13.007489 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7zc5b" event={"ID":"4ab70dd1-fa52-4bd5-bff1-73591883601d","Type":"ContainerStarted","Data":"863be859f5848919575953b259d52de863bbe70733bd6e53d748c12ac3c0f67a"} Sep 17 10:34:37.106423 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:34:37.106386 2581 generic.go:356] "Generic (PLEG): container finished" podID="4ab70dd1-fa52-4bd5-bff1-73591883601d" containerID="863be859f5848919575953b259d52de863bbe70733bd6e53d748c12ac3c0f67a" exitCode=0 Sep 17 10:34:37.106825 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:34:37.106459 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7zc5b" event={"ID":"4ab70dd1-fa52-4bd5-bff1-73591883601d","Type":"ContainerDied","Data":"863be859f5848919575953b259d52de863bbe70733bd6e53d748c12ac3c0f67a"} Sep 17 10:34:38.115763 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:34:38.115666 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7zc5b" event={"ID":"4ab70dd1-fa52-4bd5-bff1-73591883601d","Type":"ContainerStarted","Data":"152ddf77d2c02f5c52bfe83b4535eefbc5c595a0ff7418f47cde607f04c9c37a"} Sep 17 10:34:38.115763 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:34:38.115734 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7zc5b" event={"ID":"4ab70dd1-fa52-4bd5-bff1-73591883601d","Type":"ContainerStarted","Data":"5ca8a7d79ed989a7ead148ee8643585047324b4a83a1f3628437c3fbb9d7657c"} Sep 17 10:34:38.115763 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:34:38.115757 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7zc5b" event={"ID":"4ab70dd1-fa52-4bd5-bff1-73591883601d","Type":"ContainerStarted","Data":"0c706ebfb44ddc7a91483d8606e2a019adad18f56031ffde9d0633b2a3947bb9"} Sep 17 10:34:38.145242 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:34:38.145174 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7zc5b" podStartSLOduration=33.145151602 podStartE2EDuration="33.145151602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 10:34:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 10:34:38.142228432 +0000 UTC m=+1432.690785542" watchObservedRunningTime="2026-09-17 10:34:38.145151602 +0000 UTC m=+1432.693708713" Sep 17 10:34:39.336670 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:34:39.336633 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7zc5b" Sep 17 10:34:39.337140 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:34:39.336693 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7zc5b" Sep 17 10:34:39.337140 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:34:39.336714 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7zc5b" Sep 17 10:34:39.338325 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:34:39.338293 2581 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7zc5b" podUID="4ab70dd1-fa52-4bd5-bff1-73591883601d" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.37:8443: connect: connection refused" Sep 17 10:34:39.338441 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:34:39.338316 2581 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7zc5b" podUID="4ab70dd1-fa52-4bd5-bff1-73591883601d" containerName="online" probeResult="failure" output="Get \"https://10.134.0.37:6567/health\": dial tcp 10.134.0.37:6567: connect: connection refused" Sep 17 10:34:39.338441 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:34:39.338294 2581 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7zc5b" podUID="4ab70dd1-fa52-4bd5-bff1-73591883601d" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.37:8816: connect: connection refused" Sep 17 10:34:42.337099 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:34:42.337049 2581 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7zc5b" podUID="4ab70dd1-fa52-4bd5-bff1-73591883601d" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.37:8816: connect: connection refused" Sep 17 10:34:42.337707 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:34:42.337102 2581 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7zc5b" podUID="4ab70dd1-fa52-4bd5-bff1-73591883601d" containerName="online" probeResult="failure" output="Get \"https://10.134.0.37:6567/health\": dial tcp 10.134.0.37:6567: connect: connection refused" Sep 17 10:34:42.337707 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:34:42.337064 2581 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7zc5b" podUID="4ab70dd1-fa52-4bd5-bff1-73591883601d" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.37:8443: connect: connection refused" Sep 17 10:34:45.337192 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:34:45.337140 2581 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7zc5b" podUID="4ab70dd1-fa52-4bd5-bff1-73591883601d" containerName="online" probeResult="failure" output="Get \"https://10.134.0.37:6567/health\": dial tcp 10.134.0.37:6567: connect: connection refused" Sep 17 10:34:45.338032 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:34:45.338010 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7zc5b" Sep 17 10:34:45.338159 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:34:45.338099 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7zc5b" Sep 17 10:34:45.338390 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:34:45.338370 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7zc5b" Sep 17 10:34:45.338467 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:34:45.338407 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7zc5b" Sep 17 10:34:45.338735 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:34:45.338713 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7zc5b" Sep 17 10:34:45.338866 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:34:45.338851 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7zc5b" Sep 17 10:34:46.336817 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:34:46.336767 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7zc5b" Sep 17 10:34:49.247627 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:34:49.247594 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7zc5b" Sep 17 10:34:50.053840 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:34:50.053804 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7zc5b" Sep 17 10:35:45.942618 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:35:45.942590 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-b9wnf_762d3e10-7b7d-4ce9-9773-90a4fe41b474/console-operator/2.log" Sep 17 10:35:45.943110 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:35:45.942799 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-b9wnf_762d3e10-7b7d-4ce9-9773-90a4fe41b474/console-operator/2.log" Sep 17 10:35:52.474988 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:35:52.474954 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7zc5b"] Sep 17 10:35:52.475535 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:35:52.475183 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7zc5b" podUID="4ab70dd1-fa52-4bd5-bff1-73591883601d" containerName="online" containerID="cri-o://0c706ebfb44ddc7a91483d8606e2a019adad18f56031ffde9d0633b2a3947bb9" gracePeriod=30 Sep 17 10:35:52.475535 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:35:52.475224 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7zc5b" podUID="4ab70dd1-fa52-4bd5-bff1-73591883601d" containerName="ui" containerID="cri-o://152ddf77d2c02f5c52bfe83b4535eefbc5c595a0ff7418f47cde607f04c9c37a" gracePeriod=30 Sep 17 10:35:52.475535 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:35:52.475228 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7zc5b" podUID="4ab70dd1-fa52-4bd5-bff1-73591883601d" containerName="offline" containerID="cri-o://5ca8a7d79ed989a7ead148ee8643585047324b4a83a1f3628437c3fbb9d7657c" gracePeriod=30 Sep 17 10:35:52.691133 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:35:52.691096 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-dqn59"] Sep 17 10:35:52.691368 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:35:52.691338 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-dqn59" podUID="e7c9bdd3-0224-4a08-98c1-d0a648f8f70a" containerName="registry" containerID="cri-o://3a531941343edd3826bd3f9b22a39350d9e7ee225706dfaa8768cfa28f7b713d" gracePeriod=30 Sep 17 10:35:52.691452 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:35:52.691392 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-dqn59" podUID="e7c9bdd3-0224-4a08-98c1-d0a648f8f70a" containerName="offline" containerID="cri-o://119f23e8169e43350773f0c6b02aa98e75a8cab5db2b96e7480de039485467b0" gracePeriod=30 Sep 17 10:35:52.691513 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:35:52.691417 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-dqn59" podUID="e7c9bdd3-0224-4a08-98c1-d0a648f8f70a" containerName="online" containerID="cri-o://dfe8cab6ef9454ae74f7fc17ea20077ea9671f0ca33a148a4c08def24dbeef3a" gracePeriod=30 Sep 17 10:35:52.691513 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:35:52.691458 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-dqn59" podUID="e7c9bdd3-0224-4a08-98c1-d0a648f8f70a" containerName="ui" containerID="cri-o://da993cc3c9ece08c90512473d53143cb9a49de4fcbd84230bc2cd2fcf9b28988" gracePeriod=30 Sep 17 10:35:52.782378 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:35:52.782283 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-dqn59" podUID="e7c9bdd3-0224-4a08-98c1-d0a648f8f70a" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.36:8443: connect: connection refused" Sep 17 10:35:53.429037 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:35:53.429000 2581 generic.go:356] "Generic (PLEG): container finished" podID="4ab70dd1-fa52-4bd5-bff1-73591883601d" containerID="152ddf77d2c02f5c52bfe83b4535eefbc5c595a0ff7418f47cde607f04c9c37a" exitCode=0 Sep 17 10:35:53.429037 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:35:53.429026 2581 generic.go:356] "Generic (PLEG): container finished" podID="4ab70dd1-fa52-4bd5-bff1-73591883601d" containerID="0c706ebfb44ddc7a91483d8606e2a019adad18f56031ffde9d0633b2a3947bb9" exitCode=0 Sep 17 10:35:53.429259 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:35:53.429072 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7zc5b" event={"ID":"4ab70dd1-fa52-4bd5-bff1-73591883601d","Type":"ContainerDied","Data":"152ddf77d2c02f5c52bfe83b4535eefbc5c595a0ff7418f47cde607f04c9c37a"} Sep 17 10:35:53.429259 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:35:53.429113 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7zc5b" event={"ID":"4ab70dd1-fa52-4bd5-bff1-73591883601d","Type":"ContainerDied","Data":"0c706ebfb44ddc7a91483d8606e2a019adad18f56031ffde9d0633b2a3947bb9"} Sep 17 10:35:53.432206 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:35:53.432176 2581 generic.go:356] "Generic (PLEG): container finished" podID="e7c9bdd3-0224-4a08-98c1-d0a648f8f70a" containerID="da993cc3c9ece08c90512473d53143cb9a49de4fcbd84230bc2cd2fcf9b28988" exitCode=0 Sep 17 10:35:53.432206 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:35:53.432206 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-dqn59" event={"ID":"e7c9bdd3-0224-4a08-98c1-d0a648f8f70a","Type":"ContainerDied","Data":"da993cc3c9ece08c90512473d53143cb9a49de4fcbd84230bc2cd2fcf9b28988"} Sep 17 10:35:53.703917 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:35:53.703828 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-dqn59" podUID="e7c9bdd3-0224-4a08-98c1-d0a648f8f70a" containerName="online" probeResult="failure" output="Get \"https://10.134.0.36:6567/health\": dial tcp 10.134.0.36:6567: connect: connection refused" Sep 17 10:35:54.439871 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:35:54.439840 2581 generic.go:356] "Generic (PLEG): container finished" podID="e7c9bdd3-0224-4a08-98c1-d0a648f8f70a" containerID="dfe8cab6ef9454ae74f7fc17ea20077ea9671f0ca33a148a4c08def24dbeef3a" exitCode=0 Sep 17 10:35:54.440041 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:35:54.439878 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-dqn59" event={"ID":"e7c9bdd3-0224-4a08-98c1-d0a648f8f70a","Type":"ContainerDied","Data":"dfe8cab6ef9454ae74f7fc17ea20077ea9671f0ca33a148a4c08def24dbeef3a"} Sep 17 10:35:55.338720 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:35:55.338675 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7zc5b" podUID="4ab70dd1-fa52-4bd5-bff1-73591883601d" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.37:8443: connect: connection refused" Sep 17 10:35:59.249039 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:35:59.248991 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7zc5b" podUID="4ab70dd1-fa52-4bd5-bff1-73591883601d" containerName="online" probeResult="failure" output="Get \"https://10.134.0.37:6567/health\": dial tcp 10.134.0.37:6567: connect: connection refused" Sep 17 10:36:02.782028 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:36:02.781977 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-dqn59" podUID="e7c9bdd3-0224-4a08-98c1-d0a648f8f70a" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.36:8443: connect: connection refused" Sep 17 10:36:03.703416 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:36:03.703369 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-dqn59" podUID="e7c9bdd3-0224-4a08-98c1-d0a648f8f70a" containerName="online" probeResult="failure" output="Get \"https://10.134.0.36:6567/health\": dial tcp 10.134.0.36:6567: connect: connection refused" Sep 17 10:36:05.339232 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:36:05.339190 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7zc5b" podUID="4ab70dd1-fa52-4bd5-bff1-73591883601d" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.37:8443: connect: connection refused" Sep 17 10:36:09.249285 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:36:09.249199 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7zc5b" podUID="4ab70dd1-fa52-4bd5-bff1-73591883601d" containerName="online" probeResult="failure" output="Get \"https://10.134.0.37:6567/health\": dial tcp 10.134.0.37:6567: connect: connection refused" Sep 17 10:36:12.782450 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:36:12.782404 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-dqn59" podUID="e7c9bdd3-0224-4a08-98c1-d0a648f8f70a" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.36:8443: connect: connection refused" Sep 17 10:36:12.783025 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:36:12.782548 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-dqn59" Sep 17 10:36:13.703498 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:36:13.703457 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-dqn59" podUID="e7c9bdd3-0224-4a08-98c1-d0a648f8f70a" containerName="online" probeResult="failure" output="Get \"https://10.134.0.36:6567/health\": dial tcp 10.134.0.36:6567: connect: connection refused" Sep 17 10:36:13.703714 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:36:13.703593 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-dqn59" Sep 17 10:36:15.338908 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:36:15.338868 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7zc5b" podUID="4ab70dd1-fa52-4bd5-bff1-73591883601d" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.37:8443: connect: connection refused" Sep 17 10:36:15.339313 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:36:15.338979 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7zc5b" Sep 17 10:36:19.248773 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:36:19.248724 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7zc5b" podUID="4ab70dd1-fa52-4bd5-bff1-73591883601d" containerName="online" probeResult="failure" output="Get \"https://10.134.0.37:6567/health\": dial tcp 10.134.0.37:6567: connect: connection refused" Sep 17 10:36:19.249217 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:36:19.248899 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7zc5b" Sep 17 10:36:22.556068 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:36:22.556034 2581 generic.go:356] "Generic (PLEG): container finished" podID="4ab70dd1-fa52-4bd5-bff1-73591883601d" containerID="5ca8a7d79ed989a7ead148ee8643585047324b4a83a1f3628437c3fbb9d7657c" exitCode=137 Sep 17 10:36:22.556068 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:36:22.556072 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7zc5b" event={"ID":"4ab70dd1-fa52-4bd5-bff1-73591883601d","Type":"ContainerDied","Data":"5ca8a7d79ed989a7ead148ee8643585047324b4a83a1f3628437c3fbb9d7657c"} Sep 17 10:36:22.782637 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:36:22.782551 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-dqn59" podUID="e7c9bdd3-0224-4a08-98c1-d0a648f8f70a" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.36:8443: connect: connection refused" Sep 17 10:36:22.782813 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:36:22.782551 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-dqn59" podUID="e7c9bdd3-0224-4a08-98c1-d0a648f8f70a" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.36:6571: connect: connection refused" Sep 17 10:36:23.115260 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:36:23.115238 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7zc5b" Sep 17 10:36:23.175856 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:36:23.175820 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/4ab70dd1-fa52-4bd5-bff1-73591883601d-feast-data\") pod \"4ab70dd1-fa52-4bd5-bff1-73591883601d\" (UID: \"4ab70dd1-fa52-4bd5-bff1-73591883601d\") " Sep 17 10:36:23.175856 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:36:23.175860 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/4ab70dd1-fa52-4bd5-bff1-73591883601d-online-tls\") pod \"4ab70dd1-fa52-4bd5-bff1-73591883601d\" (UID: \"4ab70dd1-fa52-4bd5-bff1-73591883601d\") " Sep 17 10:36:23.176079 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:36:23.175898 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lsxz9\" (UniqueName: \"kubernetes.io/projected/4ab70dd1-fa52-4bd5-bff1-73591883601d-kube-api-access-lsxz9\") pod \"4ab70dd1-fa52-4bd5-bff1-73591883601d\" (UID: \"4ab70dd1-fa52-4bd5-bff1-73591883601d\") " Sep 17 10:36:23.176079 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:36:23.175918 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/4ab70dd1-fa52-4bd5-bff1-73591883601d-offline-tls\") pod \"4ab70dd1-fa52-4bd5-bff1-73591883601d\" (UID: \"4ab70dd1-fa52-4bd5-bff1-73591883601d\") " Sep 17 10:36:23.176079 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:36:23.175936 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/4ab70dd1-fa52-4bd5-bff1-73591883601d-registry-tls\") pod \"4ab70dd1-fa52-4bd5-bff1-73591883601d\" (UID: \"4ab70dd1-fa52-4bd5-bff1-73591883601d\") " Sep 17 10:36:23.176079 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:36:23.175971 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/4ab70dd1-fa52-4bd5-bff1-73591883601d-ui-tls\") pod \"4ab70dd1-fa52-4bd5-bff1-73591883601d\" (UID: \"4ab70dd1-fa52-4bd5-bff1-73591883601d\") " Sep 17 10:36:23.176453 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:36:23.176410 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ab70dd1-fa52-4bd5-bff1-73591883601d-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "4ab70dd1-fa52-4bd5-bff1-73591883601d" (UID: "4ab70dd1-fa52-4bd5-bff1-73591883601d"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 10:36:23.176589 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:36:23.176475 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ab70dd1-fa52-4bd5-bff1-73591883601d-feast-data" (OuterVolumeSpecName: "feast-data") pod "4ab70dd1-fa52-4bd5-bff1-73591883601d" (UID: "4ab70dd1-fa52-4bd5-bff1-73591883601d"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 10:36:23.178216 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:36:23.178171 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ab70dd1-fa52-4bd5-bff1-73591883601d-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "4ab70dd1-fa52-4bd5-bff1-73591883601d" (UID: "4ab70dd1-fa52-4bd5-bff1-73591883601d"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 10:36:23.178315 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:36:23.178244 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ab70dd1-fa52-4bd5-bff1-73591883601d-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "4ab70dd1-fa52-4bd5-bff1-73591883601d" (UID: "4ab70dd1-fa52-4bd5-bff1-73591883601d"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 10:36:23.178315 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:36:23.178259 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ab70dd1-fa52-4bd5-bff1-73591883601d-kube-api-access-lsxz9" (OuterVolumeSpecName: "kube-api-access-lsxz9") pod "4ab70dd1-fa52-4bd5-bff1-73591883601d" (UID: "4ab70dd1-fa52-4bd5-bff1-73591883601d"). InnerVolumeSpecName "kube-api-access-lsxz9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 10:36:23.178315 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:36:23.178260 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ab70dd1-fa52-4bd5-bff1-73591883601d-online-tls" (OuterVolumeSpecName: "online-tls") pod "4ab70dd1-fa52-4bd5-bff1-73591883601d" (UID: "4ab70dd1-fa52-4bd5-bff1-73591883601d"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 10:36:23.277474 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:36:23.277439 2581 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/4ab70dd1-fa52-4bd5-bff1-73591883601d-feast-data\") on node \"ip-10-0-137-203.ec2.internal\" DevicePath \"\"" Sep 17 10:36:23.277603 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:36:23.277477 2581 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/4ab70dd1-fa52-4bd5-bff1-73591883601d-online-tls\") on node \"ip-10-0-137-203.ec2.internal\" DevicePath \"\"" Sep 17 10:36:23.277603 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:36:23.277494 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lsxz9\" (UniqueName: \"kubernetes.io/projected/4ab70dd1-fa52-4bd5-bff1-73591883601d-kube-api-access-lsxz9\") on node \"ip-10-0-137-203.ec2.internal\" DevicePath \"\"" Sep 17 10:36:23.277603 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:36:23.277510 2581 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/4ab70dd1-fa52-4bd5-bff1-73591883601d-offline-tls\") on node \"ip-10-0-137-203.ec2.internal\" DevicePath \"\"" Sep 17 10:36:23.277603 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:36:23.277524 2581 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/4ab70dd1-fa52-4bd5-bff1-73591883601d-registry-tls\") on node \"ip-10-0-137-203.ec2.internal\" DevicePath \"\"" Sep 17 10:36:23.277603 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:36:23.277537 2581 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/4ab70dd1-fa52-4bd5-bff1-73591883601d-ui-tls\") on node \"ip-10-0-137-203.ec2.internal\" DevicePath \"\"" Sep 17 10:36:23.323036 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:36:23.323013 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-dqn59" Sep 17 10:36:23.378693 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:36:23.378662 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/e7c9bdd3-0224-4a08-98c1-d0a648f8f70a-registry-tls\") pod \"e7c9bdd3-0224-4a08-98c1-d0a648f8f70a\" (UID: \"e7c9bdd3-0224-4a08-98c1-d0a648f8f70a\") " Sep 17 10:36:23.378693 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:36:23.378696 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/e7c9bdd3-0224-4a08-98c1-d0a648f8f70a-offline-tls\") pod \"e7c9bdd3-0224-4a08-98c1-d0a648f8f70a\" (UID: \"e7c9bdd3-0224-4a08-98c1-d0a648f8f70a\") " Sep 17 10:36:23.378978 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:36:23.378719 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/e7c9bdd3-0224-4a08-98c1-d0a648f8f70a-feast-data\") pod \"e7c9bdd3-0224-4a08-98c1-d0a648f8f70a\" (UID: \"e7c9bdd3-0224-4a08-98c1-d0a648f8f70a\") " Sep 17 10:36:23.378978 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:36:23.378737 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-77dln\" (UniqueName: \"kubernetes.io/projected/e7c9bdd3-0224-4a08-98c1-d0a648f8f70a-kube-api-access-77dln\") pod \"e7c9bdd3-0224-4a08-98c1-d0a648f8f70a\" (UID: \"e7c9bdd3-0224-4a08-98c1-d0a648f8f70a\") " Sep 17 10:36:23.378978 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:36:23.378841 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/e7c9bdd3-0224-4a08-98c1-d0a648f8f70a-online-tls\") pod \"e7c9bdd3-0224-4a08-98c1-d0a648f8f70a\" (UID: \"e7c9bdd3-0224-4a08-98c1-d0a648f8f70a\") " Sep 17 10:36:23.378978 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:36:23.378884 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/e7c9bdd3-0224-4a08-98c1-d0a648f8f70a-ui-tls\") pod \"e7c9bdd3-0224-4a08-98c1-d0a648f8f70a\" (UID: \"e7c9bdd3-0224-4a08-98c1-d0a648f8f70a\") " Sep 17 10:36:23.379359 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:36:23.379329 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7c9bdd3-0224-4a08-98c1-d0a648f8f70a-feast-data" (OuterVolumeSpecName: "feast-data") pod "e7c9bdd3-0224-4a08-98c1-d0a648f8f70a" (UID: "e7c9bdd3-0224-4a08-98c1-d0a648f8f70a"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 10:36:23.381033 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:36:23.380999 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7c9bdd3-0224-4a08-98c1-d0a648f8f70a-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "e7c9bdd3-0224-4a08-98c1-d0a648f8f70a" (UID: "e7c9bdd3-0224-4a08-98c1-d0a648f8f70a"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 10:36:23.381151 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:36:23.381056 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7c9bdd3-0224-4a08-98c1-d0a648f8f70a-online-tls" (OuterVolumeSpecName: "online-tls") pod "e7c9bdd3-0224-4a08-98c1-d0a648f8f70a" (UID: "e7c9bdd3-0224-4a08-98c1-d0a648f8f70a"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 10:36:23.381151 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:36:23.381082 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7c9bdd3-0224-4a08-98c1-d0a648f8f70a-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "e7c9bdd3-0224-4a08-98c1-d0a648f8f70a" (UID: "e7c9bdd3-0224-4a08-98c1-d0a648f8f70a"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 10:36:23.381363 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:36:23.381344 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7c9bdd3-0224-4a08-98c1-d0a648f8f70a-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "e7c9bdd3-0224-4a08-98c1-d0a648f8f70a" (UID: "e7c9bdd3-0224-4a08-98c1-d0a648f8f70a"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 10:36:23.381457 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:36:23.381368 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7c9bdd3-0224-4a08-98c1-d0a648f8f70a-kube-api-access-77dln" (OuterVolumeSpecName: "kube-api-access-77dln") pod "e7c9bdd3-0224-4a08-98c1-d0a648f8f70a" (UID: "e7c9bdd3-0224-4a08-98c1-d0a648f8f70a"). InnerVolumeSpecName "kube-api-access-77dln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 10:36:23.479712 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:36:23.479659 2581 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/e7c9bdd3-0224-4a08-98c1-d0a648f8f70a-online-tls\") on node \"ip-10-0-137-203.ec2.internal\" DevicePath \"\"" Sep 17 10:36:23.479712 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:36:23.479706 2581 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/e7c9bdd3-0224-4a08-98c1-d0a648f8f70a-ui-tls\") on node \"ip-10-0-137-203.ec2.internal\" DevicePath \"\"" Sep 17 10:36:23.479712 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:36:23.479715 2581 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/e7c9bdd3-0224-4a08-98c1-d0a648f8f70a-registry-tls\") on node \"ip-10-0-137-203.ec2.internal\" DevicePath \"\"" Sep 17 10:36:23.479712 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:36:23.479722 2581 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/e7c9bdd3-0224-4a08-98c1-d0a648f8f70a-offline-tls\") on node \"ip-10-0-137-203.ec2.internal\" DevicePath \"\"" Sep 17 10:36:23.479712 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:36:23.479730 2581 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/e7c9bdd3-0224-4a08-98c1-d0a648f8f70a-feast-data\") on node \"ip-10-0-137-203.ec2.internal\" DevicePath \"\"" Sep 17 10:36:23.480057 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:36:23.479739 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77dln\" (UniqueName: \"kubernetes.io/projected/e7c9bdd3-0224-4a08-98c1-d0a648f8f70a-kube-api-access-77dln\") on node \"ip-10-0-137-203.ec2.internal\" DevicePath \"\"" Sep 17 10:36:23.563018 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:36:23.562981 2581 generic.go:356] "Generic (PLEG): container finished" podID="e7c9bdd3-0224-4a08-98c1-d0a648f8f70a" containerID="119f23e8169e43350773f0c6b02aa98e75a8cab5db2b96e7480de039485467b0" exitCode=137 Sep 17 10:36:23.563018 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:36:23.563008 2581 generic.go:356] "Generic (PLEG): container finished" podID="e7c9bdd3-0224-4a08-98c1-d0a648f8f70a" containerID="3a531941343edd3826bd3f9b22a39350d9e7ee225706dfaa8768cfa28f7b713d" exitCode=137 Sep 17 10:36:23.563566 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:36:23.563044 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-dqn59" Sep 17 10:36:23.563566 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:36:23.563070 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-dqn59" event={"ID":"e7c9bdd3-0224-4a08-98c1-d0a648f8f70a","Type":"ContainerDied","Data":"119f23e8169e43350773f0c6b02aa98e75a8cab5db2b96e7480de039485467b0"} Sep 17 10:36:23.563566 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:36:23.563138 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-dqn59" event={"ID":"e7c9bdd3-0224-4a08-98c1-d0a648f8f70a","Type":"ContainerDied","Data":"3a531941343edd3826bd3f9b22a39350d9e7ee225706dfaa8768cfa28f7b713d"} Sep 17 10:36:23.563566 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:36:23.563164 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-dqn59" event={"ID":"e7c9bdd3-0224-4a08-98c1-d0a648f8f70a","Type":"ContainerDied","Data":"e9facafb39090e2927fc623d2939a070b9f814fbfffbf7cb645ac9f1f0b56b1f"} Sep 17 10:36:23.563566 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:36:23.563192 2581 scope.go:117] "RemoveContainer" containerID="da993cc3c9ece08c90512473d53143cb9a49de4fcbd84230bc2cd2fcf9b28988" Sep 17 10:36:23.566202 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:36:23.566178 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7zc5b" event={"ID":"4ab70dd1-fa52-4bd5-bff1-73591883601d","Type":"ContainerDied","Data":"061339f2d9f6d0498eb69bbcb4f2a9ee22a774366d7991d7dbcd57b1c984cc6f"} Sep 17 10:36:23.566288 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:36:23.566202 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7zc5b" Sep 17 10:36:23.573221 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:36:23.573201 2581 scope.go:117] "RemoveContainer" containerID="119f23e8169e43350773f0c6b02aa98e75a8cab5db2b96e7480de039485467b0" Sep 17 10:36:23.581968 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:36:23.581948 2581 scope.go:117] "RemoveContainer" containerID="dfe8cab6ef9454ae74f7fc17ea20077ea9671f0ca33a148a4c08def24dbeef3a" Sep 17 10:36:23.589771 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:36:23.589745 2581 scope.go:117] "RemoveContainer" containerID="3a531941343edd3826bd3f9b22a39350d9e7ee225706dfaa8768cfa28f7b713d" Sep 17 10:36:23.591615 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:36:23.591592 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-dqn59"] Sep 17 10:36:23.595943 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:36:23.595921 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-dqn59"] Sep 17 10:36:23.598322 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:36:23.598306 2581 scope.go:117] "RemoveContainer" containerID="f65f00db289356bacf2cd9adbdf973a0a5806b4274f03d2f60329d055c954073" Sep 17 10:36:23.605360 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:36:23.605344 2581 scope.go:117] "RemoveContainer" containerID="7d4f33d36d615f0206a33b05be2a8ba8523d842cfc9db802c6099ef0baba91c4" Sep 17 10:36:23.609646 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:36:23.609618 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7zc5b"] Sep 17 10:36:23.613459 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:36:23.613336 2581 scope.go:117] "RemoveContainer" containerID="da993cc3c9ece08c90512473d53143cb9a49de4fcbd84230bc2cd2fcf9b28988" Sep 17 10:36:23.613721 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:36:23.613691 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da993cc3c9ece08c90512473d53143cb9a49de4fcbd84230bc2cd2fcf9b28988\": container with ID starting with da993cc3c9ece08c90512473d53143cb9a49de4fcbd84230bc2cd2fcf9b28988 not found: ID does not exist" containerID="da993cc3c9ece08c90512473d53143cb9a49de4fcbd84230bc2cd2fcf9b28988" Sep 17 10:36:23.613830 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:36:23.613731 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da993cc3c9ece08c90512473d53143cb9a49de4fcbd84230bc2cd2fcf9b28988"} err="failed to get container status \"da993cc3c9ece08c90512473d53143cb9a49de4fcbd84230bc2cd2fcf9b28988\": rpc error: code = NotFound desc = could not find container \"da993cc3c9ece08c90512473d53143cb9a49de4fcbd84230bc2cd2fcf9b28988\": container with ID starting with da993cc3c9ece08c90512473d53143cb9a49de4fcbd84230bc2cd2fcf9b28988 not found: ID does not exist" Sep 17 10:36:23.613830 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:36:23.613756 2581 scope.go:117] "RemoveContainer" containerID="119f23e8169e43350773f0c6b02aa98e75a8cab5db2b96e7480de039485467b0" Sep 17 10:36:23.614061 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:36:23.614042 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"119f23e8169e43350773f0c6b02aa98e75a8cab5db2b96e7480de039485467b0\": container with ID starting with 119f23e8169e43350773f0c6b02aa98e75a8cab5db2b96e7480de039485467b0 not found: ID does not exist" containerID="119f23e8169e43350773f0c6b02aa98e75a8cab5db2b96e7480de039485467b0" Sep 17 10:36:23.614114 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:36:23.614070 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"119f23e8169e43350773f0c6b02aa98e75a8cab5db2b96e7480de039485467b0"} err="failed to get container status \"119f23e8169e43350773f0c6b02aa98e75a8cab5db2b96e7480de039485467b0\": rpc error: code = NotFound desc = could not find container \"119f23e8169e43350773f0c6b02aa98e75a8cab5db2b96e7480de039485467b0\": container with ID starting with 119f23e8169e43350773f0c6b02aa98e75a8cab5db2b96e7480de039485467b0 not found: ID does not exist" Sep 17 10:36:23.614230 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:36:23.614091 2581 scope.go:117] "RemoveContainer" containerID="dfe8cab6ef9454ae74f7fc17ea20077ea9671f0ca33a148a4c08def24dbeef3a" Sep 17 10:36:23.619673 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:36:23.616320 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7zc5b"] Sep 17 10:36:23.619673 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:36:23.616959 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfe8cab6ef9454ae74f7fc17ea20077ea9671f0ca33a148a4c08def24dbeef3a\": container with ID starting with dfe8cab6ef9454ae74f7fc17ea20077ea9671f0ca33a148a4c08def24dbeef3a not found: ID does not exist" containerID="dfe8cab6ef9454ae74f7fc17ea20077ea9671f0ca33a148a4c08def24dbeef3a" Sep 17 10:36:23.619673 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:36:23.616990 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfe8cab6ef9454ae74f7fc17ea20077ea9671f0ca33a148a4c08def24dbeef3a"} err="failed to get container status \"dfe8cab6ef9454ae74f7fc17ea20077ea9671f0ca33a148a4c08def24dbeef3a\": rpc error: code = NotFound desc = could not find container \"dfe8cab6ef9454ae74f7fc17ea20077ea9671f0ca33a148a4c08def24dbeef3a\": container with ID starting with dfe8cab6ef9454ae74f7fc17ea20077ea9671f0ca33a148a4c08def24dbeef3a not found: ID does not exist" Sep 17 10:36:23.619673 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:36:23.617023 2581 scope.go:117] "RemoveContainer" containerID="3a531941343edd3826bd3f9b22a39350d9e7ee225706dfaa8768cfa28f7b713d" Sep 17 10:36:23.619673 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:36:23.617330 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a531941343edd3826bd3f9b22a39350d9e7ee225706dfaa8768cfa28f7b713d\": container with ID starting with 3a531941343edd3826bd3f9b22a39350d9e7ee225706dfaa8768cfa28f7b713d not found: ID does not exist" containerID="3a531941343edd3826bd3f9b22a39350d9e7ee225706dfaa8768cfa28f7b713d" Sep 17 10:36:23.619673 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:36:23.617362 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a531941343edd3826bd3f9b22a39350d9e7ee225706dfaa8768cfa28f7b713d"} err="failed to get container status \"3a531941343edd3826bd3f9b22a39350d9e7ee225706dfaa8768cfa28f7b713d\": rpc error: code = NotFound desc = could not find container \"3a531941343edd3826bd3f9b22a39350d9e7ee225706dfaa8768cfa28f7b713d\": container with ID starting with 3a531941343edd3826bd3f9b22a39350d9e7ee225706dfaa8768cfa28f7b713d not found: ID does not exist" Sep 17 10:36:23.619673 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:36:23.617383 2581 scope.go:117] "RemoveContainer" containerID="f65f00db289356bacf2cd9adbdf973a0a5806b4274f03d2f60329d055c954073" Sep 17 10:36:23.619673 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:36:23.617660 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f65f00db289356bacf2cd9adbdf973a0a5806b4274f03d2f60329d055c954073\": container with ID starting with f65f00db289356bacf2cd9adbdf973a0a5806b4274f03d2f60329d055c954073 not found: ID does not exist" containerID="f65f00db289356bacf2cd9adbdf973a0a5806b4274f03d2f60329d055c954073" Sep 17 10:36:23.619673 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:36:23.617691 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f65f00db289356bacf2cd9adbdf973a0a5806b4274f03d2f60329d055c954073"} err="failed to get container status \"f65f00db289356bacf2cd9adbdf973a0a5806b4274f03d2f60329d055c954073\": rpc error: code = NotFound desc = could not find container \"f65f00db289356bacf2cd9adbdf973a0a5806b4274f03d2f60329d055c954073\": container with ID starting with f65f00db289356bacf2cd9adbdf973a0a5806b4274f03d2f60329d055c954073 not found: ID does not exist" Sep 17 10:36:23.619673 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:36:23.617713 2581 scope.go:117] "RemoveContainer" containerID="7d4f33d36d615f0206a33b05be2a8ba8523d842cfc9db802c6099ef0baba91c4" Sep 17 10:36:23.619673 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:36:23.617994 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d4f33d36d615f0206a33b05be2a8ba8523d842cfc9db802c6099ef0baba91c4\": container with ID starting with 7d4f33d36d615f0206a33b05be2a8ba8523d842cfc9db802c6099ef0baba91c4 not found: ID does not exist" containerID="7d4f33d36d615f0206a33b05be2a8ba8523d842cfc9db802c6099ef0baba91c4" Sep 17 10:36:23.619673 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:36:23.618025 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d4f33d36d615f0206a33b05be2a8ba8523d842cfc9db802c6099ef0baba91c4"} err="failed to get container status \"7d4f33d36d615f0206a33b05be2a8ba8523d842cfc9db802c6099ef0baba91c4\": rpc error: code = NotFound desc = could not find container \"7d4f33d36d615f0206a33b05be2a8ba8523d842cfc9db802c6099ef0baba91c4\": container with ID starting with 7d4f33d36d615f0206a33b05be2a8ba8523d842cfc9db802c6099ef0baba91c4 not found: ID does not exist" Sep 17 10:36:23.619673 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:36:23.618047 2581 scope.go:117] "RemoveContainer" containerID="da993cc3c9ece08c90512473d53143cb9a49de4fcbd84230bc2cd2fcf9b28988" Sep 17 10:36:23.620247 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:36:23.619893 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da993cc3c9ece08c90512473d53143cb9a49de4fcbd84230bc2cd2fcf9b28988"} err="failed to get container status \"da993cc3c9ece08c90512473d53143cb9a49de4fcbd84230bc2cd2fcf9b28988\": rpc error: code = NotFound desc = could not find container \"da993cc3c9ece08c90512473d53143cb9a49de4fcbd84230bc2cd2fcf9b28988\": container with ID starting with da993cc3c9ece08c90512473d53143cb9a49de4fcbd84230bc2cd2fcf9b28988 not found: ID does not exist" Sep 17 10:36:23.620247 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:36:23.619919 2581 scope.go:117] "RemoveContainer" containerID="119f23e8169e43350773f0c6b02aa98e75a8cab5db2b96e7480de039485467b0" Sep 17 10:36:23.620355 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:36:23.620339 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"119f23e8169e43350773f0c6b02aa98e75a8cab5db2b96e7480de039485467b0"} err="failed to get container status \"119f23e8169e43350773f0c6b02aa98e75a8cab5db2b96e7480de039485467b0\": rpc error: code = NotFound desc = could not find container \"119f23e8169e43350773f0c6b02aa98e75a8cab5db2b96e7480de039485467b0\": container with ID starting with 119f23e8169e43350773f0c6b02aa98e75a8cab5db2b96e7480de039485467b0 not found: ID does not exist" Sep 17 10:36:23.620393 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:36:23.620356 2581 scope.go:117] "RemoveContainer" containerID="dfe8cab6ef9454ae74f7fc17ea20077ea9671f0ca33a148a4c08def24dbeef3a" Sep 17 10:36:23.620606 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:36:23.620584 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfe8cab6ef9454ae74f7fc17ea20077ea9671f0ca33a148a4c08def24dbeef3a"} err="failed to get container status \"dfe8cab6ef9454ae74f7fc17ea20077ea9671f0ca33a148a4c08def24dbeef3a\": rpc error: code = NotFound desc = could not find container \"dfe8cab6ef9454ae74f7fc17ea20077ea9671f0ca33a148a4c08def24dbeef3a\": container with ID starting with dfe8cab6ef9454ae74f7fc17ea20077ea9671f0ca33a148a4c08def24dbeef3a not found: ID does not exist" Sep 17 10:36:23.620664 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:36:23.620606 2581 scope.go:117] "RemoveContainer" containerID="3a531941343edd3826bd3f9b22a39350d9e7ee225706dfaa8768cfa28f7b713d" Sep 17 10:36:23.620872 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:36:23.620855 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a531941343edd3826bd3f9b22a39350d9e7ee225706dfaa8768cfa28f7b713d"} err="failed to get container status \"3a531941343edd3826bd3f9b22a39350d9e7ee225706dfaa8768cfa28f7b713d\": rpc error: code = NotFound desc = could not find container \"3a531941343edd3826bd3f9b22a39350d9e7ee225706dfaa8768cfa28f7b713d\": container with ID starting with 3a531941343edd3826bd3f9b22a39350d9e7ee225706dfaa8768cfa28f7b713d not found: ID does not exist" Sep 17 10:36:23.620927 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:36:23.620871 2581 scope.go:117] "RemoveContainer" containerID="f65f00db289356bacf2cd9adbdf973a0a5806b4274f03d2f60329d055c954073" Sep 17 10:36:23.621088 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:36:23.621071 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f65f00db289356bacf2cd9adbdf973a0a5806b4274f03d2f60329d055c954073"} err="failed to get container status \"f65f00db289356bacf2cd9adbdf973a0a5806b4274f03d2f60329d055c954073\": rpc error: code = NotFound desc = could not find container \"f65f00db289356bacf2cd9adbdf973a0a5806b4274f03d2f60329d055c954073\": container with ID starting with f65f00db289356bacf2cd9adbdf973a0a5806b4274f03d2f60329d055c954073 not found: ID does not exist" Sep 17 10:36:23.621128 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:36:23.621089 2581 scope.go:117] "RemoveContainer" containerID="7d4f33d36d615f0206a33b05be2a8ba8523d842cfc9db802c6099ef0baba91c4" Sep 17 10:36:23.623859 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:36:23.623828 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d4f33d36d615f0206a33b05be2a8ba8523d842cfc9db802c6099ef0baba91c4"} err="failed to get container status \"7d4f33d36d615f0206a33b05be2a8ba8523d842cfc9db802c6099ef0baba91c4\": rpc error: code = NotFound desc = could not find container \"7d4f33d36d615f0206a33b05be2a8ba8523d842cfc9db802c6099ef0baba91c4\": container with ID starting with 7d4f33d36d615f0206a33b05be2a8ba8523d842cfc9db802c6099ef0baba91c4 not found: ID does not exist" Sep 17 10:36:23.623955 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:36:23.623867 2581 scope.go:117] "RemoveContainer" containerID="152ddf77d2c02f5c52bfe83b4535eefbc5c595a0ff7418f47cde607f04c9c37a" Sep 17 10:36:23.631996 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:36:23.631981 2581 scope.go:117] "RemoveContainer" containerID="5ca8a7d79ed989a7ead148ee8643585047324b4a83a1f3628437c3fbb9d7657c" Sep 17 10:36:23.639529 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:36:23.639511 2581 scope.go:117] "RemoveContainer" containerID="0c706ebfb44ddc7a91483d8606e2a019adad18f56031ffde9d0633b2a3947bb9" Sep 17 10:36:23.646681 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:36:23.646665 2581 scope.go:117] "RemoveContainer" containerID="863be859f5848919575953b259d52de863bbe70733bd6e53d748c12ac3c0f67a" Sep 17 10:36:23.653717 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:36:23.653699 2581 scope.go:117] "RemoveContainer" containerID="c42d0049732991f9274e79f5a037c60a584f25ac548a184fe130cf268369a119" Sep 17 10:36:23.943770 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:36:23.943693 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ab70dd1-fa52-4bd5-bff1-73591883601d" path="/var/lib/kubelet/pods/4ab70dd1-fa52-4bd5-bff1-73591883601d/volumes" Sep 17 10:36:23.944275 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:36:23.944261 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e7c9bdd3-0224-4a08-98c1-d0a648f8f70a" path="/var/lib/kubelet/pods/e7c9bdd3-0224-4a08-98c1-d0a648f8f70a/volumes" Sep 17 10:37:04.350366 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:37:04.350330 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-xq9vr"] Sep 17 10:37:04.350820 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:37:04.350641 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ab70dd1-fa52-4bd5-bff1-73591883601d" containerName="feast-init" Sep 17 10:37:04.350820 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:37:04.350652 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ab70dd1-fa52-4bd5-bff1-73591883601d" containerName="feast-init" Sep 17 10:37:04.350820 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:37:04.350663 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7c9bdd3-0224-4a08-98c1-d0a648f8f70a" containerName="ui" Sep 17 10:37:04.350820 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:37:04.350669 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7c9bdd3-0224-4a08-98c1-d0a648f8f70a" containerName="ui" Sep 17 10:37:04.350820 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:37:04.350679 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7c9bdd3-0224-4a08-98c1-d0a648f8f70a" containerName="feast-apply" Sep 17 10:37:04.350820 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:37:04.350684 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7c9bdd3-0224-4a08-98c1-d0a648f8f70a" containerName="feast-apply" Sep 17 10:37:04.350820 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:37:04.350692 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7c9bdd3-0224-4a08-98c1-d0a648f8f70a" containerName="offline" Sep 17 10:37:04.350820 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:37:04.350698 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7c9bdd3-0224-4a08-98c1-d0a648f8f70a" containerName="offline" Sep 17 10:37:04.350820 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:37:04.350706 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ab70dd1-fa52-4bd5-bff1-73591883601d" containerName="feast-apply" Sep 17 10:37:04.350820 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:37:04.350712 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ab70dd1-fa52-4bd5-bff1-73591883601d" containerName="feast-apply" Sep 17 10:37:04.350820 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:37:04.350717 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7c9bdd3-0224-4a08-98c1-d0a648f8f70a" containerName="online" Sep 17 10:37:04.350820 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:37:04.350722 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7c9bdd3-0224-4a08-98c1-d0a648f8f70a" containerName="online" Sep 17 10:37:04.350820 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:37:04.350727 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7c9bdd3-0224-4a08-98c1-d0a648f8f70a" containerName="registry" Sep 17 10:37:04.350820 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:37:04.350732 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7c9bdd3-0224-4a08-98c1-d0a648f8f70a" containerName="registry" Sep 17 10:37:04.350820 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:37:04.350739 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ab70dd1-fa52-4bd5-bff1-73591883601d" containerName="online" Sep 17 10:37:04.350820 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:37:04.350744 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ab70dd1-fa52-4bd5-bff1-73591883601d" containerName="online" Sep 17 10:37:04.350820 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:37:04.350751 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ab70dd1-fa52-4bd5-bff1-73591883601d" containerName="offline" Sep 17 10:37:04.350820 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:37:04.350756 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ab70dd1-fa52-4bd5-bff1-73591883601d" containerName="offline" Sep 17 10:37:04.350820 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:37:04.350765 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7c9bdd3-0224-4a08-98c1-d0a648f8f70a" containerName="feast-init" Sep 17 10:37:04.350820 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:37:04.350770 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7c9bdd3-0224-4a08-98c1-d0a648f8f70a" containerName="feast-init" Sep 17 10:37:04.350820 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:37:04.350792 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ab70dd1-fa52-4bd5-bff1-73591883601d" containerName="ui" Sep 17 10:37:04.350820 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:37:04.350797 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ab70dd1-fa52-4bd5-bff1-73591883601d" containerName="ui" Sep 17 10:37:04.351483 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:37:04.350842 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ab70dd1-fa52-4bd5-bff1-73591883601d" containerName="online" Sep 17 10:37:04.351483 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:37:04.350850 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7c9bdd3-0224-4a08-98c1-d0a648f8f70a" containerName="offline" Sep 17 10:37:04.351483 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:37:04.350855 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7c9bdd3-0224-4a08-98c1-d0a648f8f70a" containerName="registry" Sep 17 10:37:04.351483 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:37:04.350860 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ab70dd1-fa52-4bd5-bff1-73591883601d" containerName="ui" Sep 17 10:37:04.351483 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:37:04.350867 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7c9bdd3-0224-4a08-98c1-d0a648f8f70a" containerName="online" Sep 17 10:37:04.351483 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:37:04.350872 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7c9bdd3-0224-4a08-98c1-d0a648f8f70a" containerName="ui" Sep 17 10:37:04.351483 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:37:04.350876 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ab70dd1-fa52-4bd5-bff1-73591883601d" containerName="offline" Sep 17 10:37:04.354184 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:37:04.354165 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xq9vr" Sep 17 10:37:04.357519 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:37:04.357494 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-j28bh\"" Sep 17 10:37:04.357638 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:37:04.357517 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-online-tls\"" Sep 17 10:37:04.357638 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:37:04.357521 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-ui-tls\"" Sep 17 10:37:04.358624 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:37:04.358604 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\"" Sep 17 10:37:04.358688 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:37:04.358606 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-registry-tls\"" Sep 17 10:37:04.358688 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:37:04.358615 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-offline-tls\"" Sep 17 10:37:04.358767 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:37:04.358684 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\"" Sep 17 10:37:04.369090 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:37:04.369065 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-xq9vr"] Sep 17 10:37:04.497536 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:37:04.497492 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/a62e777d-2ea3-4a50-8aeb-33228f458053-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-xq9vr\" (UID: \"a62e777d-2ea3-4a50-8aeb-33228f458053\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xq9vr" Sep 17 10:37:04.497536 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:37:04.497536 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8vbw\" (UniqueName: \"kubernetes.io/projected/a62e777d-2ea3-4a50-8aeb-33228f458053-kube-api-access-l8vbw\") pod \"feast-simple-feast-setup-57576b8fcf-xq9vr\" (UID: \"a62e777d-2ea3-4a50-8aeb-33228f458053\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xq9vr" Sep 17 10:37:04.497762 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:37:04.497620 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/a62e777d-2ea3-4a50-8aeb-33228f458053-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-xq9vr\" (UID: \"a62e777d-2ea3-4a50-8aeb-33228f458053\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xq9vr" Sep 17 10:37:04.497762 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:37:04.497701 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/a62e777d-2ea3-4a50-8aeb-33228f458053-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-xq9vr\" (UID: \"a62e777d-2ea3-4a50-8aeb-33228f458053\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xq9vr" Sep 17 10:37:04.497762 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:37:04.497726 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/a62e777d-2ea3-4a50-8aeb-33228f458053-feast-data\") pod \"feast-simple-feast-setup-57576b8fcf-xq9vr\" (UID: \"a62e777d-2ea3-4a50-8aeb-33228f458053\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xq9vr" Sep 17 10:37:04.497762 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:37:04.497748 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/a62e777d-2ea3-4a50-8aeb-33228f458053-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-xq9vr\" (UID: \"a62e777d-2ea3-4a50-8aeb-33228f458053\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xq9vr" Sep 17 10:37:04.598742 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:37:04.598701 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/a62e777d-2ea3-4a50-8aeb-33228f458053-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-xq9vr\" (UID: \"a62e777d-2ea3-4a50-8aeb-33228f458053\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xq9vr" Sep 17 10:37:04.598742 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:37:04.598741 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/a62e777d-2ea3-4a50-8aeb-33228f458053-feast-data\") pod \"feast-simple-feast-setup-57576b8fcf-xq9vr\" (UID: \"a62e777d-2ea3-4a50-8aeb-33228f458053\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xq9vr" Sep 17 10:37:04.598996 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:37:04.598769 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/a62e777d-2ea3-4a50-8aeb-33228f458053-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-xq9vr\" (UID: \"a62e777d-2ea3-4a50-8aeb-33228f458053\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xq9vr" Sep 17 10:37:04.598996 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:37:04.598851 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/a62e777d-2ea3-4a50-8aeb-33228f458053-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-xq9vr\" (UID: \"a62e777d-2ea3-4a50-8aeb-33228f458053\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xq9vr" Sep 17 10:37:04.598996 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:37:04.598879 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8vbw\" (UniqueName: \"kubernetes.io/projected/a62e777d-2ea3-4a50-8aeb-33228f458053-kube-api-access-l8vbw\") pod \"feast-simple-feast-setup-57576b8fcf-xq9vr\" (UID: \"a62e777d-2ea3-4a50-8aeb-33228f458053\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xq9vr" Sep 17 10:37:04.599153 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:37:04.599016 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/a62e777d-2ea3-4a50-8aeb-33228f458053-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-xq9vr\" (UID: \"a62e777d-2ea3-4a50-8aeb-33228f458053\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xq9vr" Sep 17 10:37:04.599217 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:37:04.599174 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/a62e777d-2ea3-4a50-8aeb-33228f458053-feast-data\") pod \"feast-simple-feast-setup-57576b8fcf-xq9vr\" (UID: \"a62e777d-2ea3-4a50-8aeb-33228f458053\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xq9vr" Sep 17 10:37:04.601409 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:37:04.601339 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/a62e777d-2ea3-4a50-8aeb-33228f458053-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-xq9vr\" (UID: \"a62e777d-2ea3-4a50-8aeb-33228f458053\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xq9vr" Sep 17 10:37:04.601538 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:37:04.601492 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/a62e777d-2ea3-4a50-8aeb-33228f458053-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-xq9vr\" (UID: \"a62e777d-2ea3-4a50-8aeb-33228f458053\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xq9vr" Sep 17 10:37:04.601538 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:37:04.601514 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/a62e777d-2ea3-4a50-8aeb-33228f458053-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-xq9vr\" (UID: \"a62e777d-2ea3-4a50-8aeb-33228f458053\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xq9vr" Sep 17 10:37:04.601652 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:37:04.601514 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/a62e777d-2ea3-4a50-8aeb-33228f458053-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-xq9vr\" (UID: \"a62e777d-2ea3-4a50-8aeb-33228f458053\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xq9vr" Sep 17 10:37:04.614973 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:37:04.614948 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8vbw\" (UniqueName: \"kubernetes.io/projected/a62e777d-2ea3-4a50-8aeb-33228f458053-kube-api-access-l8vbw\") pod \"feast-simple-feast-setup-57576b8fcf-xq9vr\" (UID: \"a62e777d-2ea3-4a50-8aeb-33228f458053\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xq9vr" Sep 17 10:37:04.663589 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:37:04.663546 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xq9vr" Sep 17 10:37:04.941809 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:37:04.941769 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-xq9vr"] Sep 17 10:37:04.943767 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:37:04.943743 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda62e777d_2ea3_4a50_8aeb_33228f458053.slice/crio-040f412f2e5e8de17c49ee027367cc361ee96e35af825346bc0807b2ee7e8190 WatchSource:0}: Error finding container 040f412f2e5e8de17c49ee027367cc361ee96e35af825346bc0807b2ee7e8190: Status 404 returned error can't find the container with id 040f412f2e5e8de17c49ee027367cc361ee96e35af825346bc0807b2ee7e8190 Sep 17 10:37:05.736817 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:37:05.736763 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xq9vr" event={"ID":"a62e777d-2ea3-4a50-8aeb-33228f458053","Type":"ContainerStarted","Data":"0e94a99378fb0de47adfea97671b9716e9353b975cdba4ecb884bc6ae2f994da"} Sep 17 10:37:05.737209 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:37:05.736831 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xq9vr" event={"ID":"a62e777d-2ea3-4a50-8aeb-33228f458053","Type":"ContainerStarted","Data":"040f412f2e5e8de17c49ee027367cc361ee96e35af825346bc0807b2ee7e8190"} Sep 17 10:37:10.759260 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:37:10.759224 2581 generic.go:356] "Generic (PLEG): container finished" podID="a62e777d-2ea3-4a50-8aeb-33228f458053" containerID="0e94a99378fb0de47adfea97671b9716e9353b975cdba4ecb884bc6ae2f994da" exitCode=0 Sep 17 10:37:10.759664 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:37:10.759264 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xq9vr" event={"ID":"a62e777d-2ea3-4a50-8aeb-33228f458053","Type":"ContainerDied","Data":"0e94a99378fb0de47adfea97671b9716e9353b975cdba4ecb884bc6ae2f994da"} Sep 17 10:37:11.765501 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:37:11.765461 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xq9vr" event={"ID":"a62e777d-2ea3-4a50-8aeb-33228f458053","Type":"ContainerStarted","Data":"acfddbc6ce9c7c9b36a061de9802ade95be64a0ff064907e94007a3134bd41a5"} Sep 17 10:37:17.790862 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:37:17.790824 2581 generic.go:356] "Generic (PLEG): container finished" podID="a62e777d-2ea3-4a50-8aeb-33228f458053" containerID="acfddbc6ce9c7c9b36a061de9802ade95be64a0ff064907e94007a3134bd41a5" exitCode=0 Sep 17 10:37:17.791299 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:37:17.790892 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xq9vr" event={"ID":"a62e777d-2ea3-4a50-8aeb-33228f458053","Type":"ContainerDied","Data":"acfddbc6ce9c7c9b36a061de9802ade95be64a0ff064907e94007a3134bd41a5"} Sep 17 10:37:18.801494 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:37:18.801442 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xq9vr" event={"ID":"a62e777d-2ea3-4a50-8aeb-33228f458053","Type":"ContainerStarted","Data":"b213c6c897dec9d48cb8f4978757d32cb999a7129871a742e58f145a1b7d8b73"} Sep 17 10:37:18.802003 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:37:18.801509 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xq9vr" event={"ID":"a62e777d-2ea3-4a50-8aeb-33228f458053","Type":"ContainerStarted","Data":"aee50c8312a03578ab6533870a2a0e378b18fd68ac65eb4733f3d78a02a90d5c"} Sep 17 10:37:18.802003 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:37:18.801531 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xq9vr" event={"ID":"a62e777d-2ea3-4a50-8aeb-33228f458053","Type":"ContainerStarted","Data":"ae586badc5f49b3d2337de13fbcadfe2b504fafaf8640b881229fea636e2f389"} Sep 17 10:37:18.802003 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:37:18.801550 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xq9vr" event={"ID":"a62e777d-2ea3-4a50-8aeb-33228f458053","Type":"ContainerStarted","Data":"5246f21b58a6ce5e4d42dd31d22f0a35650debc85504473d460b6c8fb23b4076"} Sep 17 10:37:18.833048 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:37:18.832982 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xq9vr" podStartSLOduration=14.83295652 podStartE2EDuration="14.83295652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 10:37:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 10:37:18.82985307 +0000 UTC m=+1593.378410183" watchObservedRunningTime="2026-09-17 10:37:18.83295652 +0000 UTC m=+1593.381513632" Sep 17 10:37:19.664411 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:37:19.664372 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xq9vr" Sep 17 10:37:19.664717 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:37:19.664691 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xq9vr" Sep 17 10:37:19.664717 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:37:19.664724 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xq9vr" Sep 17 10:37:19.664960 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:37:19.664742 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xq9vr" Sep 17 10:37:19.666672 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:37:19.666633 2581 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xq9vr" podUID="a62e777d-2ea3-4a50-8aeb-33228f458053" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.38:6571: connect: connection refused" Sep 17 10:37:19.666820 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:37:19.666668 2581 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xq9vr" podUID="a62e777d-2ea3-4a50-8aeb-33228f458053" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.38:8443: connect: connection refused" Sep 17 10:37:19.666820 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:37:19.666689 2581 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xq9vr" podUID="a62e777d-2ea3-4a50-8aeb-33228f458053" containerName="online" probeResult="failure" output="Get \"https://10.134.0.38:6567/health\": dial tcp 10.134.0.38:6567: connect: connection refused" Sep 17 10:37:19.666820 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:37:19.666754 2581 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xq9vr" podUID="a62e777d-2ea3-4a50-8aeb-33228f458053" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.38:8816: connect: connection refused" Sep 17 10:37:22.664839 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:37:22.664738 2581 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xq9vr" podUID="a62e777d-2ea3-4a50-8aeb-33228f458053" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.38:8816: connect: connection refused" Sep 17 10:37:22.664839 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:37:22.664816 2581 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xq9vr" podUID="a62e777d-2ea3-4a50-8aeb-33228f458053" containerName="online" probeResult="failure" output="Get \"https://10.134.0.38:6567/health\": dial tcp 10.134.0.38:6567: connect: connection refused" Sep 17 10:37:22.665389 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:37:22.664732 2581 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xq9vr" podUID="a62e777d-2ea3-4a50-8aeb-33228f458053" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.38:8443: connect: connection refused" Sep 17 10:37:22.665389 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:37:22.664732 2581 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xq9vr" podUID="a62e777d-2ea3-4a50-8aeb-33228f458053" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.38:6571: connect: connection refused" Sep 17 10:37:24.664398 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:37:24.664342 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xq9vr" Sep 17 10:37:24.664920 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:37:24.664429 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xq9vr" Sep 17 10:37:24.664920 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:37:24.664451 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xq9vr" Sep 17 10:37:24.664920 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:37:24.664472 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xq9vr" Sep 17 10:37:25.664910 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:37:25.664868 2581 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xq9vr" podUID="a62e777d-2ea3-4a50-8aeb-33228f458053" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.38:8443: connect: connection refused" Sep 17 10:37:25.665359 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:37:25.664877 2581 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xq9vr" podUID="a62e777d-2ea3-4a50-8aeb-33228f458053" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.38:6571: connect: connection refused" Sep 17 10:37:25.667116 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:37:25.667083 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xq9vr" Sep 17 10:37:25.668091 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:37:25.668067 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xq9vr" Sep 17 10:37:25.762031 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:37:25.762000 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xq9vr" Sep 17 10:37:25.847111 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:37:25.847068 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xq9vr" Sep 17 10:37:28.665030 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:37:28.664997 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xq9vr" Sep 17 10:37:28.665461 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:37:28.665310 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xq9vr" Sep 17 10:37:28.857818 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:37:28.857764 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xq9vr" Sep 17 10:37:28.857986 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:37:28.857929 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xq9vr" Sep 17 10:38:29.996234 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:38:29.996199 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-xq9vr"] Sep 17 10:38:29.998148 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:38:29.998082 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xq9vr" podUID="a62e777d-2ea3-4a50-8aeb-33228f458053" containerName="registry" containerID="cri-o://5246f21b58a6ce5e4d42dd31d22f0a35650debc85504473d460b6c8fb23b4076" gracePeriod=30 Sep 17 10:38:29.998376 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:38:29.998090 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xq9vr" podUID="a62e777d-2ea3-4a50-8aeb-33228f458053" containerName="offline" containerID="cri-o://aee50c8312a03578ab6533870a2a0e378b18fd68ac65eb4733f3d78a02a90d5c" gracePeriod=30 Sep 17 10:38:29.998658 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:38:29.998154 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xq9vr" podUID="a62e777d-2ea3-4a50-8aeb-33228f458053" containerName="ui" containerID="cri-o://b213c6c897dec9d48cb8f4978757d32cb999a7129871a742e58f145a1b7d8b73" gracePeriod=30 Sep 17 10:38:29.998658 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:38:29.998173 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xq9vr" podUID="a62e777d-2ea3-4a50-8aeb-33228f458053" containerName="online" containerID="cri-o://ae586badc5f49b3d2337de13fbcadfe2b504fafaf8640b881229fea636e2f389" gracePeriod=30 Sep 17 10:38:30.450033 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:38:30.449938 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-7pfpb"] Sep 17 10:38:30.454427 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:38:30.454405 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7pfpb" Sep 17 10:38:30.457047 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:38:30.457009 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-zlgjd\"" Sep 17 10:38:30.466383 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:38:30.466353 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-7pfpb"] Sep 17 10:38:30.508280 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:38:30.508234 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzm4t\" (UniqueName: \"kubernetes.io/projected/7358201a-ed56-4a29-9dc9-fe48ae81d86d-kube-api-access-tzm4t\") pod \"feast-simple-feast-setup-57576b8fcf-7pfpb\" (UID: \"7358201a-ed56-4a29-9dc9-fe48ae81d86d\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7pfpb" Sep 17 10:38:30.508280 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:38:30.508281 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/7358201a-ed56-4a29-9dc9-fe48ae81d86d-feast-data\") pod \"feast-simple-feast-setup-57576b8fcf-7pfpb\" (UID: \"7358201a-ed56-4a29-9dc9-fe48ae81d86d\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7pfpb" Sep 17 10:38:30.508534 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:38:30.508307 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/7358201a-ed56-4a29-9dc9-fe48ae81d86d-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-7pfpb\" (UID: \"7358201a-ed56-4a29-9dc9-fe48ae81d86d\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7pfpb" Sep 17 10:38:30.508534 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:38:30.508337 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/7358201a-ed56-4a29-9dc9-fe48ae81d86d-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-7pfpb\" (UID: \"7358201a-ed56-4a29-9dc9-fe48ae81d86d\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7pfpb" Sep 17 10:38:30.508534 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:38:30.508366 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/7358201a-ed56-4a29-9dc9-fe48ae81d86d-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-7pfpb\" (UID: \"7358201a-ed56-4a29-9dc9-fe48ae81d86d\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7pfpb" Sep 17 10:38:30.508534 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:38:30.508388 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/7358201a-ed56-4a29-9dc9-fe48ae81d86d-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-7pfpb\" (UID: \"7358201a-ed56-4a29-9dc9-fe48ae81d86d\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7pfpb" Sep 17 10:38:30.609169 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:38:30.609126 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/7358201a-ed56-4a29-9dc9-fe48ae81d86d-feast-data\") pod \"feast-simple-feast-setup-57576b8fcf-7pfpb\" (UID: \"7358201a-ed56-4a29-9dc9-fe48ae81d86d\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7pfpb" Sep 17 10:38:30.609360 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:38:30.609178 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/7358201a-ed56-4a29-9dc9-fe48ae81d86d-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-7pfpb\" (UID: \"7358201a-ed56-4a29-9dc9-fe48ae81d86d\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7pfpb" Sep 17 10:38:30.609360 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:38:30.609241 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/7358201a-ed56-4a29-9dc9-fe48ae81d86d-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-7pfpb\" (UID: \"7358201a-ed56-4a29-9dc9-fe48ae81d86d\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7pfpb" Sep 17 10:38:30.609360 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:38:30.609286 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/7358201a-ed56-4a29-9dc9-fe48ae81d86d-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-7pfpb\" (UID: \"7358201a-ed56-4a29-9dc9-fe48ae81d86d\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7pfpb" Sep 17 10:38:30.609360 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:38:30.609293 2581 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-ui-tls: secret "feast-simple-feast-setup-ui-tls" not found Sep 17 10:38:30.609360 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:38:30.609316 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/7358201a-ed56-4a29-9dc9-fe48ae81d86d-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-7pfpb\" (UID: \"7358201a-ed56-4a29-9dc9-fe48ae81d86d\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7pfpb" Sep 17 10:38:30.609619 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:38:30.609367 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7358201a-ed56-4a29-9dc9-fe48ae81d86d-ui-tls podName:7358201a-ed56-4a29-9dc9-fe48ae81d86d nodeName:}" failed. No retries permitted until 2026-09-17 10:38:31.109342541 +0000 UTC m=+1665.657899630 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "ui-tls" (UniqueName: "kubernetes.io/secret/7358201a-ed56-4a29-9dc9-fe48ae81d86d-ui-tls") pod "feast-simple-feast-setup-57576b8fcf-7pfpb" (UID: "7358201a-ed56-4a29-9dc9-fe48ae81d86d") : secret "feast-simple-feast-setup-ui-tls" not found Sep 17 10:38:30.609619 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:38:30.609417 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzm4t\" (UniqueName: \"kubernetes.io/projected/7358201a-ed56-4a29-9dc9-fe48ae81d86d-kube-api-access-tzm4t\") pod \"feast-simple-feast-setup-57576b8fcf-7pfpb\" (UID: \"7358201a-ed56-4a29-9dc9-fe48ae81d86d\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7pfpb" Sep 17 10:38:30.609619 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:38:30.609583 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/7358201a-ed56-4a29-9dc9-fe48ae81d86d-feast-data\") pod \"feast-simple-feast-setup-57576b8fcf-7pfpb\" (UID: \"7358201a-ed56-4a29-9dc9-fe48ae81d86d\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7pfpb" Sep 17 10:38:30.612199 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:38:30.612172 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/7358201a-ed56-4a29-9dc9-fe48ae81d86d-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-7pfpb\" (UID: \"7358201a-ed56-4a29-9dc9-fe48ae81d86d\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7pfpb" Sep 17 10:38:30.612387 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:38:30.612369 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/7358201a-ed56-4a29-9dc9-fe48ae81d86d-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-7pfpb\" (UID: \"7358201a-ed56-4a29-9dc9-fe48ae81d86d\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7pfpb" Sep 17 10:38:30.612608 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:38:30.612587 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/7358201a-ed56-4a29-9dc9-fe48ae81d86d-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-7pfpb\" (UID: \"7358201a-ed56-4a29-9dc9-fe48ae81d86d\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7pfpb" Sep 17 10:38:30.620931 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:38:30.620904 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzm4t\" (UniqueName: \"kubernetes.io/projected/7358201a-ed56-4a29-9dc9-fe48ae81d86d-kube-api-access-tzm4t\") pod \"feast-simple-feast-setup-57576b8fcf-7pfpb\" (UID: \"7358201a-ed56-4a29-9dc9-fe48ae81d86d\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7pfpb" Sep 17 10:38:31.106454 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:38:31.106421 2581 generic.go:356] "Generic (PLEG): container finished" podID="a62e777d-2ea3-4a50-8aeb-33228f458053" containerID="b213c6c897dec9d48cb8f4978757d32cb999a7129871a742e58f145a1b7d8b73" exitCode=0 Sep 17 10:38:31.106454 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:38:31.106445 2581 generic.go:356] "Generic (PLEG): container finished" podID="a62e777d-2ea3-4a50-8aeb-33228f458053" containerID="ae586badc5f49b3d2337de13fbcadfe2b504fafaf8640b881229fea636e2f389" exitCode=0 Sep 17 10:38:31.106884 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:38:31.106467 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xq9vr" event={"ID":"a62e777d-2ea3-4a50-8aeb-33228f458053","Type":"ContainerDied","Data":"b213c6c897dec9d48cb8f4978757d32cb999a7129871a742e58f145a1b7d8b73"} Sep 17 10:38:31.106884 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:38:31.106497 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xq9vr" event={"ID":"a62e777d-2ea3-4a50-8aeb-33228f458053","Type":"ContainerDied","Data":"ae586badc5f49b3d2337de13fbcadfe2b504fafaf8640b881229fea636e2f389"} Sep 17 10:38:31.114828 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:38:31.114802 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/7358201a-ed56-4a29-9dc9-fe48ae81d86d-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-7pfpb\" (UID: \"7358201a-ed56-4a29-9dc9-fe48ae81d86d\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7pfpb" Sep 17 10:38:31.117285 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:38:31.117264 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/7358201a-ed56-4a29-9dc9-fe48ae81d86d-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-7pfpb\" (UID: \"7358201a-ed56-4a29-9dc9-fe48ae81d86d\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7pfpb" Sep 17 10:38:31.364155 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:38:31.364050 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7pfpb" Sep 17 10:38:31.639973 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:38:31.639943 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-7pfpb"] Sep 17 10:38:31.641343 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:38:31.641318 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7358201a_ed56_4a29_9dc9_fe48ae81d86d.slice/crio-582c1059c2b0e21d79599a16a5f74d366fe472dcd61fa8927e4a737b2a4c3058 WatchSource:0}: Error finding container 582c1059c2b0e21d79599a16a5f74d366fe472dcd61fa8927e4a737b2a4c3058: Status 404 returned error can't find the container with id 582c1059c2b0e21d79599a16a5f74d366fe472dcd61fa8927e4a737b2a4c3058 Sep 17 10:38:32.112753 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:38:32.112712 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7pfpb" event={"ID":"7358201a-ed56-4a29-9dc9-fe48ae81d86d","Type":"ContainerStarted","Data":"1d385642e51926df69d994fed7efb0b7a3bb5f4be40014386b3d2e2be36a5ffd"} Sep 17 10:38:32.113265 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:38:32.112851 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7pfpb" event={"ID":"7358201a-ed56-4a29-9dc9-fe48ae81d86d","Type":"ContainerStarted","Data":"582c1059c2b0e21d79599a16a5f74d366fe472dcd61fa8927e4a737b2a4c3058"} Sep 17 10:38:35.844590 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:38:35.844540 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xq9vr" podUID="a62e777d-2ea3-4a50-8aeb-33228f458053" containerName="online" probeResult="failure" output="Get \"https://10.134.0.38:6567/health\": dial tcp 10.134.0.38:6567: connect: connection refused" Sep 17 10:38:37.136319 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:38:37.136287 2581 generic.go:356] "Generic (PLEG): container finished" podID="7358201a-ed56-4a29-9dc9-fe48ae81d86d" containerID="1d385642e51926df69d994fed7efb0b7a3bb5f4be40014386b3d2e2be36a5ffd" exitCode=0 Sep 17 10:38:37.136737 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:38:37.136351 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7pfpb" event={"ID":"7358201a-ed56-4a29-9dc9-fe48ae81d86d","Type":"ContainerDied","Data":"1d385642e51926df69d994fed7efb0b7a3bb5f4be40014386b3d2e2be36a5ffd"} Sep 17 10:38:38.142018 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:38:38.141982 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7pfpb" event={"ID":"7358201a-ed56-4a29-9dc9-fe48ae81d86d","Type":"ContainerStarted","Data":"b06c7cb66012d4dbd88d6f7df026dd20e5bcfb2564c9f151f1ee113777f84821"} Sep 17 10:38:38.857855 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:38:38.857769 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xq9vr" podUID="a62e777d-2ea3-4a50-8aeb-33228f458053" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.38:8443: connect: connection refused" Sep 17 10:38:43.164470 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:38:43.164438 2581 generic.go:356] "Generic (PLEG): container finished" podID="7358201a-ed56-4a29-9dc9-fe48ae81d86d" containerID="b06c7cb66012d4dbd88d6f7df026dd20e5bcfb2564c9f151f1ee113777f84821" exitCode=0 Sep 17 10:38:43.164949 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:38:43.164508 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7pfpb" event={"ID":"7358201a-ed56-4a29-9dc9-fe48ae81d86d","Type":"ContainerDied","Data":"b06c7cb66012d4dbd88d6f7df026dd20e5bcfb2564c9f151f1ee113777f84821"} Sep 17 10:38:44.174631 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:38:44.174589 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7pfpb" event={"ID":"7358201a-ed56-4a29-9dc9-fe48ae81d86d","Type":"ContainerStarted","Data":"ba269b09dbd046ac1d48d177c03d810b72836f6f10af8781d13fc6341045c08f"} Sep 17 10:38:44.175127 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:38:44.174657 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7pfpb" event={"ID":"7358201a-ed56-4a29-9dc9-fe48ae81d86d","Type":"ContainerStarted","Data":"2da0d58ed8adf62e5e7ce1f57f119ee9fa77fd4ef2e51f71aa3129fb304631ca"} Sep 17 10:38:44.175127 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:38:44.174679 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7pfpb" event={"ID":"7358201a-ed56-4a29-9dc9-fe48ae81d86d","Type":"ContainerStarted","Data":"417ee1a6cceb0b799231c29b5255f04e400cf1e4d02f676ead26537ec1ccc53d"} Sep 17 10:38:44.175127 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:38:44.174697 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7pfpb" event={"ID":"7358201a-ed56-4a29-9dc9-fe48ae81d86d","Type":"ContainerStarted","Data":"aac1f759bc68265a73d423fc4564b550711f117319cf81f4c2ea4bbc237d96e3"} Sep 17 10:38:44.201556 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:38:44.201484 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7pfpb" podStartSLOduration=14.201462683 podStartE2EDuration="14.201462683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 10:38:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 10:38:44.198138841 +0000 UTC m=+1678.746695953" watchObservedRunningTime="2026-09-17 10:38:44.201462683 +0000 UTC m=+1678.750019795" Sep 17 10:38:45.844219 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:38:45.844173 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xq9vr" podUID="a62e777d-2ea3-4a50-8aeb-33228f458053" containerName="online" probeResult="failure" output="Get \"https://10.134.0.38:6567/health\": dial tcp 10.134.0.38:6567: connect: connection refused" Sep 17 10:38:46.364264 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:38:46.364214 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7pfpb" Sep 17 10:38:46.364479 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:38:46.364277 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7pfpb" Sep 17 10:38:46.364957 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:38:46.364880 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7pfpb" Sep 17 10:38:46.364957 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:38:46.364927 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7pfpb" Sep 17 10:38:46.366278 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:38:46.366241 2581 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7pfpb" podUID="7358201a-ed56-4a29-9dc9-fe48ae81d86d" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.39:6571: connect: connection refused" Sep 17 10:38:46.366382 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:38:46.366273 2581 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7pfpb" podUID="7358201a-ed56-4a29-9dc9-fe48ae81d86d" containerName="online" probeResult="failure" output="Get \"https://10.134.0.39:6567/health\": dial tcp 10.134.0.39:6567: connect: connection refused" Sep 17 10:38:46.366382 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:38:46.366318 2581 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7pfpb" podUID="7358201a-ed56-4a29-9dc9-fe48ae81d86d" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.39:8816: connect: connection refused" Sep 17 10:38:46.366382 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:38:46.366260 2581 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7pfpb" podUID="7358201a-ed56-4a29-9dc9-fe48ae81d86d" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.39:8443: connect: connection refused" Sep 17 10:38:48.857395 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:38:48.857343 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xq9vr" podUID="a62e777d-2ea3-4a50-8aeb-33228f458053" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.38:8443: connect: connection refused" Sep 17 10:38:49.365152 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:38:49.365104 2581 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7pfpb" podUID="7358201a-ed56-4a29-9dc9-fe48ae81d86d" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.39:8816: connect: connection refused" Sep 17 10:38:49.365371 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:38:49.365104 2581 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7pfpb" podUID="7358201a-ed56-4a29-9dc9-fe48ae81d86d" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.39:8443: connect: connection refused" Sep 17 10:38:49.365371 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:38:49.365196 2581 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7pfpb" podUID="7358201a-ed56-4a29-9dc9-fe48ae81d86d" containerName="online" probeResult="failure" output="Get \"https://10.134.0.39:6567/health\": dial tcp 10.134.0.39:6567: connect: connection refused" Sep 17 10:38:49.365618 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:38:49.365591 2581 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7pfpb" podUID="7358201a-ed56-4a29-9dc9-fe48ae81d86d" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.39:6571: connect: connection refused" Sep 17 10:38:51.364580 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:38:51.364536 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7pfpb" Sep 17 10:38:51.365117 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:38:51.364613 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7pfpb" Sep 17 10:38:51.365117 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:38:51.364643 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7pfpb" Sep 17 10:38:51.365117 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:38:51.364660 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7pfpb" Sep 17 10:38:52.365673 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:38:52.365640 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7pfpb" Sep 17 10:38:52.366050 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:38:52.365809 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7pfpb" Sep 17 10:38:52.366110 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:38:52.366059 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7pfpb" Sep 17 10:38:52.370677 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:38:52.370657 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7pfpb" Sep 17 10:38:53.224981 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:38:53.224949 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7pfpb" Sep 17 10:38:53.225168 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:38:53.225029 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7pfpb" Sep 17 10:38:53.225168 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:38:53.225147 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7pfpb" Sep 17 10:38:53.227936 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:38:53.227908 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7pfpb" Sep 17 10:38:55.844021 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:38:55.843980 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xq9vr" podUID="a62e777d-2ea3-4a50-8aeb-33228f458053" containerName="online" probeResult="failure" output="Get \"https://10.134.0.38:6567/health\": dial tcp 10.134.0.38:6567: connect: connection refused" Sep 17 10:38:55.844413 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:38:55.844096 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xq9vr" Sep 17 10:38:58.857927 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:38:58.857883 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xq9vr" podUID="a62e777d-2ea3-4a50-8aeb-33228f458053" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.38:8443: connect: connection refused" Sep 17 10:38:58.858347 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:38:58.858021 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xq9vr" Sep 17 10:39:00.255935 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:39:00.255848 2581 generic.go:356] "Generic (PLEG): container finished" podID="a62e777d-2ea3-4a50-8aeb-33228f458053" containerID="aee50c8312a03578ab6533870a2a0e378b18fd68ac65eb4733f3d78a02a90d5c" exitCode=137 Sep 17 10:39:00.255935 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:39:00.255872 2581 generic.go:356] "Generic (PLEG): container finished" podID="a62e777d-2ea3-4a50-8aeb-33228f458053" containerID="5246f21b58a6ce5e4d42dd31d22f0a35650debc85504473d460b6c8fb23b4076" exitCode=137 Sep 17 10:39:00.255935 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:39:00.255916 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xq9vr" event={"ID":"a62e777d-2ea3-4a50-8aeb-33228f458053","Type":"ContainerDied","Data":"aee50c8312a03578ab6533870a2a0e378b18fd68ac65eb4733f3d78a02a90d5c"} Sep 17 10:39:00.256356 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:39:00.255960 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xq9vr" event={"ID":"a62e777d-2ea3-4a50-8aeb-33228f458053","Type":"ContainerDied","Data":"5246f21b58a6ce5e4d42dd31d22f0a35650debc85504473d460b6c8fb23b4076"} Sep 17 10:39:00.650570 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:39:00.650542 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xq9vr" Sep 17 10:39:00.690353 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:39:00.690321 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/a62e777d-2ea3-4a50-8aeb-33228f458053-registry-tls\") pod \"a62e777d-2ea3-4a50-8aeb-33228f458053\" (UID: \"a62e777d-2ea3-4a50-8aeb-33228f458053\") " Sep 17 10:39:00.690574 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:39:00.690391 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l8vbw\" (UniqueName: \"kubernetes.io/projected/a62e777d-2ea3-4a50-8aeb-33228f458053-kube-api-access-l8vbw\") pod \"a62e777d-2ea3-4a50-8aeb-33228f458053\" (UID: \"a62e777d-2ea3-4a50-8aeb-33228f458053\") " Sep 17 10:39:00.690574 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:39:00.690415 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/a62e777d-2ea3-4a50-8aeb-33228f458053-online-tls\") pod \"a62e777d-2ea3-4a50-8aeb-33228f458053\" (UID: \"a62e777d-2ea3-4a50-8aeb-33228f458053\") " Sep 17 10:39:00.690574 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:39:00.690462 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/a62e777d-2ea3-4a50-8aeb-33228f458053-offline-tls\") pod \"a62e777d-2ea3-4a50-8aeb-33228f458053\" (UID: \"a62e777d-2ea3-4a50-8aeb-33228f458053\") " Sep 17 10:39:00.690574 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:39:00.690483 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/a62e777d-2ea3-4a50-8aeb-33228f458053-feast-data\") pod \"a62e777d-2ea3-4a50-8aeb-33228f458053\" (UID: \"a62e777d-2ea3-4a50-8aeb-33228f458053\") " Sep 17 10:39:00.690574 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:39:00.690512 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/a62e777d-2ea3-4a50-8aeb-33228f458053-ui-tls\") pod \"a62e777d-2ea3-4a50-8aeb-33228f458053\" (UID: \"a62e777d-2ea3-4a50-8aeb-33228f458053\") " Sep 17 10:39:00.691337 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:39:00.691301 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a62e777d-2ea3-4a50-8aeb-33228f458053-feast-data" (OuterVolumeSpecName: "feast-data") pod "a62e777d-2ea3-4a50-8aeb-33228f458053" (UID: "a62e777d-2ea3-4a50-8aeb-33228f458053"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 10:39:00.693023 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:39:00.692990 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a62e777d-2ea3-4a50-8aeb-33228f458053-kube-api-access-l8vbw" (OuterVolumeSpecName: "kube-api-access-l8vbw") pod "a62e777d-2ea3-4a50-8aeb-33228f458053" (UID: "a62e777d-2ea3-4a50-8aeb-33228f458053"). InnerVolumeSpecName "kube-api-access-l8vbw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 10:39:00.693168 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:39:00.693040 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a62e777d-2ea3-4a50-8aeb-33228f458053-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "a62e777d-2ea3-4a50-8aeb-33228f458053" (UID: "a62e777d-2ea3-4a50-8aeb-33228f458053"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 10:39:00.693168 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:39:00.693077 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a62e777d-2ea3-4a50-8aeb-33228f458053-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "a62e777d-2ea3-4a50-8aeb-33228f458053" (UID: "a62e777d-2ea3-4a50-8aeb-33228f458053"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 10:39:00.693289 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:39:00.693166 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a62e777d-2ea3-4a50-8aeb-33228f458053-online-tls" (OuterVolumeSpecName: "online-tls") pod "a62e777d-2ea3-4a50-8aeb-33228f458053" (UID: "a62e777d-2ea3-4a50-8aeb-33228f458053"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 10:39:00.693398 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:39:00.693383 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a62e777d-2ea3-4a50-8aeb-33228f458053-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "a62e777d-2ea3-4a50-8aeb-33228f458053" (UID: "a62e777d-2ea3-4a50-8aeb-33228f458053"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 10:39:00.791928 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:39:00.791843 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8vbw\" (UniqueName: \"kubernetes.io/projected/a62e777d-2ea3-4a50-8aeb-33228f458053-kube-api-access-l8vbw\") on node \"ip-10-0-137-203.ec2.internal\" DevicePath \"\"" Sep 17 10:39:00.791928 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:39:00.791873 2581 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/a62e777d-2ea3-4a50-8aeb-33228f458053-online-tls\") on node \"ip-10-0-137-203.ec2.internal\" DevicePath \"\"" Sep 17 10:39:00.791928 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:39:00.791884 2581 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/a62e777d-2ea3-4a50-8aeb-33228f458053-offline-tls\") on node \"ip-10-0-137-203.ec2.internal\" DevicePath \"\"" Sep 17 10:39:00.791928 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:39:00.791896 2581 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/a62e777d-2ea3-4a50-8aeb-33228f458053-feast-data\") on node \"ip-10-0-137-203.ec2.internal\" DevicePath \"\"" Sep 17 10:39:00.791928 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:39:00.791907 2581 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/a62e777d-2ea3-4a50-8aeb-33228f458053-ui-tls\") on node \"ip-10-0-137-203.ec2.internal\" DevicePath \"\"" Sep 17 10:39:00.791928 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:39:00.791914 2581 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/a62e777d-2ea3-4a50-8aeb-33228f458053-registry-tls\") on node \"ip-10-0-137-203.ec2.internal\" DevicePath \"\"" Sep 17 10:39:01.263024 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:39:01.262988 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xq9vr" event={"ID":"a62e777d-2ea3-4a50-8aeb-33228f458053","Type":"ContainerDied","Data":"040f412f2e5e8de17c49ee027367cc361ee96e35af825346bc0807b2ee7e8190"} Sep 17 10:39:01.263492 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:39:01.263039 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xq9vr" Sep 17 10:39:01.263492 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:39:01.263048 2581 scope.go:117] "RemoveContainer" containerID="b213c6c897dec9d48cb8f4978757d32cb999a7129871a742e58f145a1b7d8b73" Sep 17 10:39:01.272461 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:39:01.272447 2581 scope.go:117] "RemoveContainer" containerID="aee50c8312a03578ab6533870a2a0e378b18fd68ac65eb4733f3d78a02a90d5c" Sep 17 10:39:01.280254 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:39:01.280238 2581 scope.go:117] "RemoveContainer" containerID="ae586badc5f49b3d2337de13fbcadfe2b504fafaf8640b881229fea636e2f389" Sep 17 10:39:01.288308 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:39:01.288282 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-xq9vr"] Sep 17 10:39:01.288697 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:39:01.288679 2581 scope.go:117] "RemoveContainer" containerID="5246f21b58a6ce5e4d42dd31d22f0a35650debc85504473d460b6c8fb23b4076" Sep 17 10:39:01.297894 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:39:01.295006 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-xq9vr"] Sep 17 10:39:01.302173 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:39:01.302142 2581 scope.go:117] "RemoveContainer" containerID="acfddbc6ce9c7c9b36a061de9802ade95be64a0ff064907e94007a3134bd41a5" Sep 17 10:39:01.309598 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:39:01.309578 2581 scope.go:117] "RemoveContainer" containerID="0e94a99378fb0de47adfea97671b9716e9353b975cdba4ecb884bc6ae2f994da" Sep 17 10:39:01.944147 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:39:01.944107 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a62e777d-2ea3-4a50-8aeb-33228f458053" path="/var/lib/kubelet/pods/a62e777d-2ea3-4a50-8aeb-33228f458053/volumes" Sep 17 10:39:55.045418 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:39:55.045382 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-bcx4h"] Sep 17 10:39:55.045880 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:39:55.045675 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a62e777d-2ea3-4a50-8aeb-33228f458053" containerName="feast-init" Sep 17 10:39:55.045880 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:39:55.045685 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a62e777d-2ea3-4a50-8aeb-33228f458053" containerName="feast-init" Sep 17 10:39:55.045880 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:39:55.045699 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a62e777d-2ea3-4a50-8aeb-33228f458053" containerName="feast-apply" Sep 17 10:39:55.045880 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:39:55.045705 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a62e777d-2ea3-4a50-8aeb-33228f458053" containerName="feast-apply" Sep 17 10:39:55.045880 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:39:55.045715 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a62e777d-2ea3-4a50-8aeb-33228f458053" containerName="offline" Sep 17 10:39:55.045880 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:39:55.045721 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a62e777d-2ea3-4a50-8aeb-33228f458053" containerName="offline" Sep 17 10:39:55.045880 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:39:55.045729 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a62e777d-2ea3-4a50-8aeb-33228f458053" containerName="registry" Sep 17 10:39:55.045880 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:39:55.045735 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a62e777d-2ea3-4a50-8aeb-33228f458053" containerName="registry" Sep 17 10:39:55.045880 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:39:55.045740 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a62e777d-2ea3-4a50-8aeb-33228f458053" containerName="online" Sep 17 10:39:55.045880 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:39:55.045745 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a62e777d-2ea3-4a50-8aeb-33228f458053" containerName="online" Sep 17 10:39:55.045880 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:39:55.045750 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a62e777d-2ea3-4a50-8aeb-33228f458053" containerName="ui" Sep 17 10:39:55.045880 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:39:55.045755 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a62e777d-2ea3-4a50-8aeb-33228f458053" containerName="ui" Sep 17 10:39:55.045880 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:39:55.045822 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="a62e777d-2ea3-4a50-8aeb-33228f458053" containerName="offline" Sep 17 10:39:55.045880 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:39:55.045830 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="a62e777d-2ea3-4a50-8aeb-33228f458053" containerName="registry" Sep 17 10:39:55.045880 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:39:55.045835 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="a62e777d-2ea3-4a50-8aeb-33228f458053" containerName="ui" Sep 17 10:39:55.045880 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:39:55.045840 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="a62e777d-2ea3-4a50-8aeb-33228f458053" containerName="online" Sep 17 10:39:55.048999 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:39:55.048982 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-bcx4h" Sep 17 10:39:55.051525 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:39:55.051494 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"openshift-service-ca.crt\"" Sep 17 10:39:55.051723 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:39:55.051707 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-offline-tls\"" Sep 17 10:39:55.051866 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:39:55.051768 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-client-ca\"" Sep 17 10:39:55.051866 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:39:55.051772 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-online-tls\"" Sep 17 10:39:55.052004 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:39:55.051867 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"kube-root-ca.crt\"" Sep 17 10:39:55.052004 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:39:55.051912 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-ui-tls\"" Sep 17 10:39:55.052004 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:39:55.051984 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-dockercfg-zspk2\"" Sep 17 10:39:55.059468 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:39:55.059445 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-bcx4h"] Sep 17 10:39:55.133583 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:39:55.133556 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nw65\" (UniqueName: \"kubernetes.io/projected/f8c27ad2-e1fb-48f2-b61f-90ed65890247-kube-api-access-5nw65\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-bcx4h\" (UID: \"f8c27ad2-e1fb-48f2-b61f-90ed65890247\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-bcx4h" Sep 17 10:39:55.133749 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:39:55.133589 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/f8c27ad2-e1fb-48f2-b61f-90ed65890247-offline-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-bcx4h\" (UID: \"f8c27ad2-e1fb-48f2-b61f-90ed65890247\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-bcx4h" Sep 17 10:39:55.133749 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:39:55.133615 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/f8c27ad2-e1fb-48f2-b61f-90ed65890247-registry-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-bcx4h\" (UID: \"f8c27ad2-e1fb-48f2-b61f-90ed65890247\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-bcx4h" Sep 17 10:39:55.133749 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:39:55.133689 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/f8c27ad2-e1fb-48f2-b61f-90ed65890247-feast-data\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-bcx4h\" (UID: \"f8c27ad2-e1fb-48f2-b61f-90ed65890247\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-bcx4h" Sep 17 10:39:55.133749 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:39:55.133739 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/f8c27ad2-e1fb-48f2-b61f-90ed65890247-online-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-bcx4h\" (UID: \"f8c27ad2-e1fb-48f2-b61f-90ed65890247\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-bcx4h" Sep 17 10:39:55.133952 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:39:55.133801 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/f8c27ad2-e1fb-48f2-b61f-90ed65890247-ui-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-bcx4h\" (UID: \"f8c27ad2-e1fb-48f2-b61f-90ed65890247\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-bcx4h" Sep 17 10:39:55.234160 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:39:55.234127 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/f8c27ad2-e1fb-48f2-b61f-90ed65890247-online-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-bcx4h\" (UID: \"f8c27ad2-e1fb-48f2-b61f-90ed65890247\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-bcx4h" Sep 17 10:39:55.234160 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:39:55.234166 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/f8c27ad2-e1fb-48f2-b61f-90ed65890247-ui-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-bcx4h\" (UID: \"f8c27ad2-e1fb-48f2-b61f-90ed65890247\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-bcx4h" Sep 17 10:39:55.234425 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:39:55.234204 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nw65\" (UniqueName: \"kubernetes.io/projected/f8c27ad2-e1fb-48f2-b61f-90ed65890247-kube-api-access-5nw65\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-bcx4h\" (UID: \"f8c27ad2-e1fb-48f2-b61f-90ed65890247\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-bcx4h" Sep 17 10:39:55.234425 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:39:55.234224 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/f8c27ad2-e1fb-48f2-b61f-90ed65890247-offline-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-bcx4h\" (UID: \"f8c27ad2-e1fb-48f2-b61f-90ed65890247\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-bcx4h" Sep 17 10:39:55.234425 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:39:55.234246 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/f8c27ad2-e1fb-48f2-b61f-90ed65890247-registry-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-bcx4h\" (UID: \"f8c27ad2-e1fb-48f2-b61f-90ed65890247\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-bcx4h" Sep 17 10:39:55.234425 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:39:55.234263 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/f8c27ad2-e1fb-48f2-b61f-90ed65890247-feast-data\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-bcx4h\" (UID: \"f8c27ad2-e1fb-48f2-b61f-90ed65890247\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-bcx4h" Sep 17 10:39:55.234425 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:39:55.234375 2581 secret.go:189] Couldn't get secret test-ns-remote-registry/feast-simple-feast-remote-setup-offline-tls: secret "feast-simple-feast-remote-setup-offline-tls" not found Sep 17 10:39:55.234718 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:39:55.234444 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8c27ad2-e1fb-48f2-b61f-90ed65890247-offline-tls podName:f8c27ad2-e1fb-48f2-b61f-90ed65890247 nodeName:}" failed. No retries permitted until 2026-09-17 10:39:55.734423965 +0000 UTC m=+1750.282981076 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/f8c27ad2-e1fb-48f2-b61f-90ed65890247-offline-tls") pod "feast-simple-feast-remote-setup-c5d49d8c6-bcx4h" (UID: "f8c27ad2-e1fb-48f2-b61f-90ed65890247") : secret "feast-simple-feast-remote-setup-offline-tls" not found Sep 17 10:39:55.234718 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:39:55.234689 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/f8c27ad2-e1fb-48f2-b61f-90ed65890247-feast-data\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-bcx4h\" (UID: \"f8c27ad2-e1fb-48f2-b61f-90ed65890247\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-bcx4h" Sep 17 10:39:55.235157 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:39:55.235135 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/f8c27ad2-e1fb-48f2-b61f-90ed65890247-registry-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-bcx4h\" (UID: \"f8c27ad2-e1fb-48f2-b61f-90ed65890247\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-bcx4h" Sep 17 10:39:55.236723 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:39:55.236701 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/f8c27ad2-e1fb-48f2-b61f-90ed65890247-ui-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-bcx4h\" (UID: \"f8c27ad2-e1fb-48f2-b61f-90ed65890247\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-bcx4h" Sep 17 10:39:55.236838 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:39:55.236754 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/f8c27ad2-e1fb-48f2-b61f-90ed65890247-online-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-bcx4h\" (UID: \"f8c27ad2-e1fb-48f2-b61f-90ed65890247\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-bcx4h" Sep 17 10:39:55.244447 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:39:55.244428 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nw65\" (UniqueName: \"kubernetes.io/projected/f8c27ad2-e1fb-48f2-b61f-90ed65890247-kube-api-access-5nw65\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-bcx4h\" (UID: \"f8c27ad2-e1fb-48f2-b61f-90ed65890247\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-bcx4h" Sep 17 10:39:55.737750 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:39:55.737696 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/f8c27ad2-e1fb-48f2-b61f-90ed65890247-offline-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-bcx4h\" (UID: \"f8c27ad2-e1fb-48f2-b61f-90ed65890247\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-bcx4h" Sep 17 10:39:55.740135 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:39:55.740114 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/f8c27ad2-e1fb-48f2-b61f-90ed65890247-offline-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-bcx4h\" (UID: \"f8c27ad2-e1fb-48f2-b61f-90ed65890247\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-bcx4h" Sep 17 10:39:55.958202 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:39:55.958172 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-bcx4h" Sep 17 10:39:56.232821 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:39:56.232798 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-bcx4h"] Sep 17 10:39:56.235385 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:39:56.235354 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf8c27ad2_e1fb_48f2_b61f_90ed65890247.slice/crio-fc10cc7e399befe4f513cfbd9fbaeed85e8dc3ee414f7a37a48c9431aadc1696 WatchSource:0}: Error finding container fc10cc7e399befe4f513cfbd9fbaeed85e8dc3ee414f7a37a48c9431aadc1696: Status 404 returned error can't find the container with id fc10cc7e399befe4f513cfbd9fbaeed85e8dc3ee414f7a37a48c9431aadc1696 Sep 17 10:39:56.480857 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:39:56.480817 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-bcx4h" event={"ID":"f8c27ad2-e1fb-48f2-b61f-90ed65890247","Type":"ContainerStarted","Data":"f8f6f8dd7892e22c658af9292eab8dbcda0be62ec2b06a3cde865e3d3467122f"} Sep 17 10:39:56.481060 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:39:56.480875 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-bcx4h" event={"ID":"f8c27ad2-e1fb-48f2-b61f-90ed65890247","Type":"ContainerStarted","Data":"fc10cc7e399befe4f513cfbd9fbaeed85e8dc3ee414f7a37a48c9431aadc1696"} Sep 17 10:40:01.502887 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:40:01.502859 2581 generic.go:356] "Generic (PLEG): container finished" podID="f8c27ad2-e1fb-48f2-b61f-90ed65890247" containerID="f8f6f8dd7892e22c658af9292eab8dbcda0be62ec2b06a3cde865e3d3467122f" exitCode=0 Sep 17 10:40:01.503326 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:40:01.502935 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-bcx4h" event={"ID":"f8c27ad2-e1fb-48f2-b61f-90ed65890247","Type":"ContainerDied","Data":"f8f6f8dd7892e22c658af9292eab8dbcda0be62ec2b06a3cde865e3d3467122f"} Sep 17 10:40:02.509246 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:40:02.509209 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-bcx4h" event={"ID":"f8c27ad2-e1fb-48f2-b61f-90ed65890247","Type":"ContainerStarted","Data":"ccc17593f6db397ee91f0ea59682351fbdb302f5ffe9784768aab414cf2402ce"} Sep 17 10:40:24.601692 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:40:24.601655 2581 generic.go:356] "Generic (PLEG): container finished" podID="f8c27ad2-e1fb-48f2-b61f-90ed65890247" containerID="ccc17593f6db397ee91f0ea59682351fbdb302f5ffe9784768aab414cf2402ce" exitCode=0 Sep 17 10:40:24.602141 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:40:24.601728 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-bcx4h" event={"ID":"f8c27ad2-e1fb-48f2-b61f-90ed65890247","Type":"ContainerDied","Data":"ccc17593f6db397ee91f0ea59682351fbdb302f5ffe9784768aab414cf2402ce"} Sep 17 10:40:25.610599 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:40:25.610562 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-bcx4h" event={"ID":"f8c27ad2-e1fb-48f2-b61f-90ed65890247","Type":"ContainerStarted","Data":"212eeca86bf75813efa388bec95c65ccacf152fe2bbe316d8441beabe4013ad7"} Sep 17 10:40:25.611095 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:40:25.610617 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-bcx4h" event={"ID":"f8c27ad2-e1fb-48f2-b61f-90ed65890247","Type":"ContainerStarted","Data":"d248f648912b1a9387ddbf4b8a8252474a0592bb8428cad2f7d573250e2b07c9"} Sep 17 10:40:25.611095 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:40:25.610635 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-bcx4h" event={"ID":"f8c27ad2-e1fb-48f2-b61f-90ed65890247","Type":"ContainerStarted","Data":"0ed91b91b503380c2ef6024bfff93ea1518b84eb7aa62f15287e8f299cbd53aa"} Sep 17 10:40:25.634870 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:40:25.634803 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-bcx4h" podStartSLOduration=30.63476198 podStartE2EDuration="30.63476198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 10:39:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 10:40:25.634221911 +0000 UTC m=+1780.182779022" watchObservedRunningTime="2026-09-17 10:40:25.63476198 +0000 UTC m=+1780.183319092" Sep 17 10:40:25.958520 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:40:25.958426 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-bcx4h" Sep 17 10:40:25.958520 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:40:25.958498 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-bcx4h" Sep 17 10:40:25.958520 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:40:25.958518 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-bcx4h" Sep 17 10:40:25.958804 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:40:25.958535 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-bcx4h" Sep 17 10:40:25.958804 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:40:25.958551 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-bcx4h" Sep 17 10:40:25.958804 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:40:25.958566 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-bcx4h" Sep 17 10:40:25.960213 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:40:25.960172 2581 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-bcx4h" podUID="f8c27ad2-e1fb-48f2-b61f-90ed65890247" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.40:8816: connect: connection refused" Sep 17 10:40:25.960213 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:40:25.960193 2581 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-bcx4h" podUID="f8c27ad2-e1fb-48f2-b61f-90ed65890247" containerName="online" probeResult="failure" output="Get \"https://10.134.0.40:6567/health\": dial tcp 10.134.0.40:6567: connect: connection refused" Sep 17 10:40:25.960377 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:40:25.960172 2581 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-bcx4h" podUID="f8c27ad2-e1fb-48f2-b61f-90ed65890247" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.40:8443: connect: connection refused" Sep 17 10:40:28.958711 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:40:28.958667 2581 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-bcx4h" podUID="f8c27ad2-e1fb-48f2-b61f-90ed65890247" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.40:8443: connect: connection refused" Sep 17 10:40:28.959123 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:40:28.958667 2581 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-bcx4h" podUID="f8c27ad2-e1fb-48f2-b61f-90ed65890247" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.40:8816: connect: connection refused" Sep 17 10:40:28.959123 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:40:28.958675 2581 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-bcx4h" podUID="f8c27ad2-e1fb-48f2-b61f-90ed65890247" containerName="online" probeResult="failure" output="Get \"https://10.134.0.40:6567/health\": dial tcp 10.134.0.40:6567: connect: connection refused" Sep 17 10:40:31.959365 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:40:31.959321 2581 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-bcx4h" podUID="f8c27ad2-e1fb-48f2-b61f-90ed65890247" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.40:8443: connect: connection refused" Sep 17 10:40:31.960005 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:40:31.959400 2581 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-bcx4h" podUID="f8c27ad2-e1fb-48f2-b61f-90ed65890247" containerName="online" probeResult="failure" output="Get \"https://10.134.0.40:6567/health\": dial tcp 10.134.0.40:6567: connect: connection refused" Sep 17 10:40:31.960005 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:40:31.959322 2581 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-bcx4h" podUID="f8c27ad2-e1fb-48f2-b61f-90ed65890247" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.40:8816: connect: connection refused" Sep 17 10:40:34.960181 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:40:34.960096 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-bcx4h" Sep 17 10:40:34.960715 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:40:34.960688 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-bcx4h" Sep 17 10:40:34.961007 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:40:34.960989 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-bcx4h" Sep 17 10:40:35.661636 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:40:35.661606 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-bcx4h" Sep 17 10:40:35.691243 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:40:35.691216 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-bcx4h" Sep 17 10:40:36.713419 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:40:36.713389 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-bcx4h" Sep 17 10:40:36.962926 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:40:36.962884 2581 prober.go:120] "Probe failed" probeType="Liveness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-bcx4h" podUID="f8c27ad2-e1fb-48f2-b61f-90ed65890247" containerName="online" probeResult="failure" output="Get \"https://10.134.0.40:6567/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 17 10:40:37.716930 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:40:37.716890 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-bcx4h" podUID="f8c27ad2-e1fb-48f2-b61f-90ed65890247" containerName="online" probeResult="failure" output="Get \"https://10.134.0.40:6567/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 17 10:40:45.971881 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:40:45.971850 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-b9wnf_762d3e10-7b7d-4ce9-9773-90a4fe41b474/console-operator/2.log" Sep 17 10:40:45.973522 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:40:45.973500 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-b9wnf_762d3e10-7b7d-4ce9-9773-90a4fe41b474/console-operator/2.log" Sep 17 10:41:39.175231 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:41:39.175194 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-bcx4h"] Sep 17 10:41:39.175766 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:41:39.175458 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-bcx4h" podUID="f8c27ad2-e1fb-48f2-b61f-90ed65890247" containerName="online" containerID="cri-o://0ed91b91b503380c2ef6024bfff93ea1518b84eb7aa62f15287e8f299cbd53aa" gracePeriod=30 Sep 17 10:41:39.175766 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:41:39.175506 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-bcx4h" podUID="f8c27ad2-e1fb-48f2-b61f-90ed65890247" containerName="ui" containerID="cri-o://212eeca86bf75813efa388bec95c65ccacf152fe2bbe316d8441beabe4013ad7" gracePeriod=30 Sep 17 10:41:39.175766 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:41:39.175608 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-bcx4h" podUID="f8c27ad2-e1fb-48f2-b61f-90ed65890247" containerName="offline" containerID="cri-o://d248f648912b1a9387ddbf4b8a8252474a0592bb8428cad2f7d573250e2b07c9" gracePeriod=30 Sep 17 10:41:39.375037 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:41:39.375001 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-7pfpb"] Sep 17 10:41:39.375330 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:41:39.375274 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7pfpb" podUID="7358201a-ed56-4a29-9dc9-fe48ae81d86d" containerName="registry" containerID="cri-o://aac1f759bc68265a73d423fc4564b550711f117319cf81f4c2ea4bbc237d96e3" gracePeriod=30 Sep 17 10:41:39.375470 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:41:39.375307 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7pfpb" podUID="7358201a-ed56-4a29-9dc9-fe48ae81d86d" containerName="ui" containerID="cri-o://ba269b09dbd046ac1d48d177c03d810b72836f6f10af8781d13fc6341045c08f" gracePeriod=30 Sep 17 10:41:39.375575 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:41:39.375334 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7pfpb" podUID="7358201a-ed56-4a29-9dc9-fe48ae81d86d" containerName="online" containerID="cri-o://417ee1a6cceb0b799231c29b5255f04e400cf1e4d02f676ead26537ec1ccc53d" gracePeriod=30 Sep 17 10:41:39.375660 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:41:39.375377 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7pfpb" podUID="7358201a-ed56-4a29-9dc9-fe48ae81d86d" containerName="offline" containerID="cri-o://2da0d58ed8adf62e5e7ce1f57f119ee9fa77fd4ef2e51f71aa3129fb304631ca" gracePeriod=30 Sep 17 10:41:40.928233 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:41:40.928193 2581 generic.go:356] "Generic (PLEG): container finished" podID="7358201a-ed56-4a29-9dc9-fe48ae81d86d" containerID="ba269b09dbd046ac1d48d177c03d810b72836f6f10af8781d13fc6341045c08f" exitCode=0 Sep 17 10:41:40.928233 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:41:40.928219 2581 generic.go:356] "Generic (PLEG): container finished" podID="7358201a-ed56-4a29-9dc9-fe48ae81d86d" containerID="417ee1a6cceb0b799231c29b5255f04e400cf1e4d02f676ead26537ec1ccc53d" exitCode=0 Sep 17 10:41:40.928713 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:41:40.928267 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7pfpb" event={"ID":"7358201a-ed56-4a29-9dc9-fe48ae81d86d","Type":"ContainerDied","Data":"ba269b09dbd046ac1d48d177c03d810b72836f6f10af8781d13fc6341045c08f"} Sep 17 10:41:40.928713 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:41:40.928309 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7pfpb" event={"ID":"7358201a-ed56-4a29-9dc9-fe48ae81d86d","Type":"ContainerDied","Data":"417ee1a6cceb0b799231c29b5255f04e400cf1e4d02f676ead26537ec1ccc53d"} Sep 17 10:41:40.930897 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:41:40.930875 2581 generic.go:356] "Generic (PLEG): container finished" podID="f8c27ad2-e1fb-48f2-b61f-90ed65890247" containerID="212eeca86bf75813efa388bec95c65ccacf152fe2bbe316d8441beabe4013ad7" exitCode=0 Sep 17 10:41:40.930897 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:41:40.930897 2581 generic.go:356] "Generic (PLEG): container finished" podID="f8c27ad2-e1fb-48f2-b61f-90ed65890247" containerID="0ed91b91b503380c2ef6024bfff93ea1518b84eb7aa62f15287e8f299cbd53aa" exitCode=0 Sep 17 10:41:40.931025 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:41:40.930941 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-bcx4h" event={"ID":"f8c27ad2-e1fb-48f2-b61f-90ed65890247","Type":"ContainerDied","Data":"212eeca86bf75813efa388bec95c65ccacf152fe2bbe316d8441beabe4013ad7"} Sep 17 10:41:40.931025 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:41:40.930975 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-bcx4h" event={"ID":"f8c27ad2-e1fb-48f2-b61f-90ed65890247","Type":"ContainerDied","Data":"0ed91b91b503380c2ef6024bfff93ea1518b84eb7aa62f15287e8f299cbd53aa"} Sep 17 10:41:43.225314 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:41:43.225270 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7pfpb" podUID="7358201a-ed56-4a29-9dc9-fe48ae81d86d" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.39:8443: connect: connection refused" Sep 17 10:41:43.225709 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:41:43.225353 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7pfpb" podUID="7358201a-ed56-4a29-9dc9-fe48ae81d86d" containerName="online" probeResult="failure" output="Get \"https://10.134.0.39:6567/health\": dial tcp 10.134.0.39:6567: connect: connection refused" Sep 17 10:41:45.662028 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:41:45.661987 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-bcx4h" podUID="f8c27ad2-e1fb-48f2-b61f-90ed65890247" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.40:8443: connect: connection refused" Sep 17 10:41:46.714689 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:41:46.714649 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-bcx4h" podUID="f8c27ad2-e1fb-48f2-b61f-90ed65890247" containerName="online" probeResult="failure" output="Get \"https://10.134.0.40:6567/health\": dial tcp 10.134.0.40:6567: connect: connection refused" Sep 17 10:41:53.225419 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:41:53.225350 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7pfpb" podUID="7358201a-ed56-4a29-9dc9-fe48ae81d86d" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.39:8443: connect: connection refused" Sep 17 10:41:53.225869 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:41:53.225381 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7pfpb" podUID="7358201a-ed56-4a29-9dc9-fe48ae81d86d" containerName="online" probeResult="failure" output="Get \"https://10.134.0.39:6567/health\": dial tcp 10.134.0.39:6567: connect: connection refused" Sep 17 10:41:55.661908 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:41:55.661868 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-bcx4h" podUID="f8c27ad2-e1fb-48f2-b61f-90ed65890247" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.40:8443: connect: connection refused" Sep 17 10:41:56.714082 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:41:56.714037 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-bcx4h" podUID="f8c27ad2-e1fb-48f2-b61f-90ed65890247" containerName="online" probeResult="failure" output="Get \"https://10.134.0.40:6567/health\": dial tcp 10.134.0.40:6567: connect: connection refused" Sep 17 10:42:03.225581 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:03.225530 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7pfpb" podUID="7358201a-ed56-4a29-9dc9-fe48ae81d86d" containerName="online" probeResult="failure" output="Get \"https://10.134.0.39:6567/health\": dial tcp 10.134.0.39:6567: connect: connection refused" Sep 17 10:42:03.225581 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:03.225547 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7pfpb" podUID="7358201a-ed56-4a29-9dc9-fe48ae81d86d" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.39:8443: connect: connection refused" Sep 17 10:42:03.226147 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:03.225678 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7pfpb" Sep 17 10:42:03.226147 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:03.225754 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7pfpb" Sep 17 10:42:05.662001 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:05.661955 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-bcx4h" podUID="f8c27ad2-e1fb-48f2-b61f-90ed65890247" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.40:8443: connect: connection refused" Sep 17 10:42:05.662460 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:05.662114 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-bcx4h" Sep 17 10:42:06.714217 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:06.714174 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-bcx4h" podUID="f8c27ad2-e1fb-48f2-b61f-90ed65890247" containerName="online" probeResult="failure" output="Get \"https://10.134.0.40:6567/health\": dial tcp 10.134.0.40:6567: connect: connection refused" Sep 17 10:42:06.714616 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:06.714294 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-bcx4h" Sep 17 10:42:09.817907 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:09.817883 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-bcx4h" Sep 17 10:42:09.913521 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:09.913489 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/f8c27ad2-e1fb-48f2-b61f-90ed65890247-offline-tls\") pod \"f8c27ad2-e1fb-48f2-b61f-90ed65890247\" (UID: \"f8c27ad2-e1fb-48f2-b61f-90ed65890247\") " Sep 17 10:42:09.913676 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:09.913547 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/f8c27ad2-e1fb-48f2-b61f-90ed65890247-feast-data\") pod \"f8c27ad2-e1fb-48f2-b61f-90ed65890247\" (UID: \"f8c27ad2-e1fb-48f2-b61f-90ed65890247\") " Sep 17 10:42:09.913676 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:09.913572 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/f8c27ad2-e1fb-48f2-b61f-90ed65890247-ui-tls\") pod \"f8c27ad2-e1fb-48f2-b61f-90ed65890247\" (UID: \"f8c27ad2-e1fb-48f2-b61f-90ed65890247\") " Sep 17 10:42:09.913676 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:09.913599 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/f8c27ad2-e1fb-48f2-b61f-90ed65890247-registry-tls\") pod \"f8c27ad2-e1fb-48f2-b61f-90ed65890247\" (UID: \"f8c27ad2-e1fb-48f2-b61f-90ed65890247\") " Sep 17 10:42:09.913676 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:09.913628 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/f8c27ad2-e1fb-48f2-b61f-90ed65890247-online-tls\") pod \"f8c27ad2-e1fb-48f2-b61f-90ed65890247\" (UID: \"f8c27ad2-e1fb-48f2-b61f-90ed65890247\") " Sep 17 10:42:09.913676 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:09.913668 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5nw65\" (UniqueName: \"kubernetes.io/projected/f8c27ad2-e1fb-48f2-b61f-90ed65890247-kube-api-access-5nw65\") pod \"f8c27ad2-e1fb-48f2-b61f-90ed65890247\" (UID: \"f8c27ad2-e1fb-48f2-b61f-90ed65890247\") " Sep 17 10:42:09.914046 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:09.914008 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8c27ad2-e1fb-48f2-b61f-90ed65890247-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "f8c27ad2-e1fb-48f2-b61f-90ed65890247" (UID: "f8c27ad2-e1fb-48f2-b61f-90ed65890247"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 10:42:09.914297 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:09.914270 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8c27ad2-e1fb-48f2-b61f-90ed65890247-feast-data" (OuterVolumeSpecName: "feast-data") pod "f8c27ad2-e1fb-48f2-b61f-90ed65890247" (UID: "f8c27ad2-e1fb-48f2-b61f-90ed65890247"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 10:42:09.915934 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:09.915898 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8c27ad2-e1fb-48f2-b61f-90ed65890247-online-tls" (OuterVolumeSpecName: "online-tls") pod "f8c27ad2-e1fb-48f2-b61f-90ed65890247" (UID: "f8c27ad2-e1fb-48f2-b61f-90ed65890247"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 10:42:09.916038 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:09.915958 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8c27ad2-e1fb-48f2-b61f-90ed65890247-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "f8c27ad2-e1fb-48f2-b61f-90ed65890247" (UID: "f8c27ad2-e1fb-48f2-b61f-90ed65890247"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 10:42:09.916038 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:09.915986 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8c27ad2-e1fb-48f2-b61f-90ed65890247-kube-api-access-5nw65" (OuterVolumeSpecName: "kube-api-access-5nw65") pod "f8c27ad2-e1fb-48f2-b61f-90ed65890247" (UID: "f8c27ad2-e1fb-48f2-b61f-90ed65890247"). InnerVolumeSpecName "kube-api-access-5nw65". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 10:42:09.916136 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:09.916037 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8c27ad2-e1fb-48f2-b61f-90ed65890247-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "f8c27ad2-e1fb-48f2-b61f-90ed65890247" (UID: "f8c27ad2-e1fb-48f2-b61f-90ed65890247"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 10:42:10.014228 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:10.014201 2581 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/f8c27ad2-e1fb-48f2-b61f-90ed65890247-offline-tls\") on node \"ip-10-0-137-203.ec2.internal\" DevicePath \"\"" Sep 17 10:42:10.014228 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:10.014224 2581 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/f8c27ad2-e1fb-48f2-b61f-90ed65890247-feast-data\") on node \"ip-10-0-137-203.ec2.internal\" DevicePath \"\"" Sep 17 10:42:10.014228 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:10.014233 2581 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/f8c27ad2-e1fb-48f2-b61f-90ed65890247-ui-tls\") on node \"ip-10-0-137-203.ec2.internal\" DevicePath \"\"" Sep 17 10:42:10.014420 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:10.014241 2581 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/f8c27ad2-e1fb-48f2-b61f-90ed65890247-registry-tls\") on node \"ip-10-0-137-203.ec2.internal\" DevicePath \"\"" Sep 17 10:42:10.014420 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:10.014250 2581 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/f8c27ad2-e1fb-48f2-b61f-90ed65890247-online-tls\") on node \"ip-10-0-137-203.ec2.internal\" DevicePath \"\"" Sep 17 10:42:10.014420 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:10.014257 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5nw65\" (UniqueName: \"kubernetes.io/projected/f8c27ad2-e1fb-48f2-b61f-90ed65890247-kube-api-access-5nw65\") on node \"ip-10-0-137-203.ec2.internal\" DevicePath \"\"" Sep 17 10:42:10.014420 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:10.014377 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7pfpb" Sep 17 10:42:10.053421 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:10.053392 2581 generic.go:356] "Generic (PLEG): container finished" podID="f8c27ad2-e1fb-48f2-b61f-90ed65890247" containerID="d248f648912b1a9387ddbf4b8a8252474a0592bb8428cad2f7d573250e2b07c9" exitCode=137 Sep 17 10:42:10.053563 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:10.053454 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-bcx4h" Sep 17 10:42:10.053563 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:10.053453 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-bcx4h" event={"ID":"f8c27ad2-e1fb-48f2-b61f-90ed65890247","Type":"ContainerDied","Data":"d248f648912b1a9387ddbf4b8a8252474a0592bb8428cad2f7d573250e2b07c9"} Sep 17 10:42:10.053664 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:10.053569 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-bcx4h" event={"ID":"f8c27ad2-e1fb-48f2-b61f-90ed65890247","Type":"ContainerDied","Data":"fc10cc7e399befe4f513cfbd9fbaeed85e8dc3ee414f7a37a48c9431aadc1696"} Sep 17 10:42:10.053664 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:10.053599 2581 scope.go:117] "RemoveContainer" containerID="212eeca86bf75813efa388bec95c65ccacf152fe2bbe316d8441beabe4013ad7" Sep 17 10:42:10.057113 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:10.057089 2581 generic.go:356] "Generic (PLEG): container finished" podID="7358201a-ed56-4a29-9dc9-fe48ae81d86d" containerID="2da0d58ed8adf62e5e7ce1f57f119ee9fa77fd4ef2e51f71aa3129fb304631ca" exitCode=137 Sep 17 10:42:10.057235 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:10.057115 2581 generic.go:356] "Generic (PLEG): container finished" podID="7358201a-ed56-4a29-9dc9-fe48ae81d86d" containerID="aac1f759bc68265a73d423fc4564b550711f117319cf81f4c2ea4bbc237d96e3" exitCode=137 Sep 17 10:42:10.057235 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:10.057147 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7pfpb" event={"ID":"7358201a-ed56-4a29-9dc9-fe48ae81d86d","Type":"ContainerDied","Data":"2da0d58ed8adf62e5e7ce1f57f119ee9fa77fd4ef2e51f71aa3129fb304631ca"} Sep 17 10:42:10.057235 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:10.057155 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7pfpb" Sep 17 10:42:10.057235 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:10.057181 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7pfpb" event={"ID":"7358201a-ed56-4a29-9dc9-fe48ae81d86d","Type":"ContainerDied","Data":"aac1f759bc68265a73d423fc4564b550711f117319cf81f4c2ea4bbc237d96e3"} Sep 17 10:42:10.057235 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:10.057203 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7pfpb" event={"ID":"7358201a-ed56-4a29-9dc9-fe48ae81d86d","Type":"ContainerDied","Data":"582c1059c2b0e21d79599a16a5f74d366fe472dcd61fa8927e4a737b2a4c3058"} Sep 17 10:42:10.062797 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:10.062769 2581 scope.go:117] "RemoveContainer" containerID="d248f648912b1a9387ddbf4b8a8252474a0592bb8428cad2f7d573250e2b07c9" Sep 17 10:42:10.070577 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:10.070556 2581 scope.go:117] "RemoveContainer" containerID="0ed91b91b503380c2ef6024bfff93ea1518b84eb7aa62f15287e8f299cbd53aa" Sep 17 10:42:10.075890 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:10.075864 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-bcx4h"] Sep 17 10:42:10.078433 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:10.078419 2581 scope.go:117] "RemoveContainer" containerID="ccc17593f6db397ee91f0ea59682351fbdb302f5ffe9784768aab414cf2402ce" Sep 17 10:42:10.081530 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:10.081505 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-bcx4h"] Sep 17 10:42:10.086178 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:10.086161 2581 scope.go:117] "RemoveContainer" containerID="f8f6f8dd7892e22c658af9292eab8dbcda0be62ec2b06a3cde865e3d3467122f" Sep 17 10:42:10.093038 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:10.093022 2581 scope.go:117] "RemoveContainer" containerID="212eeca86bf75813efa388bec95c65ccacf152fe2bbe316d8441beabe4013ad7" Sep 17 10:42:10.093310 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:42:10.093281 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"212eeca86bf75813efa388bec95c65ccacf152fe2bbe316d8441beabe4013ad7\": container with ID starting with 212eeca86bf75813efa388bec95c65ccacf152fe2bbe316d8441beabe4013ad7 not found: ID does not exist" containerID="212eeca86bf75813efa388bec95c65ccacf152fe2bbe316d8441beabe4013ad7" Sep 17 10:42:10.093405 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:10.093315 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"212eeca86bf75813efa388bec95c65ccacf152fe2bbe316d8441beabe4013ad7"} err="failed to get container status \"212eeca86bf75813efa388bec95c65ccacf152fe2bbe316d8441beabe4013ad7\": rpc error: code = NotFound desc = could not find container \"212eeca86bf75813efa388bec95c65ccacf152fe2bbe316d8441beabe4013ad7\": container with ID starting with 212eeca86bf75813efa388bec95c65ccacf152fe2bbe316d8441beabe4013ad7 not found: ID does not exist" Sep 17 10:42:10.093405 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:10.093333 2581 scope.go:117] "RemoveContainer" containerID="d248f648912b1a9387ddbf4b8a8252474a0592bb8428cad2f7d573250e2b07c9" Sep 17 10:42:10.093601 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:42:10.093583 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d248f648912b1a9387ddbf4b8a8252474a0592bb8428cad2f7d573250e2b07c9\": container with ID starting with d248f648912b1a9387ddbf4b8a8252474a0592bb8428cad2f7d573250e2b07c9 not found: ID does not exist" containerID="d248f648912b1a9387ddbf4b8a8252474a0592bb8428cad2f7d573250e2b07c9" Sep 17 10:42:10.093642 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:10.093607 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d248f648912b1a9387ddbf4b8a8252474a0592bb8428cad2f7d573250e2b07c9"} err="failed to get container status \"d248f648912b1a9387ddbf4b8a8252474a0592bb8428cad2f7d573250e2b07c9\": rpc error: code = NotFound desc = could not find container \"d248f648912b1a9387ddbf4b8a8252474a0592bb8428cad2f7d573250e2b07c9\": container with ID starting with d248f648912b1a9387ddbf4b8a8252474a0592bb8428cad2f7d573250e2b07c9 not found: ID does not exist" Sep 17 10:42:10.093642 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:10.093623 2581 scope.go:117] "RemoveContainer" containerID="0ed91b91b503380c2ef6024bfff93ea1518b84eb7aa62f15287e8f299cbd53aa" Sep 17 10:42:10.093866 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:42:10.093848 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ed91b91b503380c2ef6024bfff93ea1518b84eb7aa62f15287e8f299cbd53aa\": container with ID starting with 0ed91b91b503380c2ef6024bfff93ea1518b84eb7aa62f15287e8f299cbd53aa not found: ID does not exist" containerID="0ed91b91b503380c2ef6024bfff93ea1518b84eb7aa62f15287e8f299cbd53aa" Sep 17 10:42:10.093944 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:10.093874 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ed91b91b503380c2ef6024bfff93ea1518b84eb7aa62f15287e8f299cbd53aa"} err="failed to get container status \"0ed91b91b503380c2ef6024bfff93ea1518b84eb7aa62f15287e8f299cbd53aa\": rpc error: code = NotFound desc = could not find container \"0ed91b91b503380c2ef6024bfff93ea1518b84eb7aa62f15287e8f299cbd53aa\": container with ID starting with 0ed91b91b503380c2ef6024bfff93ea1518b84eb7aa62f15287e8f299cbd53aa not found: ID does not exist" Sep 17 10:42:10.093944 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:10.093894 2581 scope.go:117] "RemoveContainer" containerID="ccc17593f6db397ee91f0ea59682351fbdb302f5ffe9784768aab414cf2402ce" Sep 17 10:42:10.094147 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:42:10.094131 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccc17593f6db397ee91f0ea59682351fbdb302f5ffe9784768aab414cf2402ce\": container with ID starting with ccc17593f6db397ee91f0ea59682351fbdb302f5ffe9784768aab414cf2402ce not found: ID does not exist" containerID="ccc17593f6db397ee91f0ea59682351fbdb302f5ffe9784768aab414cf2402ce" Sep 17 10:42:10.094195 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:10.094153 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccc17593f6db397ee91f0ea59682351fbdb302f5ffe9784768aab414cf2402ce"} err="failed to get container status \"ccc17593f6db397ee91f0ea59682351fbdb302f5ffe9784768aab414cf2402ce\": rpc error: code = NotFound desc = could not find container \"ccc17593f6db397ee91f0ea59682351fbdb302f5ffe9784768aab414cf2402ce\": container with ID starting with ccc17593f6db397ee91f0ea59682351fbdb302f5ffe9784768aab414cf2402ce not found: ID does not exist" Sep 17 10:42:10.094195 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:10.094172 2581 scope.go:117] "RemoveContainer" containerID="f8f6f8dd7892e22c658af9292eab8dbcda0be62ec2b06a3cde865e3d3467122f" Sep 17 10:42:10.094403 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:42:10.094388 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8f6f8dd7892e22c658af9292eab8dbcda0be62ec2b06a3cde865e3d3467122f\": container with ID starting with f8f6f8dd7892e22c658af9292eab8dbcda0be62ec2b06a3cde865e3d3467122f not found: ID does not exist" containerID="f8f6f8dd7892e22c658af9292eab8dbcda0be62ec2b06a3cde865e3d3467122f" Sep 17 10:42:10.094453 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:10.094406 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8f6f8dd7892e22c658af9292eab8dbcda0be62ec2b06a3cde865e3d3467122f"} err="failed to get container status \"f8f6f8dd7892e22c658af9292eab8dbcda0be62ec2b06a3cde865e3d3467122f\": rpc error: code = NotFound desc = could not find container \"f8f6f8dd7892e22c658af9292eab8dbcda0be62ec2b06a3cde865e3d3467122f\": container with ID starting with f8f6f8dd7892e22c658af9292eab8dbcda0be62ec2b06a3cde865e3d3467122f not found: ID does not exist" Sep 17 10:42:10.094453 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:10.094418 2581 scope.go:117] "RemoveContainer" containerID="ba269b09dbd046ac1d48d177c03d810b72836f6f10af8781d13fc6341045c08f" Sep 17 10:42:10.101452 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:10.101429 2581 scope.go:117] "RemoveContainer" containerID="2da0d58ed8adf62e5e7ce1f57f119ee9fa77fd4ef2e51f71aa3129fb304631ca" Sep 17 10:42:10.108609 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:10.108588 2581 scope.go:117] "RemoveContainer" containerID="417ee1a6cceb0b799231c29b5255f04e400cf1e4d02f676ead26537ec1ccc53d" Sep 17 10:42:10.114716 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:10.114695 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/7358201a-ed56-4a29-9dc9-fe48ae81d86d-online-tls\") pod \"7358201a-ed56-4a29-9dc9-fe48ae81d86d\" (UID: \"7358201a-ed56-4a29-9dc9-fe48ae81d86d\") " Sep 17 10:42:10.114840 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:10.114731 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tzm4t\" (UniqueName: \"kubernetes.io/projected/7358201a-ed56-4a29-9dc9-fe48ae81d86d-kube-api-access-tzm4t\") pod \"7358201a-ed56-4a29-9dc9-fe48ae81d86d\" (UID: \"7358201a-ed56-4a29-9dc9-fe48ae81d86d\") " Sep 17 10:42:10.114840 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:10.114750 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/7358201a-ed56-4a29-9dc9-fe48ae81d86d-ui-tls\") pod \"7358201a-ed56-4a29-9dc9-fe48ae81d86d\" (UID: \"7358201a-ed56-4a29-9dc9-fe48ae81d86d\") " Sep 17 10:42:10.114840 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:10.114767 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/7358201a-ed56-4a29-9dc9-fe48ae81d86d-offline-tls\") pod \"7358201a-ed56-4a29-9dc9-fe48ae81d86d\" (UID: \"7358201a-ed56-4a29-9dc9-fe48ae81d86d\") " Sep 17 10:42:10.114840 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:10.114818 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/7358201a-ed56-4a29-9dc9-fe48ae81d86d-feast-data\") pod \"7358201a-ed56-4a29-9dc9-fe48ae81d86d\" (UID: \"7358201a-ed56-4a29-9dc9-fe48ae81d86d\") " Sep 17 10:42:10.115042 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:10.114880 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/7358201a-ed56-4a29-9dc9-fe48ae81d86d-registry-tls\") pod \"7358201a-ed56-4a29-9dc9-fe48ae81d86d\" (UID: \"7358201a-ed56-4a29-9dc9-fe48ae81d86d\") " Sep 17 10:42:10.116845 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:10.115761 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7358201a-ed56-4a29-9dc9-fe48ae81d86d-feast-data" (OuterVolumeSpecName: "feast-data") pod "7358201a-ed56-4a29-9dc9-fe48ae81d86d" (UID: "7358201a-ed56-4a29-9dc9-fe48ae81d86d"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 10:42:10.118094 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:10.118069 2581 scope.go:117] "RemoveContainer" containerID="aac1f759bc68265a73d423fc4564b550711f117319cf81f4c2ea4bbc237d96e3" Sep 17 10:42:10.119902 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:10.119018 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7358201a-ed56-4a29-9dc9-fe48ae81d86d-online-tls" (OuterVolumeSpecName: "online-tls") pod "7358201a-ed56-4a29-9dc9-fe48ae81d86d" (UID: "7358201a-ed56-4a29-9dc9-fe48ae81d86d"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 10:42:10.119902 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:10.119114 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7358201a-ed56-4a29-9dc9-fe48ae81d86d-kube-api-access-tzm4t" (OuterVolumeSpecName: "kube-api-access-tzm4t") pod "7358201a-ed56-4a29-9dc9-fe48ae81d86d" (UID: "7358201a-ed56-4a29-9dc9-fe48ae81d86d"). InnerVolumeSpecName "kube-api-access-tzm4t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 10:42:10.119902 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:10.119163 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7358201a-ed56-4a29-9dc9-fe48ae81d86d-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "7358201a-ed56-4a29-9dc9-fe48ae81d86d" (UID: "7358201a-ed56-4a29-9dc9-fe48ae81d86d"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 10:42:10.122969 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:10.122940 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7358201a-ed56-4a29-9dc9-fe48ae81d86d-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "7358201a-ed56-4a29-9dc9-fe48ae81d86d" (UID: "7358201a-ed56-4a29-9dc9-fe48ae81d86d"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 10:42:10.123070 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:10.122982 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7358201a-ed56-4a29-9dc9-fe48ae81d86d-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "7358201a-ed56-4a29-9dc9-fe48ae81d86d" (UID: "7358201a-ed56-4a29-9dc9-fe48ae81d86d"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 10:42:10.130098 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:10.130074 2581 scope.go:117] "RemoveContainer" containerID="b06c7cb66012d4dbd88d6f7df026dd20e5bcfb2564c9f151f1ee113777f84821" Sep 17 10:42:10.137146 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:10.137129 2581 scope.go:117] "RemoveContainer" containerID="1d385642e51926df69d994fed7efb0b7a3bb5f4be40014386b3d2e2be36a5ffd" Sep 17 10:42:10.144486 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:10.144468 2581 scope.go:117] "RemoveContainer" containerID="ba269b09dbd046ac1d48d177c03d810b72836f6f10af8781d13fc6341045c08f" Sep 17 10:42:10.144728 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:42:10.144712 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba269b09dbd046ac1d48d177c03d810b72836f6f10af8781d13fc6341045c08f\": container with ID starting with ba269b09dbd046ac1d48d177c03d810b72836f6f10af8781d13fc6341045c08f not found: ID does not exist" containerID="ba269b09dbd046ac1d48d177c03d810b72836f6f10af8781d13fc6341045c08f" Sep 17 10:42:10.144773 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:10.144735 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba269b09dbd046ac1d48d177c03d810b72836f6f10af8781d13fc6341045c08f"} err="failed to get container status \"ba269b09dbd046ac1d48d177c03d810b72836f6f10af8781d13fc6341045c08f\": rpc error: code = NotFound desc = could not find container \"ba269b09dbd046ac1d48d177c03d810b72836f6f10af8781d13fc6341045c08f\": container with ID starting with ba269b09dbd046ac1d48d177c03d810b72836f6f10af8781d13fc6341045c08f not found: ID does not exist" Sep 17 10:42:10.144773 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:10.144754 2581 scope.go:117] "RemoveContainer" containerID="2da0d58ed8adf62e5e7ce1f57f119ee9fa77fd4ef2e51f71aa3129fb304631ca" Sep 17 10:42:10.145106 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:42:10.145081 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2da0d58ed8adf62e5e7ce1f57f119ee9fa77fd4ef2e51f71aa3129fb304631ca\": container with ID starting with 2da0d58ed8adf62e5e7ce1f57f119ee9fa77fd4ef2e51f71aa3129fb304631ca not found: ID does not exist" containerID="2da0d58ed8adf62e5e7ce1f57f119ee9fa77fd4ef2e51f71aa3129fb304631ca" Sep 17 10:42:10.145190 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:10.145113 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2da0d58ed8adf62e5e7ce1f57f119ee9fa77fd4ef2e51f71aa3129fb304631ca"} err="failed to get container status \"2da0d58ed8adf62e5e7ce1f57f119ee9fa77fd4ef2e51f71aa3129fb304631ca\": rpc error: code = NotFound desc = could not find container \"2da0d58ed8adf62e5e7ce1f57f119ee9fa77fd4ef2e51f71aa3129fb304631ca\": container with ID starting with 2da0d58ed8adf62e5e7ce1f57f119ee9fa77fd4ef2e51f71aa3129fb304631ca not found: ID does not exist" Sep 17 10:42:10.145190 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:10.145129 2581 scope.go:117] "RemoveContainer" containerID="417ee1a6cceb0b799231c29b5255f04e400cf1e4d02f676ead26537ec1ccc53d" Sep 17 10:42:10.145372 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:42:10.145353 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"417ee1a6cceb0b799231c29b5255f04e400cf1e4d02f676ead26537ec1ccc53d\": container with ID starting with 417ee1a6cceb0b799231c29b5255f04e400cf1e4d02f676ead26537ec1ccc53d not found: ID does not exist" containerID="417ee1a6cceb0b799231c29b5255f04e400cf1e4d02f676ead26537ec1ccc53d" Sep 17 10:42:10.145439 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:10.145381 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"417ee1a6cceb0b799231c29b5255f04e400cf1e4d02f676ead26537ec1ccc53d"} err="failed to get container status \"417ee1a6cceb0b799231c29b5255f04e400cf1e4d02f676ead26537ec1ccc53d\": rpc error: code = NotFound desc = could not find container \"417ee1a6cceb0b799231c29b5255f04e400cf1e4d02f676ead26537ec1ccc53d\": container with ID starting with 417ee1a6cceb0b799231c29b5255f04e400cf1e4d02f676ead26537ec1ccc53d not found: ID does not exist" Sep 17 10:42:10.145439 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:10.145403 2581 scope.go:117] "RemoveContainer" containerID="aac1f759bc68265a73d423fc4564b550711f117319cf81f4c2ea4bbc237d96e3" Sep 17 10:42:10.145670 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:42:10.145654 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aac1f759bc68265a73d423fc4564b550711f117319cf81f4c2ea4bbc237d96e3\": container with ID starting with aac1f759bc68265a73d423fc4564b550711f117319cf81f4c2ea4bbc237d96e3 not found: ID does not exist" containerID="aac1f759bc68265a73d423fc4564b550711f117319cf81f4c2ea4bbc237d96e3" Sep 17 10:42:10.145715 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:10.145675 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aac1f759bc68265a73d423fc4564b550711f117319cf81f4c2ea4bbc237d96e3"} err="failed to get container status \"aac1f759bc68265a73d423fc4564b550711f117319cf81f4c2ea4bbc237d96e3\": rpc error: code = NotFound desc = could not find container \"aac1f759bc68265a73d423fc4564b550711f117319cf81f4c2ea4bbc237d96e3\": container with ID starting with aac1f759bc68265a73d423fc4564b550711f117319cf81f4c2ea4bbc237d96e3 not found: ID does not exist" Sep 17 10:42:10.145715 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:10.145688 2581 scope.go:117] "RemoveContainer" containerID="b06c7cb66012d4dbd88d6f7df026dd20e5bcfb2564c9f151f1ee113777f84821" Sep 17 10:42:10.145926 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:42:10.145910 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b06c7cb66012d4dbd88d6f7df026dd20e5bcfb2564c9f151f1ee113777f84821\": container with ID starting with b06c7cb66012d4dbd88d6f7df026dd20e5bcfb2564c9f151f1ee113777f84821 not found: ID does not exist" containerID="b06c7cb66012d4dbd88d6f7df026dd20e5bcfb2564c9f151f1ee113777f84821" Sep 17 10:42:10.145997 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:10.145932 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b06c7cb66012d4dbd88d6f7df026dd20e5bcfb2564c9f151f1ee113777f84821"} err="failed to get container status \"b06c7cb66012d4dbd88d6f7df026dd20e5bcfb2564c9f151f1ee113777f84821\": rpc error: code = NotFound desc = could not find container \"b06c7cb66012d4dbd88d6f7df026dd20e5bcfb2564c9f151f1ee113777f84821\": container with ID starting with b06c7cb66012d4dbd88d6f7df026dd20e5bcfb2564c9f151f1ee113777f84821 not found: ID does not exist" Sep 17 10:42:10.145997 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:10.145951 2581 scope.go:117] "RemoveContainer" containerID="1d385642e51926df69d994fed7efb0b7a3bb5f4be40014386b3d2e2be36a5ffd" Sep 17 10:42:10.146175 ip-10-0-137-203 kubenswrapper[2581]: E0917 10:42:10.146156 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d385642e51926df69d994fed7efb0b7a3bb5f4be40014386b3d2e2be36a5ffd\": container with ID starting with 1d385642e51926df69d994fed7efb0b7a3bb5f4be40014386b3d2e2be36a5ffd not found: ID does not exist" containerID="1d385642e51926df69d994fed7efb0b7a3bb5f4be40014386b3d2e2be36a5ffd" Sep 17 10:42:10.146220 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:10.146178 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d385642e51926df69d994fed7efb0b7a3bb5f4be40014386b3d2e2be36a5ffd"} err="failed to get container status \"1d385642e51926df69d994fed7efb0b7a3bb5f4be40014386b3d2e2be36a5ffd\": rpc error: code = NotFound desc = could not find container \"1d385642e51926df69d994fed7efb0b7a3bb5f4be40014386b3d2e2be36a5ffd\": container with ID starting with 1d385642e51926df69d994fed7efb0b7a3bb5f4be40014386b3d2e2be36a5ffd not found: ID does not exist" Sep 17 10:42:10.146220 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:10.146189 2581 scope.go:117] "RemoveContainer" containerID="ba269b09dbd046ac1d48d177c03d810b72836f6f10af8781d13fc6341045c08f" Sep 17 10:42:10.146390 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:10.146372 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba269b09dbd046ac1d48d177c03d810b72836f6f10af8781d13fc6341045c08f"} err="failed to get container status \"ba269b09dbd046ac1d48d177c03d810b72836f6f10af8781d13fc6341045c08f\": rpc error: code = NotFound desc = could not find container \"ba269b09dbd046ac1d48d177c03d810b72836f6f10af8781d13fc6341045c08f\": container with ID starting with ba269b09dbd046ac1d48d177c03d810b72836f6f10af8781d13fc6341045c08f not found: ID does not exist" Sep 17 10:42:10.146455 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:10.146391 2581 scope.go:117] "RemoveContainer" containerID="2da0d58ed8adf62e5e7ce1f57f119ee9fa77fd4ef2e51f71aa3129fb304631ca" Sep 17 10:42:10.146659 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:10.146641 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2da0d58ed8adf62e5e7ce1f57f119ee9fa77fd4ef2e51f71aa3129fb304631ca"} err="failed to get container status \"2da0d58ed8adf62e5e7ce1f57f119ee9fa77fd4ef2e51f71aa3129fb304631ca\": rpc error: code = NotFound desc = could not find container \"2da0d58ed8adf62e5e7ce1f57f119ee9fa77fd4ef2e51f71aa3129fb304631ca\": container with ID starting with 2da0d58ed8adf62e5e7ce1f57f119ee9fa77fd4ef2e51f71aa3129fb304631ca not found: ID does not exist" Sep 17 10:42:10.146715 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:10.146661 2581 scope.go:117] "RemoveContainer" containerID="417ee1a6cceb0b799231c29b5255f04e400cf1e4d02f676ead26537ec1ccc53d" Sep 17 10:42:10.146930 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:10.146913 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"417ee1a6cceb0b799231c29b5255f04e400cf1e4d02f676ead26537ec1ccc53d"} err="failed to get container status \"417ee1a6cceb0b799231c29b5255f04e400cf1e4d02f676ead26537ec1ccc53d\": rpc error: code = NotFound desc = could not find container \"417ee1a6cceb0b799231c29b5255f04e400cf1e4d02f676ead26537ec1ccc53d\": container with ID starting with 417ee1a6cceb0b799231c29b5255f04e400cf1e4d02f676ead26537ec1ccc53d not found: ID does not exist" Sep 17 10:42:10.146930 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:10.146930 2581 scope.go:117] "RemoveContainer" containerID="aac1f759bc68265a73d423fc4564b550711f117319cf81f4c2ea4bbc237d96e3" Sep 17 10:42:10.147127 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:10.147108 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aac1f759bc68265a73d423fc4564b550711f117319cf81f4c2ea4bbc237d96e3"} err="failed to get container status \"aac1f759bc68265a73d423fc4564b550711f117319cf81f4c2ea4bbc237d96e3\": rpc error: code = NotFound desc = could not find container \"aac1f759bc68265a73d423fc4564b550711f117319cf81f4c2ea4bbc237d96e3\": container with ID starting with aac1f759bc68265a73d423fc4564b550711f117319cf81f4c2ea4bbc237d96e3 not found: ID does not exist" Sep 17 10:42:10.147167 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:10.147127 2581 scope.go:117] "RemoveContainer" containerID="b06c7cb66012d4dbd88d6f7df026dd20e5bcfb2564c9f151f1ee113777f84821" Sep 17 10:42:10.147361 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:10.147332 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b06c7cb66012d4dbd88d6f7df026dd20e5bcfb2564c9f151f1ee113777f84821"} err="failed to get container status \"b06c7cb66012d4dbd88d6f7df026dd20e5bcfb2564c9f151f1ee113777f84821\": rpc error: code = NotFound desc = could not find container \"b06c7cb66012d4dbd88d6f7df026dd20e5bcfb2564c9f151f1ee113777f84821\": container with ID starting with b06c7cb66012d4dbd88d6f7df026dd20e5bcfb2564c9f151f1ee113777f84821 not found: ID does not exist" Sep 17 10:42:10.147420 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:10.147363 2581 scope.go:117] "RemoveContainer" containerID="1d385642e51926df69d994fed7efb0b7a3bb5f4be40014386b3d2e2be36a5ffd" Sep 17 10:42:10.147610 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:10.147592 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d385642e51926df69d994fed7efb0b7a3bb5f4be40014386b3d2e2be36a5ffd"} err="failed to get container status \"1d385642e51926df69d994fed7efb0b7a3bb5f4be40014386b3d2e2be36a5ffd\": rpc error: code = NotFound desc = could not find container \"1d385642e51926df69d994fed7efb0b7a3bb5f4be40014386b3d2e2be36a5ffd\": container with ID starting with 1d385642e51926df69d994fed7efb0b7a3bb5f4be40014386b3d2e2be36a5ffd not found: ID does not exist" Sep 17 10:42:10.216251 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:10.216175 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzm4t\" (UniqueName: \"kubernetes.io/projected/7358201a-ed56-4a29-9dc9-fe48ae81d86d-kube-api-access-tzm4t\") on node \"ip-10-0-137-203.ec2.internal\" DevicePath \"\"" Sep 17 10:42:10.216251 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:10.216203 2581 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/7358201a-ed56-4a29-9dc9-fe48ae81d86d-ui-tls\") on node \"ip-10-0-137-203.ec2.internal\" DevicePath \"\"" Sep 17 10:42:10.216251 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:10.216213 2581 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/7358201a-ed56-4a29-9dc9-fe48ae81d86d-offline-tls\") on node \"ip-10-0-137-203.ec2.internal\" DevicePath \"\"" Sep 17 10:42:10.216251 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:10.216222 2581 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/7358201a-ed56-4a29-9dc9-fe48ae81d86d-feast-data\") on node \"ip-10-0-137-203.ec2.internal\" DevicePath \"\"" Sep 17 10:42:10.216251 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:10.216230 2581 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/7358201a-ed56-4a29-9dc9-fe48ae81d86d-registry-tls\") on node \"ip-10-0-137-203.ec2.internal\" DevicePath \"\"" Sep 17 10:42:10.216251 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:10.216257 2581 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/7358201a-ed56-4a29-9dc9-fe48ae81d86d-online-tls\") on node \"ip-10-0-137-203.ec2.internal\" DevicePath \"\"" Sep 17 10:42:10.380213 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:10.380182 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-7pfpb"] Sep 17 10:42:10.383800 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:10.383764 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-7pfpb"] Sep 17 10:42:11.943361 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:11.943325 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7358201a-ed56-4a29-9dc9-fe48ae81d86d" path="/var/lib/kubelet/pods/7358201a-ed56-4a29-9dc9-fe48ae81d86d/volumes" Sep 17 10:42:11.943988 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:11.943973 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8c27ad2-e1fb-48f2-b61f-90ed65890247" path="/var/lib/kubelet/pods/f8c27ad2-e1fb-48f2-b61f-90ed65890247/volumes" Sep 17 10:42:45.643401 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:45.643365 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-4l2vp/must-gather-g9qz4"] Sep 17 10:42:45.644007 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:45.643774 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8c27ad2-e1fb-48f2-b61f-90ed65890247" containerName="offline" Sep 17 10:42:45.644007 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:45.643814 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8c27ad2-e1fb-48f2-b61f-90ed65890247" containerName="offline" Sep 17 10:42:45.644007 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:45.643829 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8c27ad2-e1fb-48f2-b61f-90ed65890247" containerName="feast-apply" Sep 17 10:42:45.644007 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:45.643838 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8c27ad2-e1fb-48f2-b61f-90ed65890247" containerName="feast-apply" Sep 17 10:42:45.644007 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:45.643863 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8c27ad2-e1fb-48f2-b61f-90ed65890247" containerName="feast-init" Sep 17 10:42:45.644007 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:45.643871 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8c27ad2-e1fb-48f2-b61f-90ed65890247" containerName="feast-init" Sep 17 10:42:45.644007 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:45.643883 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8c27ad2-e1fb-48f2-b61f-90ed65890247" containerName="online" Sep 17 10:42:45.644007 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:45.643891 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8c27ad2-e1fb-48f2-b61f-90ed65890247" containerName="online" Sep 17 10:42:45.644007 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:45.643901 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7358201a-ed56-4a29-9dc9-fe48ae81d86d" containerName="registry" Sep 17 10:42:45.644007 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:45.643909 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7358201a-ed56-4a29-9dc9-fe48ae81d86d" containerName="registry" Sep 17 10:42:45.644007 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:45.643925 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7358201a-ed56-4a29-9dc9-fe48ae81d86d" containerName="offline" Sep 17 10:42:45.644007 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:45.643933 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7358201a-ed56-4a29-9dc9-fe48ae81d86d" containerName="offline" Sep 17 10:42:45.644007 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:45.643943 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8c27ad2-e1fb-48f2-b61f-90ed65890247" containerName="ui" Sep 17 10:42:45.644007 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:45.643952 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8c27ad2-e1fb-48f2-b61f-90ed65890247" containerName="ui" Sep 17 10:42:45.644007 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:45.643964 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7358201a-ed56-4a29-9dc9-fe48ae81d86d" containerName="feast-init" Sep 17 10:42:45.644007 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:45.643973 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7358201a-ed56-4a29-9dc9-fe48ae81d86d" containerName="feast-init" Sep 17 10:42:45.644007 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:45.643984 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7358201a-ed56-4a29-9dc9-fe48ae81d86d" containerName="online" Sep 17 10:42:45.644007 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:45.643992 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7358201a-ed56-4a29-9dc9-fe48ae81d86d" containerName="online" Sep 17 10:42:45.644007 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:45.644002 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7358201a-ed56-4a29-9dc9-fe48ae81d86d" containerName="ui" Sep 17 10:42:45.644007 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:45.644010 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7358201a-ed56-4a29-9dc9-fe48ae81d86d" containerName="ui" Sep 17 10:42:45.645043 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:45.644024 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7358201a-ed56-4a29-9dc9-fe48ae81d86d" containerName="feast-apply" Sep 17 10:42:45.645043 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:45.644033 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7358201a-ed56-4a29-9dc9-fe48ae81d86d" containerName="feast-apply" Sep 17 10:42:45.645043 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:45.644122 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="7358201a-ed56-4a29-9dc9-fe48ae81d86d" containerName="online" Sep 17 10:42:45.645043 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:45.644136 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8c27ad2-e1fb-48f2-b61f-90ed65890247" containerName="online" Sep 17 10:42:45.645043 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:45.644147 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8c27ad2-e1fb-48f2-b61f-90ed65890247" containerName="ui" Sep 17 10:42:45.645043 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:45.644156 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="7358201a-ed56-4a29-9dc9-fe48ae81d86d" containerName="offline" Sep 17 10:42:45.645043 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:45.644168 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="7358201a-ed56-4a29-9dc9-fe48ae81d86d" containerName="registry" Sep 17 10:42:45.645043 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:45.644176 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="7358201a-ed56-4a29-9dc9-fe48ae81d86d" containerName="ui" Sep 17 10:42:45.645043 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:45.644185 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8c27ad2-e1fb-48f2-b61f-90ed65890247" containerName="offline" Sep 17 10:42:45.647542 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:45.647521 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4l2vp/must-gather-g9qz4" Sep 17 10:42:45.650576 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:45.650551 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4l2vp\"/\"kube-root-ca.crt\"" Sep 17 10:42:45.650678 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:45.650551 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4l2vp\"/\"openshift-service-ca.crt\"" Sep 17 10:42:45.651733 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:45.651702 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-4l2vp\"/\"default-dockercfg-qshd2\"" Sep 17 10:42:45.663663 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:45.663639 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4l2vp/must-gather-g9qz4"] Sep 17 10:42:45.788443 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:45.788414 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/99a381e0-eabc-4983-8a5a-5b1b23765bb7-must-gather-output\") pod \"must-gather-g9qz4\" (UID: \"99a381e0-eabc-4983-8a5a-5b1b23765bb7\") " pod="openshift-must-gather-4l2vp/must-gather-g9qz4" Sep 17 10:42:45.788584 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:45.788456 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcwfm\" (UniqueName: \"kubernetes.io/projected/99a381e0-eabc-4983-8a5a-5b1b23765bb7-kube-api-access-mcwfm\") pod \"must-gather-g9qz4\" (UID: \"99a381e0-eabc-4983-8a5a-5b1b23765bb7\") " pod="openshift-must-gather-4l2vp/must-gather-g9qz4" Sep 17 10:42:45.889042 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:45.889008 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/99a381e0-eabc-4983-8a5a-5b1b23765bb7-must-gather-output\") pod \"must-gather-g9qz4\" (UID: \"99a381e0-eabc-4983-8a5a-5b1b23765bb7\") " pod="openshift-must-gather-4l2vp/must-gather-g9qz4" Sep 17 10:42:45.889214 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:45.889057 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcwfm\" (UniqueName: \"kubernetes.io/projected/99a381e0-eabc-4983-8a5a-5b1b23765bb7-kube-api-access-mcwfm\") pod \"must-gather-g9qz4\" (UID: \"99a381e0-eabc-4983-8a5a-5b1b23765bb7\") " pod="openshift-must-gather-4l2vp/must-gather-g9qz4" Sep 17 10:42:45.889333 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:45.889311 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/99a381e0-eabc-4983-8a5a-5b1b23765bb7-must-gather-output\") pod \"must-gather-g9qz4\" (UID: \"99a381e0-eabc-4983-8a5a-5b1b23765bb7\") " pod="openshift-must-gather-4l2vp/must-gather-g9qz4" Sep 17 10:42:45.898174 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:45.898124 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4l2vp\"/\"kube-root-ca.crt\"" Sep 17 10:42:45.909019 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:45.909004 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4l2vp\"/\"openshift-service-ca.crt\"" Sep 17 10:42:45.919307 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:45.919289 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcwfm\" (UniqueName: \"kubernetes.io/projected/99a381e0-eabc-4983-8a5a-5b1b23765bb7-kube-api-access-mcwfm\") pod \"must-gather-g9qz4\" (UID: \"99a381e0-eabc-4983-8a5a-5b1b23765bb7\") " pod="openshift-must-gather-4l2vp/must-gather-g9qz4" Sep 17 10:42:45.958728 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:45.958708 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-4l2vp\"/\"default-dockercfg-qshd2\"" Sep 17 10:42:45.966615 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:45.966598 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4l2vp/must-gather-g9qz4" Sep 17 10:42:46.234050 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:46.234024 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4l2vp/must-gather-g9qz4"] Sep 17 10:42:46.235825 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:42:46.235798 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod99a381e0_eabc_4983_8a5a_5b1b23765bb7.slice/crio-dd3fb319082996b611d29b1d99301b624291cc4a1b3c1546188fbd9af453bc68 WatchSource:0}: Error finding container dd3fb319082996b611d29b1d99301b624291cc4a1b3c1546188fbd9af453bc68: Status 404 returned error can't find the container with id dd3fb319082996b611d29b1d99301b624291cc4a1b3c1546188fbd9af453bc68 Sep 17 10:42:46.237934 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:46.237919 2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 17 10:42:47.202619 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:47.202586 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4l2vp/must-gather-g9qz4" event={"ID":"99a381e0-eabc-4983-8a5a-5b1b23765bb7","Type":"ContainerStarted","Data":"dd3fb319082996b611d29b1d99301b624291cc4a1b3c1546188fbd9af453bc68"} Sep 17 10:42:48.209535 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:48.209487 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4l2vp/must-gather-g9qz4" event={"ID":"99a381e0-eabc-4983-8a5a-5b1b23765bb7","Type":"ContainerStarted","Data":"19f9f4dce155f5c764698630bd0f7b52d8224b72cb62869714de293c9087779b"} Sep 17 10:42:48.209984 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:48.209547 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4l2vp/must-gather-g9qz4" event={"ID":"99a381e0-eabc-4983-8a5a-5b1b23765bb7","Type":"ContainerStarted","Data":"1579414890bc6f558145791197ff614d44e4d1fda605a20add321e8463b2486a"} Sep 17 10:42:48.225858 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:48.225766 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-4l2vp/must-gather-g9qz4" podStartSLOduration=2.33462836 podStartE2EDuration="3.225743539s" totalImagesPullingTime="891.115179ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 10:42:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 10:42:48.224131994 +0000 UTC m=+1922.772689106" watchObservedRunningTime="2026-09-17 10:42:48.225743539 +0000 UTC m=+1922.774300706" Sep 17 10:42:48.511554 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:48.511524 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-v9xpw_c304d2b0-b1ce-4a29-86b1-341d5db70d4f/global-pull-secret-syncer/0.log" Sep 17 10:42:48.583627 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:48.583585 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-jkx62_de933a7f-7eae-4e7b-870f-aa9c56565481/konnectivity-agent/0.log" Sep 17 10:42:48.664534 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:48.664505 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-137-203.ec2.internal_fcac5a3f4a7d9d4b6710d0af7b8c7711/haproxy/0.log" Sep 17 10:42:51.685812 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:51.685757 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5d85778b9d-fkbh2_43bccb30-f06f-4511-8927-b8face2f6dda/monitoring-plugin/0.log" Sep 17 10:42:51.741678 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:51.741644 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2z4bl_a384f6e7-e487-4749-b0d0-6b679aa60fae/node-exporter/0.log" Sep 17 10:42:51.773887 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:51.773851 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2z4bl_a384f6e7-e487-4749-b0d0-6b679aa60fae/kube-rbac-proxy/0.log" Sep 17 10:42:51.803113 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:51.803007 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2z4bl_a384f6e7-e487-4749-b0d0-6b679aa60fae/init-textfile/0.log" Sep 17 10:42:52.342620 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:52.342587 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-vzrdc_4a7d0e53-a082-4e4a-bb1d-3da2d679e92c/prometheus-operator/0.log" Sep 17 10:42:52.363046 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:52.362989 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-vzrdc_4a7d0e53-a082-4e4a-bb1d-3da2d679e92c/kube-rbac-proxy/0.log" Sep 17 10:42:52.397067 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:52.397031 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5864cd99d7-hfwvn_f3233a3b-aa93-403e-99d2-78a522e8a006/prometheus-operator-admission-webhook/0.log" Sep 17 10:42:52.513564 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:52.513474 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-56bd659dfc-dqdhc_8453a7d4-d2f3-4b4b-be12-0c6e6faa675d/thanos-query/0.log" Sep 17 10:42:52.537032 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:52.536996 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-56bd659dfc-dqdhc_8453a7d4-d2f3-4b4b-be12-0c6e6faa675d/kube-rbac-proxy-web/0.log" Sep 17 10:42:52.558232 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:52.558178 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-56bd659dfc-dqdhc_8453a7d4-d2f3-4b4b-be12-0c6e6faa675d/kube-rbac-proxy/0.log" Sep 17 10:42:52.585602 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:52.585574 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-56bd659dfc-dqdhc_8453a7d4-d2f3-4b4b-be12-0c6e6faa675d/prom-label-proxy/0.log" Sep 17 10:42:52.608033 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:52.608004 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-56bd659dfc-dqdhc_8453a7d4-d2f3-4b4b-be12-0c6e6faa675d/kube-rbac-proxy-rules/0.log" Sep 17 10:42:52.633064 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:52.633028 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-56bd659dfc-dqdhc_8453a7d4-d2f3-4b4b-be12-0c6e6faa675d/kube-rbac-proxy-metrics/0.log" Sep 17 10:42:54.535803 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:54.535752 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-b9wnf_762d3e10-7b7d-4ce9-9773-90a4fe41b474/console-operator/2.log" Sep 17 10:42:54.541552 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:54.541521 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-b9wnf_762d3e10-7b7d-4ce9-9773-90a4fe41b474/console-operator/3.log" Sep 17 10:42:54.970440 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:54.970370 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-8fcc87966-pblxr_d85944aa-7e5a-4243-9b09-8a5f451b3bd1/console/0.log" Sep 17 10:42:55.007437 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:55.007401 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-76f99584f4-klnsv_97aaddeb-c0ee-423c-a916-f6ad15782799/download-server/0.log" Sep 17 10:42:55.484967 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:55.484928 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-86695b7cc7-d6hwv_1ddcc945-9657-4614-bc59-f0cfcf613c1e/volume-data-source-validator/0.log" Sep 17 10:42:55.878260 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:55.878172 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-4l2vp/perf-node-gather-daemonset-l7kfc"] Sep 17 10:42:55.884373 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:55.884345 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4l2vp/perf-node-gather-daemonset-l7kfc" Sep 17 10:42:55.890282 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:55.890260 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4l2vp/perf-node-gather-daemonset-l7kfc"] Sep 17 10:42:55.984205 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:55.984168 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c961728f-1c9b-4de5-b659-c3d8eeeab219-podres\") pod \"perf-node-gather-daemonset-l7kfc\" (UID: \"c961728f-1c9b-4de5-b659-c3d8eeeab219\") " pod="openshift-must-gather-4l2vp/perf-node-gather-daemonset-l7kfc" Sep 17 10:42:55.984527 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:55.984498 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sz699\" (UniqueName: \"kubernetes.io/projected/c961728f-1c9b-4de5-b659-c3d8eeeab219-kube-api-access-sz699\") pod \"perf-node-gather-daemonset-l7kfc\" (UID: \"c961728f-1c9b-4de5-b659-c3d8eeeab219\") " pod="openshift-must-gather-4l2vp/perf-node-gather-daemonset-l7kfc" Sep 17 10:42:55.984636 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:55.984549 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c961728f-1c9b-4de5-b659-c3d8eeeab219-sys\") pod \"perf-node-gather-daemonset-l7kfc\" (UID: \"c961728f-1c9b-4de5-b659-c3d8eeeab219\") " pod="openshift-must-gather-4l2vp/perf-node-gather-daemonset-l7kfc" Sep 17 10:42:55.984636 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:55.984596 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c961728f-1c9b-4de5-b659-c3d8eeeab219-lib-modules\") pod \"perf-node-gather-daemonset-l7kfc\" (UID: \"c961728f-1c9b-4de5-b659-c3d8eeeab219\") " pod="openshift-must-gather-4l2vp/perf-node-gather-daemonset-l7kfc" Sep 17 10:42:55.984636 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:55.984633 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c961728f-1c9b-4de5-b659-c3d8eeeab219-proc\") pod \"perf-node-gather-daemonset-l7kfc\" (UID: \"c961728f-1c9b-4de5-b659-c3d8eeeab219\") " pod="openshift-must-gather-4l2vp/perf-node-gather-daemonset-l7kfc" Sep 17 10:42:56.085587 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:56.085556 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sz699\" (UniqueName: \"kubernetes.io/projected/c961728f-1c9b-4de5-b659-c3d8eeeab219-kube-api-access-sz699\") pod \"perf-node-gather-daemonset-l7kfc\" (UID: \"c961728f-1c9b-4de5-b659-c3d8eeeab219\") " pod="openshift-must-gather-4l2vp/perf-node-gather-daemonset-l7kfc" Sep 17 10:42:56.085587 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:56.085594 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c961728f-1c9b-4de5-b659-c3d8eeeab219-sys\") pod \"perf-node-gather-daemonset-l7kfc\" (UID: \"c961728f-1c9b-4de5-b659-c3d8eeeab219\") " pod="openshift-must-gather-4l2vp/perf-node-gather-daemonset-l7kfc" Sep 17 10:42:56.085894 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:56.085633 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c961728f-1c9b-4de5-b659-c3d8eeeab219-lib-modules\") pod \"perf-node-gather-daemonset-l7kfc\" (UID: \"c961728f-1c9b-4de5-b659-c3d8eeeab219\") " pod="openshift-must-gather-4l2vp/perf-node-gather-daemonset-l7kfc" Sep 17 10:42:56.085894 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:56.085666 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c961728f-1c9b-4de5-b659-c3d8eeeab219-proc\") pod \"perf-node-gather-daemonset-l7kfc\" (UID: \"c961728f-1c9b-4de5-b659-c3d8eeeab219\") " pod="openshift-must-gather-4l2vp/perf-node-gather-daemonset-l7kfc" Sep 17 10:42:56.085894 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:56.085715 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c961728f-1c9b-4de5-b659-c3d8eeeab219-podres\") pod \"perf-node-gather-daemonset-l7kfc\" (UID: \"c961728f-1c9b-4de5-b659-c3d8eeeab219\") " pod="openshift-must-gather-4l2vp/perf-node-gather-daemonset-l7kfc" Sep 17 10:42:56.085894 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:56.085807 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c961728f-1c9b-4de5-b659-c3d8eeeab219-sys\") pod \"perf-node-gather-daemonset-l7kfc\" (UID: \"c961728f-1c9b-4de5-b659-c3d8eeeab219\") " pod="openshift-must-gather-4l2vp/perf-node-gather-daemonset-l7kfc" Sep 17 10:42:56.085894 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:56.085853 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c961728f-1c9b-4de5-b659-c3d8eeeab219-lib-modules\") pod \"perf-node-gather-daemonset-l7kfc\" (UID: \"c961728f-1c9b-4de5-b659-c3d8eeeab219\") " pod="openshift-must-gather-4l2vp/perf-node-gather-daemonset-l7kfc" Sep 17 10:42:56.085894 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:56.085851 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c961728f-1c9b-4de5-b659-c3d8eeeab219-proc\") pod \"perf-node-gather-daemonset-l7kfc\" (UID: \"c961728f-1c9b-4de5-b659-c3d8eeeab219\") " pod="openshift-must-gather-4l2vp/perf-node-gather-daemonset-l7kfc" Sep 17 10:42:56.085894 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:56.085872 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c961728f-1c9b-4de5-b659-c3d8eeeab219-podres\") pod \"perf-node-gather-daemonset-l7kfc\" (UID: \"c961728f-1c9b-4de5-b659-c3d8eeeab219\") " pod="openshift-must-gather-4l2vp/perf-node-gather-daemonset-l7kfc" Sep 17 10:42:56.094242 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:56.094217 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sz699\" (UniqueName: \"kubernetes.io/projected/c961728f-1c9b-4de5-b659-c3d8eeeab219-kube-api-access-sz699\") pod \"perf-node-gather-daemonset-l7kfc\" (UID: \"c961728f-1c9b-4de5-b659-c3d8eeeab219\") " pod="openshift-must-gather-4l2vp/perf-node-gather-daemonset-l7kfc" Sep 17 10:42:56.199280 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:56.199196 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4l2vp/perf-node-gather-daemonset-l7kfc" Sep 17 10:42:56.359764 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:56.359734 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-ds6sz_026a85ed-57e7-474d-9f05-c8824906de72/dns/0.log" Sep 17 10:42:56.379770 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:56.379742 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-ds6sz_026a85ed-57e7-474d-9f05-c8824906de72/kube-rbac-proxy/0.log" Sep 17 10:42:56.467199 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:56.467117 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-7f64n_6f7323dd-d769-4f58-92d7-e8a36c2f2854/dns-node-resolver/0.log" Sep 17 10:42:56.482349 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:56.482319 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4l2vp/perf-node-gather-daemonset-l7kfc"] Sep 17 10:42:56.484884 ip-10-0-137-203 kubenswrapper[2581]: W0917 10:42:56.484853 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podc961728f_1c9b_4de5_b659_c3d8eeeab219.slice/crio-581e9564312fd8c2108848c9b3612ec26d854c600cbb74cf77ff5bf9f8bbe398 WatchSource:0}: Error finding container 581e9564312fd8c2108848c9b3612ec26d854c600cbb74cf77ff5bf9f8bbe398: Status 404 returned error can't find the container with id 581e9564312fd8c2108848c9b3612ec26d854c600cbb74cf77ff5bf9f8bbe398 Sep 17 10:42:56.931938 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:56.931902 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-r2dcq_2c409c2a-64a4-4164-8a14-c2a87597dd93/node-ca/0.log" Sep 17 10:42:57.258471 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:57.258405 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4l2vp/perf-node-gather-daemonset-l7kfc" event={"ID":"c961728f-1c9b-4de5-b659-c3d8eeeab219","Type":"ContainerStarted","Data":"80cd5054ed97f1d114eab732d8184390b92b7413864b1c62162a7fd6a1702a27"} Sep 17 10:42:57.258668 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:57.258493 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-4l2vp/perf-node-gather-daemonset-l7kfc" Sep 17 10:42:57.258668 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:57.258521 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4l2vp/perf-node-gather-daemonset-l7kfc" event={"ID":"c961728f-1c9b-4de5-b659-c3d8eeeab219","Type":"ContainerStarted","Data":"581e9564312fd8c2108848c9b3612ec26d854c600cbb74cf77ff5bf9f8bbe398"} Sep 17 10:42:57.273565 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:57.273511 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-4l2vp/perf-node-gather-daemonset-l7kfc" podStartSLOduration=2.273491871 podStartE2EDuration="2.273491871s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 10:42:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 10:42:57.273161549 +0000 UTC m=+1931.821718661" watchObservedRunningTime="2026-09-17 10:42:57.273491871 +0000 UTC m=+1931.822048983" Sep 17 10:42:57.589430 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:57.589358 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-85865fbbd8-bkwfh_e18215ee-6cb8-49d5-bb04-aaa79b65f012/router/0.log" Sep 17 10:42:57.920752 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:57.920684 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-lpfcm_2769e90f-7bdb-4fce-bfa6-ebf02a0963da/serve-healthcheck-canary/0.log" Sep 17 10:42:58.278155 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:58.278120 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-fdc72_a13ce06b-1f1b-4614-a6f1-7e95dbd6f1d4/insights-operator/0.log" Sep 17 10:42:58.278763 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:58.278747 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-fdc72_a13ce06b-1f1b-4614-a6f1-7e95dbd6f1d4/insights-operator/1.log" Sep 17 10:42:58.300767 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:58.300745 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2bn9v_c65e3cab-f0ee-47b2-95f3-e059983bf8f7/kube-rbac-proxy/0.log" Sep 17 10:42:58.320093 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:58.320070 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2bn9v_c65e3cab-f0ee-47b2-95f3-e059983bf8f7/exporter/0.log" Sep 17 10:42:58.339727 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:42:58.339704 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2bn9v_c65e3cab-f0ee-47b2-95f3-e059983bf8f7/extractor/0.log" Sep 17 10:43:02.739224 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:43:02.739192 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-mjspg_da9abd04-2db6-44cc-b6bb-c5d2999ec8c4/kube-storage-version-migrator-operator/1.log" Sep 17 10:43:02.740073 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:43:02.740052 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-mjspg_da9abd04-2db6-44cc-b6bb-c5d2999ec8c4/kube-storage-version-migrator-operator/0.log" Sep 17 10:43:03.273255 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:43:03.273224 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-4l2vp/perf-node-gather-daemonset-l7kfc" Sep 17 10:43:03.644452 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:43:03.644380 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4w7ff_a80abea0-a53c-4d9e-8934-7a7f073d71e3/kube-multus-additional-cni-plugins/0.log" Sep 17 10:43:03.663995 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:43:03.663970 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4w7ff_a80abea0-a53c-4d9e-8934-7a7f073d71e3/egress-router-binary-copy/0.log" Sep 17 10:43:03.682765 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:43:03.682737 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4w7ff_a80abea0-a53c-4d9e-8934-7a7f073d71e3/cni-plugins/0.log" Sep 17 10:43:03.701983 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:43:03.701954 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4w7ff_a80abea0-a53c-4d9e-8934-7a7f073d71e3/bond-cni-plugin/0.log" Sep 17 10:43:03.721736 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:43:03.721689 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4w7ff_a80abea0-a53c-4d9e-8934-7a7f073d71e3/routeoverride-cni/0.log" Sep 17 10:43:03.740980 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:43:03.740957 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4w7ff_a80abea0-a53c-4d9e-8934-7a7f073d71e3/whereabouts-cni-bincopy/0.log" Sep 17 10:43:03.760282 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:43:03.760250 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4w7ff_a80abea0-a53c-4d9e-8934-7a7f073d71e3/whereabouts-cni/0.log" Sep 17 10:43:04.014143 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:43:04.014095 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-nxbjh_ae147838-ff96-4652-bf25-ca99bc25be02/kube-multus/0.log" Sep 17 10:43:04.080645 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:43:04.080607 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-wfccr_c84a7003-1537-4470-8094-ea10a89a86a2/network-metrics-daemon/0.log" Sep 17 10:43:04.101139 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:43:04.101107 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-wfccr_c84a7003-1537-4470-8094-ea10a89a86a2/kube-rbac-proxy/0.log" Sep 17 10:43:05.160798 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:43:05.160761 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9zbjf_896c7cf8-3f20-4809-8f6d-a6852a068ad2/ovn-controller/0.log" Sep 17 10:43:05.179036 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:43:05.179009 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9zbjf_896c7cf8-3f20-4809-8f6d-a6852a068ad2/ovn-acl-logging/0.log" Sep 17 10:43:05.206241 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:43:05.206161 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9zbjf_896c7cf8-3f20-4809-8f6d-a6852a068ad2/kube-rbac-proxy-node/0.log" Sep 17 10:43:05.227569 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:43:05.227541 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9zbjf_896c7cf8-3f20-4809-8f6d-a6852a068ad2/kube-rbac-proxy-ovn-metrics/0.log" Sep 17 10:43:05.246855 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:43:05.246821 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9zbjf_896c7cf8-3f20-4809-8f6d-a6852a068ad2/northd/0.log" Sep 17 10:43:05.267180 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:43:05.267153 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9zbjf_896c7cf8-3f20-4809-8f6d-a6852a068ad2/nbdb/0.log" Sep 17 10:43:05.288311 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:43:05.288284 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9zbjf_896c7cf8-3f20-4809-8f6d-a6852a068ad2/sbdb/0.log" Sep 17 10:43:05.332072 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:43:05.332047 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9zbjf_896c7cf8-3f20-4809-8f6d-a6852a068ad2/ovnkube-controller/0.log" Sep 17 10:43:06.523647 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:43:06.523619 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-fppwm_f6509b04-a69d-49d5-a08c-aabdd8bf6c86/network-check-target-container/0.log" Sep 17 10:43:07.383524 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:43:07.383448 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-g8lzm_b506216d-c159-420e-9111-56acab7568df/iptables-alerter/0.log" Sep 17 10:43:07.986111 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:43:07.986066 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-kg2hz_e4b73c92-0ac6-4d8f-90d8-249cfbcadf4b/tuned/0.log" Sep 17 10:43:09.638415 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:43:09.638385 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-86499c89c4-sv877_c802c004-f487-44df-b73a-fb50df751388/cluster-samples-operator/0.log" Sep 17 10:43:09.653650 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:43:09.653626 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-86499c89c4-sv877_c802c004-f487-44df-b73a-fb50df751388/cluster-samples-operator-watch/0.log" Sep 17 10:43:11.261071 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:43:11.261040 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-kcztl_606f20f7-c795-451a-b146-03f3a4f92684/csi-driver/0.log" Sep 17 10:43:11.280767 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:43:11.280728 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-kcztl_606f20f7-c795-451a-b146-03f3a4f92684/csi-node-driver-registrar/0.log" Sep 17 10:43:11.308350 ip-10-0-137-203 kubenswrapper[2581]: I0917 10:43:11.308321 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-kcztl_606f20f7-c795-451a-b146-03f3a4f92684/csi-liveness-probe/0.log"