Sep 11 09:10:47.837957 ip-10-0-135-119 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Sep 11 09:10:47.837970 ip-10-0-135-119 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Sep 11 09:10:47.837980 ip-10-0-135-119 systemd[1]: kubelet.service: Failed with result 'resources'. Sep 11 09:10:47.838197 ip-10-0-135-119 systemd[1]: Failed to start Kubernetes Kubelet. Sep 11 09:10:59.248056 ip-10-0-135-119 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Sep 11 09:10:59.248077 ip-10-0-135-119 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot ce8b753d494c449fa240fee960841644 -- Sep 11 09:13:02.466197 ip-10-0-135-119 systemd[1]: Starting Kubernetes Kubelet... Sep 11 09:13:02.800141 ip-10-0-135-119 kubenswrapper[2575]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 11 09:13:02.800141 ip-10-0-135-119 kubenswrapper[2575]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 11 09:13:02.800141 ip-10-0-135-119 kubenswrapper[2575]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 11 09:13:02.800141 ip-10-0-135-119 kubenswrapper[2575]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 11 09:13:02.800141 ip-10-0-135-119 kubenswrapper[2575]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 11 09:13:02.801056 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.800962 2575 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 11 09:13:02.805005 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.804983 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 09:13:02.805005 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.805001 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 09:13:02.805005 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.805006 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 09:13:02.805005 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.805010 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 09:13:02.805250 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.805014 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 09:13:02.805250 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.805019 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 09:13:02.805250 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.805022 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 09:13:02.805250 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.805026 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 09:13:02.805250 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.805030 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 09:13:02.805250 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.805034 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 09:13:02.805250 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.805037 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 09:13:02.805250 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.805043 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 09:13:02.805250 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.805046 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 09:13:02.805250 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.805050 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 09:13:02.805250 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.805054 2575 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 09:13:02.805250 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.805058 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 09:13:02.805250 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.805062 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 09:13:02.805250 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.805067 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 09:13:02.805250 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.805071 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 09:13:02.805250 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.805075 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 09:13:02.805250 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.805081 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 09:13:02.805250 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.805086 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 09:13:02.805250 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.805090 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 09:13:02.805250 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.805094 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 09:13:02.806210 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.805098 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 09:13:02.806210 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.805102 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 09:13:02.806210 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.805106 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 09:13:02.806210 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.805110 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 09:13:02.806210 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.805114 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 09:13:02.806210 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.805118 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 09:13:02.806210 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.805123 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 09:13:02.806210 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.805126 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 09:13:02.806210 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.805130 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 09:13:02.806210 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.805134 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 09:13:02.806210 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.805138 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 09:13:02.806210 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.805143 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 09:13:02.806210 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.805148 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 09:13:02.806210 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.805153 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 09:13:02.806210 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.805157 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 09:13:02.806210 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.805161 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 09:13:02.806210 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.805166 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 09:13:02.806210 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.805170 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 09:13:02.806210 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.805174 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 09:13:02.807100 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.805178 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 09:13:02.807100 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.805182 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 09:13:02.807100 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.805187 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 09:13:02.807100 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.805191 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 09:13:02.807100 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.805195 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 09:13:02.807100 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.805200 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 09:13:02.807100 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.805204 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 09:13:02.807100 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.805208 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 09:13:02.807100 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.805213 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 09:13:02.807100 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.805219 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 09:13:02.807100 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.805224 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 09:13:02.807100 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.805229 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 09:13:02.807100 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.805233 2575 feature_gate.go:328] unrecognized feature gate: Example Sep 11 09:13:02.807100 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.805238 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 09:13:02.807100 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.805242 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 09:13:02.807100 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.805249 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 09:13:02.807100 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.805255 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 09:13:02.807100 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.805260 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 09:13:02.807100 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.805265 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 09:13:02.807100 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.805270 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 09:13:02.807895 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.805274 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 09:13:02.807895 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.805279 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 09:13:02.807895 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.805283 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 09:13:02.807895 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.805287 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 09:13:02.807895 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.805292 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 09:13:02.807895 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.805297 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 09:13:02.807895 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.805302 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 09:13:02.807895 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.805307 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 09:13:02.807895 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.805311 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 09:13:02.807895 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.805316 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 09:13:02.807895 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.805320 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 09:13:02.807895 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.805324 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 09:13:02.807895 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.805329 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 09:13:02.807895 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.805333 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 09:13:02.807895 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.805340 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 09:13:02.807895 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.805344 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 09:13:02.807895 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.805348 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 09:13:02.807895 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.805353 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 09:13:02.807895 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.805357 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 09:13:02.807895 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.805361 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 09:13:02.808602 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.805366 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 09:13:02.808602 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.805370 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 09:13:02.808602 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.805374 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 09:13:02.808602 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.805380 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 09:13:02.808602 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.805386 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 09:13:02.808602 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.805392 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 09:13:02.808602 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.805397 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 09:13:02.808602 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.805401 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 09:13:02.808602 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.805405 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 09:13:02.808602 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.805410 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 09:13:02.808602 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.805414 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 09:13:02.808602 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.805418 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 09:13:02.808602 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.805422 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 09:13:02.808602 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.805426 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 09:13:02.808602 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.805430 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 09:13:02.808602 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.805437 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 09:13:02.808602 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.806617 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 09:13:02.808602 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.806628 2575 feature_gate.go:328] unrecognized feature gate: Example Sep 11 09:13:02.808602 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.806633 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 09:13:02.809091 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.806638 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 09:13:02.809091 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.806643 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 09:13:02.809091 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.806649 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 09:13:02.809091 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.806655 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 09:13:02.809091 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.806659 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 09:13:02.809091 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.806664 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 09:13:02.809091 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.806668 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 09:13:02.809091 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.806672 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 09:13:02.809091 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.806678 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 09:13:02.809091 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.806684 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 09:13:02.809091 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.806689 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 09:13:02.809091 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.806695 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 09:13:02.809091 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.806700 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 09:13:02.809091 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.806704 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 09:13:02.809091 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.806709 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 09:13:02.809091 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.806714 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 09:13:02.809091 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.806719 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 09:13:02.809091 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.806724 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 09:13:02.809091 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.806728 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 09:13:02.809586 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.806733 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 09:13:02.809586 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.806738 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 09:13:02.809586 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.806744 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 09:13:02.809586 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.806748 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 09:13:02.809586 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.806752 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 09:13:02.809586 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.806756 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 09:13:02.809586 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.806760 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 09:13:02.809586 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.806766 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 09:13:02.809586 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.806770 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 09:13:02.809586 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.806774 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 09:13:02.809586 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.806779 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 09:13:02.809586 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.806784 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 09:13:02.809586 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.806789 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 09:13:02.809586 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.806793 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 09:13:02.809586 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.806798 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 09:13:02.809586 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.806802 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 09:13:02.809586 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.806809 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 09:13:02.809586 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.806814 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 09:13:02.809586 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.806818 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 09:13:02.810112 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.806822 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 09:13:02.810112 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.806827 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 09:13:02.810112 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.806831 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 09:13:02.810112 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.806835 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 09:13:02.810112 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.806839 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 09:13:02.810112 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.806861 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 09:13:02.810112 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.806866 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 09:13:02.810112 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.806870 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 09:13:02.810112 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.806874 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 09:13:02.810112 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.806878 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 09:13:02.810112 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.806882 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 09:13:02.810112 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.806886 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 09:13:02.810112 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.806892 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 09:13:02.810112 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.806896 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 09:13:02.810112 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.806902 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 09:13:02.810112 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.806906 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 09:13:02.810112 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.806910 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 09:13:02.810112 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.806914 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 09:13:02.810112 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.806918 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 09:13:02.810112 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.806923 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 09:13:02.810598 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.806927 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 09:13:02.810598 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.806931 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 09:13:02.810598 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.806935 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 09:13:02.810598 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.806941 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 09:13:02.810598 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.806946 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 09:13:02.810598 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.806950 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 09:13:02.810598 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.806955 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 09:13:02.810598 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.806962 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 09:13:02.810598 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.806967 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 09:13:02.810598 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.806973 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 09:13:02.810598 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.806978 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 09:13:02.810598 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.806983 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 09:13:02.810598 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.806987 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 09:13:02.810598 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.806992 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 09:13:02.810598 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.806996 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 09:13:02.810598 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.807001 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 09:13:02.810598 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.807006 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 09:13:02.810598 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.807010 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 09:13:02.810598 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.807014 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 09:13:02.811085 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.807018 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 09:13:02.811085 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.807023 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 09:13:02.811085 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.807027 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 09:13:02.811085 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.807031 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 09:13:02.811085 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.807035 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 09:13:02.811085 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.807039 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 09:13:02.811085 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.807053 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 09:13:02.811085 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.807058 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 09:13:02.811085 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.807062 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 09:13:02.811085 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.807067 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 09:13:02.811085 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.807071 2575 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 09:13:02.811085 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.807075 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 09:13:02.811085 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.807079 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 09:13:02.811085 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.807083 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 09:13:02.811085 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.807088 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 09:13:02.811085 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.807092 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 09:13:02.811085 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.807096 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 09:13:02.811085 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.807100 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 09:13:02.811085 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.807108 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 09:13:02.811085 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.809143 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 09:13:02.811576 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.809156 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 09:13:02.811576 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.809160 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 09:13:02.811576 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.809163 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 09:13:02.811576 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.809166 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 09:13:02.811576 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.809169 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 09:13:02.811576 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.809173 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 09:13:02.811576 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.809176 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 09:13:02.811576 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.809179 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 09:13:02.811576 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.809182 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 09:13:02.811576 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.809184 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 09:13:02.811576 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.809188 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 09:13:02.811576 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.809190 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 09:13:02.811576 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.809193 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 09:13:02.811576 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.809195 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 09:13:02.811576 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.809199 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 09:13:02.811576 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.809201 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 09:13:02.811576 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.809204 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 09:13:02.811576 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.809207 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 09:13:02.811576 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.809210 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 09:13:02.811576 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.809213 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 09:13:02.812091 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.809215 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 09:13:02.812091 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.809218 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 09:13:02.812091 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.809220 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 09:13:02.812091 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.809223 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 09:13:02.812091 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.809226 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 09:13:02.812091 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.809232 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 09:13:02.812091 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.809235 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 09:13:02.812091 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.809238 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 09:13:02.812091 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.809240 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 09:13:02.812091 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.809243 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 09:13:02.812091 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.809245 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 09:13:02.812091 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.809248 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 09:13:02.812091 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.809252 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 09:13:02.812091 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.809254 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 09:13:02.812091 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.809257 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 09:13:02.812091 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.809260 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 09:13:02.812091 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.809263 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 09:13:02.812091 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.809266 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 09:13:02.812091 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.809268 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 09:13:02.812091 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.809271 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 09:13:02.812589 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.809274 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 09:13:02.812589 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.809277 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 09:13:02.812589 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.809279 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 09:13:02.812589 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.809282 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 09:13:02.812589 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.809285 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 09:13:02.812589 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.809287 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 09:13:02.812589 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.809290 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 09:13:02.812589 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.809293 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 09:13:02.812589 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.809296 2575 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 09:13:02.812589 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.809298 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 09:13:02.812589 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.809301 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 09:13:02.812589 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.809306 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 09:13:02.812589 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.809311 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 09:13:02.812589 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.809316 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 09:13:02.812589 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.809319 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 09:13:02.812589 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.809322 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 09:13:02.812589 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.809325 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 09:13:02.812589 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.809327 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 09:13:02.812589 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.809330 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 09:13:02.813059 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.809332 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 09:13:02.813059 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.809335 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 09:13:02.813059 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.809338 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 09:13:02.813059 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.809340 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 09:13:02.813059 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.809344 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 09:13:02.813059 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.809349 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 09:13:02.813059 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.809352 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 09:13:02.813059 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.809355 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 09:13:02.813059 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.809358 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 09:13:02.813059 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.809361 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 09:13:02.813059 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.809364 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 09:13:02.813059 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.809366 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 09:13:02.813059 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.809369 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 09:13:02.813059 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.809372 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 09:13:02.813059 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.809374 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 09:13:02.813059 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.809377 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 09:13:02.813059 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.809379 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 09:13:02.813059 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.809382 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 09:13:02.813059 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.809384 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 09:13:02.813059 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.809387 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 09:13:02.813561 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.809390 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 09:13:02.813561 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.809392 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 09:13:02.813561 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.809395 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 09:13:02.813561 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.809404 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 09:13:02.813561 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.809408 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 09:13:02.813561 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.809411 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 09:13:02.813561 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.809414 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 09:13:02.813561 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.809417 2575 feature_gate.go:328] unrecognized feature gate: Example Sep 11 09:13:02.813561 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.809420 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 09:13:02.813561 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.809423 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 09:13:02.813561 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.809425 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 09:13:02.813561 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.809428 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 09:13:02.813561 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.809430 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 09:13:02.813561 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.809433 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 09:13:02.813561 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.809436 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 09:13:02.813561 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.809438 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 09:13:02.813561 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.809442 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 09:13:02.813561 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.809445 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 09:13:02.813561 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.809448 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 09:13:02.813561 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.809526 2575 flags.go:64] FLAG: --address="0.0.0.0" Sep 11 09:13:02.813561 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.809569 2575 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 11 09:13:02.814092 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.809582 2575 flags.go:64] FLAG: --anonymous-auth="true" Sep 11 09:13:02.814092 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.809587 2575 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 11 09:13:02.814092 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.809591 2575 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 11 09:13:02.814092 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.809595 2575 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 11 09:13:02.814092 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.809599 2575 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 11 09:13:02.814092 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.809604 2575 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 11 09:13:02.814092 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.809607 2575 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 11 09:13:02.814092 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.809610 2575 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 11 09:13:02.814092 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.809614 2575 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 11 09:13:02.814092 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.809617 2575 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 11 09:13:02.814092 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.809621 2575 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 11 09:13:02.814092 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.809624 2575 flags.go:64] FLAG: --cgroup-root="" Sep 11 09:13:02.814092 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.809627 2575 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 11 09:13:02.814092 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.809630 2575 flags.go:64] FLAG: --client-ca-file="" Sep 11 09:13:02.814092 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.809632 2575 flags.go:64] FLAG: --cloud-provider="external" Sep 11 09:13:02.814092 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.809641 2575 flags.go:64] FLAG: --cluster-dns="[]" Sep 11 09:13:02.814092 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.809648 2575 flags.go:64] FLAG: --cluster-domain="" Sep 11 09:13:02.814092 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.809651 2575 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 11 09:13:02.814092 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.809654 2575 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 11 09:13:02.814092 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.809657 2575 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 11 09:13:02.814092 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.809661 2575 flags.go:64] FLAG: --container-log-max-files="5" Sep 11 09:13:02.814092 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.809665 2575 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 11 09:13:02.814092 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.809668 2575 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 11 09:13:02.814646 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.809671 2575 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 11 09:13:02.814646 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.809675 2575 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 11 09:13:02.814646 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.809678 2575 flags.go:64] FLAG: --contention-profiling="false" Sep 11 09:13:02.814646 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.809681 2575 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 11 09:13:02.814646 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.809684 2575 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 11 09:13:02.814646 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.809687 2575 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 11 09:13:02.814646 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.809690 2575 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 11 09:13:02.814646 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.809695 2575 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 11 09:13:02.814646 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.809698 2575 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 11 09:13:02.814646 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.809701 2575 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 11 09:13:02.814646 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.809704 2575 flags.go:64] FLAG: --enable-load-reader="false" Sep 11 09:13:02.814646 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.809707 2575 flags.go:64] FLAG: --enable-server="true" Sep 11 09:13:02.814646 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.809710 2575 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 11 09:13:02.814646 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.809717 2575 flags.go:64] FLAG: --event-burst="100" Sep 11 09:13:02.814646 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.809721 2575 flags.go:64] FLAG: --event-qps="50" Sep 11 09:13:02.814646 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.809724 2575 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 11 09:13:02.814646 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.809727 2575 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 11 09:13:02.814646 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.809730 2575 flags.go:64] FLAG: --eviction-hard="" Sep 11 09:13:02.814646 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.809734 2575 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 11 09:13:02.814646 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.809736 2575 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 11 09:13:02.814646 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.809740 2575 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 11 09:13:02.814646 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.809743 2575 flags.go:64] FLAG: --eviction-soft="" Sep 11 09:13:02.814646 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.809746 2575 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 11 09:13:02.814646 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.809749 2575 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 11 09:13:02.814646 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.809752 2575 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 11 09:13:02.815324 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.809761 2575 flags.go:64] FLAG: --experimental-mounter-path="" Sep 11 09:13:02.815324 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.809764 2575 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 11 09:13:02.815324 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.809767 2575 flags.go:64] FLAG: --fail-swap-on="true" Sep 11 09:13:02.815324 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.809771 2575 flags.go:64] FLAG: --feature-gates="" Sep 11 09:13:02.815324 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.809775 2575 flags.go:64] FLAG: --file-check-frequency="20s" Sep 11 09:13:02.815324 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.809778 2575 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 11 09:13:02.815324 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.809781 2575 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 11 09:13:02.815324 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.809784 2575 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 11 09:13:02.815324 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.809787 2575 flags.go:64] FLAG: --healthz-port="10248" Sep 11 09:13:02.815324 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.809790 2575 flags.go:64] FLAG: --help="false" Sep 11 09:13:02.815324 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.809793 2575 flags.go:64] FLAG: --hostname-override="ip-10-0-135-119.ec2.internal" Sep 11 09:13:02.815324 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.809796 2575 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 11 09:13:02.815324 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.809799 2575 flags.go:64] FLAG: --http-check-frequency="20s" Sep 11 09:13:02.815324 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.809802 2575 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 11 09:13:02.815324 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.809805 2575 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 11 09:13:02.815324 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.809809 2575 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 11 09:13:02.815324 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.809812 2575 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 11 09:13:02.815324 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.809815 2575 flags.go:64] FLAG: --image-service-endpoint="" Sep 11 09:13:02.815324 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.809817 2575 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 11 09:13:02.815324 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.809820 2575 flags.go:64] FLAG: --kube-api-burst="100" Sep 11 09:13:02.815324 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.809823 2575 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 11 09:13:02.815324 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.809826 2575 flags.go:64] FLAG: --kube-api-qps="50" Sep 11 09:13:02.815324 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.809829 2575 flags.go:64] FLAG: --kube-reserved="" Sep 11 09:13:02.815324 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.809832 2575 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 11 09:13:02.815951 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.809835 2575 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 11 09:13:02.815951 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.809838 2575 flags.go:64] FLAG: --kubelet-cgroups="" Sep 11 09:13:02.815951 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.809840 2575 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 11 09:13:02.815951 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.809860 2575 flags.go:64] FLAG: --lock-file="" Sep 11 09:13:02.815951 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.809863 2575 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 11 09:13:02.815951 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.809865 2575 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 11 09:13:02.815951 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.809868 2575 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 11 09:13:02.815951 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.809874 2575 flags.go:64] FLAG: --log-json-split-stream="false" Sep 11 09:13:02.815951 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.809877 2575 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 11 09:13:02.815951 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.809886 2575 flags.go:64] FLAG: --log-text-split-stream="false" Sep 11 09:13:02.815951 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.809889 2575 flags.go:64] FLAG: --logging-format="text" Sep 11 09:13:02.815951 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.809893 2575 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 11 09:13:02.815951 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.809897 2575 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 11 09:13:02.815951 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.809900 2575 flags.go:64] FLAG: --manifest-url="" Sep 11 09:13:02.815951 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.809903 2575 flags.go:64] FLAG: --manifest-url-header="" Sep 11 09:13:02.815951 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.809908 2575 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 11 09:13:02.815951 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.809911 2575 flags.go:64] FLAG: --max-open-files="1000000" Sep 11 09:13:02.815951 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.809915 2575 flags.go:64] FLAG: --max-pods="110" Sep 11 09:13:02.815951 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.809918 2575 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 11 09:13:02.815951 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.809921 2575 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 11 09:13:02.815951 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.809924 2575 flags.go:64] FLAG: --memory-manager-policy="None" Sep 11 09:13:02.815951 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.809927 2575 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 11 09:13:02.815951 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.809930 2575 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 11 09:13:02.815951 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.809933 2575 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 11 09:13:02.815951 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.809937 2575 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 11 09:13:02.816559 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.809947 2575 flags.go:64] FLAG: --node-status-max-images="50" Sep 11 09:13:02.816559 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.809950 2575 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 11 09:13:02.816559 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.809953 2575 flags.go:64] FLAG: --oom-score-adj="-999" Sep 11 09:13:02.816559 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.809956 2575 flags.go:64] FLAG: --pod-cidr="" Sep 11 09:13:02.816559 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.809959 2575 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10" Sep 11 09:13:02.816559 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.809965 2575 flags.go:64] FLAG: --pod-manifest-path="" Sep 11 09:13:02.816559 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.809968 2575 flags.go:64] FLAG: --pod-max-pids="-1" Sep 11 09:13:02.816559 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.809971 2575 flags.go:64] FLAG: --pods-per-core="0" Sep 11 09:13:02.816559 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.809974 2575 flags.go:64] FLAG: --port="10250" Sep 11 09:13:02.816559 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.809977 2575 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 11 09:13:02.816559 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.809980 2575 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-00da1022d36aca44b" Sep 11 09:13:02.816559 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.809983 2575 flags.go:64] FLAG: --qos-reserved="" Sep 11 09:13:02.816559 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.809986 2575 flags.go:64] FLAG: --read-only-port="10255" Sep 11 09:13:02.816559 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.809989 2575 flags.go:64] FLAG: --register-node="true" Sep 11 09:13:02.816559 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.809992 2575 flags.go:64] FLAG: --register-with-taints="" Sep 11 09:13:02.816559 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.809996 2575 flags.go:64] FLAG: --registry-burst="10" Sep 11 09:13:02.816559 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.809999 2575 flags.go:64] FLAG: --registry-qps="5" Sep 11 09:13:02.816559 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.810008 2575 flags.go:64] FLAG: --reserved-cpus="" Sep 11 09:13:02.816559 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.810014 2575 flags.go:64] FLAG: --reserved-memory="" Sep 11 09:13:02.816559 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.810018 2575 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 11 09:13:02.816559 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.810021 2575 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 11 09:13:02.816559 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.810024 2575 flags.go:64] FLAG: --rotate-certificates="false" Sep 11 09:13:02.816559 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.810027 2575 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 11 09:13:02.816559 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.810030 2575 flags.go:64] FLAG: --runonce="false" Sep 11 09:13:02.816559 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.810032 2575 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 11 09:13:02.817216 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.810036 2575 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 11 09:13:02.817216 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.810039 2575 flags.go:64] FLAG: --seccomp-default="false" Sep 11 09:13:02.817216 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.810041 2575 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 11 09:13:02.817216 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.810044 2575 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 11 09:13:02.817216 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.810048 2575 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 11 09:13:02.817216 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.810051 2575 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 11 09:13:02.817216 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.810054 2575 flags.go:64] FLAG: --storage-driver-password="root" Sep 11 09:13:02.817216 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.810057 2575 flags.go:64] FLAG: --storage-driver-secure="false" Sep 11 09:13:02.817216 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.810060 2575 flags.go:64] FLAG: --storage-driver-table="stats" Sep 11 09:13:02.817216 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.810063 2575 flags.go:64] FLAG: --storage-driver-user="root" Sep 11 09:13:02.817216 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.810066 2575 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 11 09:13:02.817216 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.810069 2575 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 11 09:13:02.817216 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.810072 2575 flags.go:64] FLAG: --system-cgroups="" Sep 11 09:13:02.817216 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.810075 2575 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 11 09:13:02.817216 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.810080 2575 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 11 09:13:02.817216 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.810083 2575 flags.go:64] FLAG: --tls-cert-file="" Sep 11 09:13:02.817216 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.810086 2575 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 11 09:13:02.817216 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.810092 2575 flags.go:64] FLAG: --tls-min-version="" Sep 11 09:13:02.817216 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.810095 2575 flags.go:64] FLAG: --tls-private-key-file="" Sep 11 09:13:02.817216 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.810098 2575 flags.go:64] FLAG: --topology-manager-policy="none" Sep 11 09:13:02.817216 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.810100 2575 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 11 09:13:02.817216 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.810103 2575 flags.go:64] FLAG: --topology-manager-scope="container" Sep 11 09:13:02.817216 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.810106 2575 flags.go:64] FLAG: --v="2" Sep 11 09:13:02.817216 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.810110 2575 flags.go:64] FLAG: --version="false" Sep 11 09:13:02.817216 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.810114 2575 flags.go:64] FLAG: --vmodule="" Sep 11 09:13:02.817863 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.810126 2575 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 11 09:13:02.817863 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.810129 2575 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 11 09:13:02.817863 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.810261 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 09:13:02.817863 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.810264 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 09:13:02.817863 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.810268 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 09:13:02.817863 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.810271 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 09:13:02.817863 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.810274 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 09:13:02.817863 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.810276 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 09:13:02.817863 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.810279 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 09:13:02.817863 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.810282 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 09:13:02.817863 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.810286 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 09:13:02.817863 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.810289 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 09:13:02.817863 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.810292 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 09:13:02.817863 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.810295 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 09:13:02.817863 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.810298 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 09:13:02.817863 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.810300 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 09:13:02.817863 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.810303 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 09:13:02.817863 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.810306 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 09:13:02.817863 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.810309 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 09:13:02.818365 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.810311 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 09:13:02.818365 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.810314 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 09:13:02.818365 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.810316 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 09:13:02.818365 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.810319 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 09:13:02.818365 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.810322 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 09:13:02.818365 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.810324 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 09:13:02.818365 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.810327 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 09:13:02.818365 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.810329 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 09:13:02.818365 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.810332 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 09:13:02.818365 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.810334 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 09:13:02.818365 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.810337 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 09:13:02.818365 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.810340 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 09:13:02.818365 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.810346 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 09:13:02.818365 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.810351 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 09:13:02.818365 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.810360 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 09:13:02.818365 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.810362 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 09:13:02.818365 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.810365 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 09:13:02.818365 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.810368 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 09:13:02.818365 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.810370 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 09:13:02.818835 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.810373 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 09:13:02.818835 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.810376 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 09:13:02.818835 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.810378 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 09:13:02.818835 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.810381 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 09:13:02.818835 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.810383 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 09:13:02.818835 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.810386 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 09:13:02.818835 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.810389 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 09:13:02.818835 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.810392 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 09:13:02.818835 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.810394 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 09:13:02.818835 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.810397 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 09:13:02.818835 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.810400 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 09:13:02.818835 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.810403 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 09:13:02.818835 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.810405 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 09:13:02.818835 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.810408 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 09:13:02.818835 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.810411 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 09:13:02.818835 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.810413 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 09:13:02.818835 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.810416 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 09:13:02.818835 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.810419 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 09:13:02.818835 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.810421 2575 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 09:13:02.818835 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.810424 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 09:13:02.819363 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.810426 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 09:13:02.819363 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.810429 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 09:13:02.819363 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.810432 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 09:13:02.819363 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.810434 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 09:13:02.819363 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.810437 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 09:13:02.819363 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.810441 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 09:13:02.819363 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.810444 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 09:13:02.819363 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.810447 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 09:13:02.819363 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.810455 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 09:13:02.819363 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.810458 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 09:13:02.819363 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.810461 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 09:13:02.819363 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.810463 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 09:13:02.819363 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.810466 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 09:13:02.819363 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.810469 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 09:13:02.819363 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.810471 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 09:13:02.819363 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.810474 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 09:13:02.819363 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.810477 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 09:13:02.819363 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.810479 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 09:13:02.819363 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.810482 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 09:13:02.819363 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.810485 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 09:13:02.819886 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.810487 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 09:13:02.819886 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.810490 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 09:13:02.819886 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.810493 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 09:13:02.819886 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.810495 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 09:13:02.819886 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.810498 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 09:13:02.819886 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.810500 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 09:13:02.819886 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.810503 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 09:13:02.819886 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.810506 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 09:13:02.819886 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.810508 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 09:13:02.819886 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.810511 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 09:13:02.819886 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.810513 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 09:13:02.819886 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.810516 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 09:13:02.819886 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.810519 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 09:13:02.819886 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.810521 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 09:13:02.819886 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.810524 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 09:13:02.819886 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.810527 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 09:13:02.819886 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.810529 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 09:13:02.819886 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.810534 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 09:13:02.819886 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.810538 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 09:13:02.819886 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.810540 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 09:13:02.820404 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.810544 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 09:13:02.820404 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.810557 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 09:13:02.820404 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.810561 2575 feature_gate.go:328] unrecognized feature gate: Example Sep 11 09:13:02.820404 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.811070 2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 11 09:13:02.820404 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.818512 2575 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 11 09:13:02.820404 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.818530 2575 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 11 09:13:02.820404 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.819236 2575 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 11 09:13:02.820404 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.819248 2575 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 11 09:13:02.820404 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819307 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 09:13:02.820404 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819312 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 09:13:02.820404 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819316 2575 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 09:13:02.820404 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819319 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 09:13:02.820404 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819323 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 09:13:02.820404 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819326 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 09:13:02.820766 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819329 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 09:13:02.820766 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819332 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 09:13:02.820766 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819334 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 09:13:02.820766 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819337 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 09:13:02.820766 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819340 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 09:13:02.820766 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819342 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 09:13:02.820766 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819345 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 09:13:02.820766 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819348 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 09:13:02.820766 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819351 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 09:13:02.820766 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819353 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 09:13:02.820766 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819356 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 09:13:02.820766 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819359 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 09:13:02.820766 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819361 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 09:13:02.820766 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819364 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 09:13:02.820766 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819367 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 09:13:02.820766 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819370 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 09:13:02.820766 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819373 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 09:13:02.820766 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819376 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 09:13:02.820766 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819379 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 09:13:02.820766 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819382 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 09:13:02.821351 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819385 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 09:13:02.821351 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819388 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 09:13:02.821351 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819391 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 09:13:02.821351 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819396 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 09:13:02.821351 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819399 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 09:13:02.821351 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819401 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 09:13:02.821351 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819404 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 09:13:02.821351 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819407 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 09:13:02.821351 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819410 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 09:13:02.821351 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819413 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 09:13:02.821351 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819415 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 09:13:02.821351 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819418 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 09:13:02.821351 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819420 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 09:13:02.821351 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819423 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 09:13:02.821351 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819426 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 09:13:02.821351 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819429 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 09:13:02.821351 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819431 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 09:13:02.821351 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819434 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 09:13:02.821351 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819438 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 09:13:02.821351 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819442 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 09:13:02.821884 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819445 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 09:13:02.821884 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819448 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 09:13:02.821884 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819450 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 09:13:02.821884 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819453 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 09:13:02.821884 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819456 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 09:13:02.821884 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819458 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 09:13:02.821884 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819461 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 09:13:02.821884 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819464 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 09:13:02.821884 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819466 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 09:13:02.821884 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819469 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 09:13:02.821884 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819471 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 09:13:02.821884 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819474 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 09:13:02.821884 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819476 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 09:13:02.821884 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819480 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 09:13:02.821884 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819483 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 09:13:02.821884 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819486 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 09:13:02.821884 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819490 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 09:13:02.821884 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819493 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 09:13:02.821884 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819495 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 09:13:02.822384 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819499 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 09:13:02.822384 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819503 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 09:13:02.822384 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819506 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 09:13:02.822384 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819509 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 09:13:02.822384 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819511 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 09:13:02.822384 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819514 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 09:13:02.822384 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819516 2575 feature_gate.go:328] unrecognized feature gate: Example Sep 11 09:13:02.822384 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819519 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 09:13:02.822384 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819522 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 09:13:02.822384 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819525 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 09:13:02.822384 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819527 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 09:13:02.822384 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819530 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 09:13:02.822384 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819533 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 09:13:02.822384 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819535 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 09:13:02.822384 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819538 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 09:13:02.822384 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819540 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 09:13:02.822384 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819543 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 09:13:02.822384 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819547 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 09:13:02.822384 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819551 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 09:13:02.822927 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819554 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 09:13:02.822927 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819556 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 09:13:02.822927 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819559 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 09:13:02.822927 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819562 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 09:13:02.822927 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819565 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 09:13:02.822927 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819567 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 09:13:02.822927 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819570 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 09:13:02.822927 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819573 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 09:13:02.822927 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819576 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 09:13:02.822927 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819579 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 09:13:02.822927 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819582 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 09:13:02.822927 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819585 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 09:13:02.822927 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819588 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 09:13:02.822927 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819590 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 09:13:02.822927 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819593 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 09:13:02.823311 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.819598 2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 11 09:13:02.823311 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819714 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 09:13:02.823311 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819718 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 09:13:02.823311 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819723 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 09:13:02.823311 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819726 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 09:13:02.823311 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819729 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 09:13:02.823311 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819732 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 09:13:02.823311 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819736 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 09:13:02.823311 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819739 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 09:13:02.823311 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819742 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 09:13:02.823311 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819744 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 09:13:02.823311 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819747 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 09:13:02.823311 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819750 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 09:13:02.823311 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819753 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 09:13:02.823673 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819756 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 09:13:02.823673 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819758 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 09:13:02.823673 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819761 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 09:13:02.823673 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819763 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 09:13:02.823673 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819766 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 09:13:02.823673 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819769 2575 feature_gate.go:328] unrecognized feature gate: Example Sep 11 09:13:02.823673 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819771 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 09:13:02.823673 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819774 2575 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 09:13:02.823673 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819776 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 09:13:02.823673 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819780 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 09:13:02.823673 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819783 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 09:13:02.823673 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819787 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 09:13:02.823673 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819790 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 09:13:02.823673 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819793 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 09:13:02.823673 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819795 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 09:13:02.823673 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819799 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 09:13:02.823673 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819802 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 09:13:02.823673 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819804 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 09:13:02.823673 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819807 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 09:13:02.823673 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819809 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 09:13:02.824224 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819812 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 09:13:02.824224 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819815 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 09:13:02.824224 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819817 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 09:13:02.824224 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819819 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 09:13:02.824224 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819822 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 09:13:02.824224 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819825 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 09:13:02.824224 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819828 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 09:13:02.824224 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819830 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 09:13:02.824224 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819833 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 09:13:02.824224 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819835 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 09:13:02.824224 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819838 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 09:13:02.824224 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819857 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 09:13:02.824224 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819860 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 09:13:02.824224 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819863 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 09:13:02.824224 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819866 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 09:13:02.824224 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819869 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 09:13:02.824224 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819871 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 09:13:02.824224 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819874 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 09:13:02.824224 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819877 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 09:13:02.824224 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819879 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 09:13:02.824716 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819882 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 09:13:02.824716 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819885 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 09:13:02.824716 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819887 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 09:13:02.824716 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819890 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 09:13:02.824716 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819893 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 09:13:02.824716 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819896 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 09:13:02.824716 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819899 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 09:13:02.824716 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819903 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 09:13:02.824716 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819906 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 09:13:02.824716 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819909 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 09:13:02.824716 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819911 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 09:13:02.824716 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819914 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 09:13:02.824716 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819918 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 09:13:02.824716 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819922 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 09:13:02.824716 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819925 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 09:13:02.824716 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819928 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 09:13:02.824716 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819930 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 09:13:02.824716 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819933 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 09:13:02.824716 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819936 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 09:13:02.825218 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819938 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 09:13:02.825218 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819941 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 09:13:02.825218 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819944 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 09:13:02.825218 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819946 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 09:13:02.825218 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819949 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 09:13:02.825218 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819951 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 09:13:02.825218 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819954 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 09:13:02.825218 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819957 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 09:13:02.825218 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819959 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 09:13:02.825218 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819962 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 09:13:02.825218 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819964 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 09:13:02.825218 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819967 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 09:13:02.825218 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819970 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 09:13:02.825218 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819972 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 09:13:02.825218 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819975 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 09:13:02.825218 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819977 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 09:13:02.825218 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819980 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 09:13:02.825218 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819983 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 09:13:02.825218 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819986 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 09:13:02.825218 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819989 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 09:13:02.825772 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819992 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 09:13:02.825772 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819994 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 09:13:02.825772 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.819997 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 09:13:02.825772 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.820000 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 09:13:02.825772 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.820003 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 09:13:02.825772 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.820005 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 09:13:02.825772 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:02.820008 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 09:13:02.825772 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.820013 2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 11 09:13:02.825772 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.820154 2575 server.go:962] "Client rotation is on, will bootstrap in background" Sep 11 09:13:02.825772 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.822441 2575 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 11 09:13:02.825772 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.823126 2575 server.go:1019] "Starting client certificate rotation" Sep 11 09:13:02.825772 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.823254 2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 11 09:13:02.825772 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.823324 2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 11 09:13:02.826127 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.826089 2575 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 11 09:13:02.829193 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.829179 2575 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 11 09:13:02.840308 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.840104 2575 log.go:25] "Validated CRI v1 runtime API" Sep 11 09:13:02.843476 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.843457 2575 log.go:25] "Validated CRI v1 image API" Sep 11 09:13:02.844244 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.844227 2575 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 11 09:13:02.848164 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.848144 2575 fs.go:135] Filesystem UUIDs: map[357a9ab1-5f83-4b5f-97c4-bd637869ccb1:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 bc98e40d-62b3-40ef-a2c5-7f54fb477f10:/dev/nvme0n1p3] Sep 11 09:13:02.848235 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.848163 2575 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 11 09:13:02.852103 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.852086 2575 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 11 09:13:02.853822 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.853699 2575 manager.go:216] Machine: {Timestamp:2026-09-11 09:13:02.853203098 +0000 UTC m=+0.287435847 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100318 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec260f395135d25d7590edf0f5c14640 SystemUUID:ec260f39-5135-d25d-7590-edf0f5c14640 BootID:ce8b753d-494c-449f-a240-fee960841644 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:b0:80:7f:6f:d1 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:b0:80:7f:6f:d1 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:5e:96:30:2a:81:80 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 11 09:13:02.853916 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.853824 2575 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 11 09:13:02.853956 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.853942 2575 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 11 09:13:02.857230 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.857205 2575 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 11 09:13:02.857379 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.857233 2575 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-119.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 11 09:13:02.857981 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.857971 2575 topology_manager.go:138] "Creating topology manager with none policy" Sep 11 09:13:02.858013 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.857984 2575 container_manager_linux.go:309] "Creating device plugin manager" Sep 11 09:13:02.858013 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.857992 2575 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 11 09:13:02.858628 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.858619 2575 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 11 09:13:02.859699 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.859687 2575 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 11 09:13:02.864212 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.864200 2575 state_mem.go:36] "Initialized new in-memory state store" Sep 11 09:13:02.864337 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.864328 2575 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 11 09:13:02.864371 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.864362 2575 kubelet.go:486] "Attempting to sync node with API server" Sep 11 09:13:02.864403 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.864371 2575 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 11 09:13:02.864403 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.864385 2575 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 11 09:13:02.864403 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.864396 2575 kubelet.go:398] "Adding apiserver pod source" Sep 11 09:13:02.864403 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.864404 2575 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 11 09:13:02.864822 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.864807 2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 11 09:13:02.864897 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.864834 2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 11 09:13:02.867184 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.867164 2575 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 11 09:13:02.869366 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.869349 2575 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 11 09:13:02.869436 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.869382 2575 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 11 09:13:02.871129 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.871113 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 11 09:13:02.871129 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.871131 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 11 09:13:02.871227 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.871138 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 11 09:13:02.871227 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.871144 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 11 09:13:02.871227 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.871153 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 11 09:13:02.871227 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.871159 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 11 09:13:02.871227 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.871174 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 11 09:13:02.871227 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.871181 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 11 09:13:02.871227 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.871191 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 11 09:13:02.871227 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.871199 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 11 09:13:02.871227 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.871220 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 11 09:13:02.871227 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.871229 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 11 09:13:02.872658 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.872643 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 11 09:13:02.872658 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.872659 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 11 09:13:02.874652 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:02.874628 2575 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 11 09:13:02.874696 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:02.874634 2575 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-119.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 11 09:13:02.876540 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.876527 2575 server.go:1268] "Started kubelet" Sep 11 09:13:02.876645 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.876622 2575 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 11 09:13:02.876708 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.876653 2575 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 11 09:13:02.876752 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.876738 2575 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 11 09:13:02.877557 ip-10-0-135-119 systemd[1]: Started Kubernetes Kubelet. Sep 11 09:13:02.880023 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.880001 2575 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 11 09:13:02.881543 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:02.881258 2575 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 11 09:13:02.881543 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.881004 2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-k6lvv" Sep 11 09:13:02.881655 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.881633 2575 server.go:310] "Adding debug handlers to kubelet server" Sep 11 09:13:02.886320 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.886297 2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 11 09:13:02.886722 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.886704 2575 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-119.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 11 09:13:02.886788 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.886770 2575 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 11 09:13:02.887124 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:02.887104 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-119.ec2.internal\" not found" Sep 11 09:13:02.887429 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.887397 2575 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 11 09:13:02.887429 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:02.886601 2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-119.ec2.internal.18d43953869cc266 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-119.ec2.internal,UID:ip-10-0-135-119.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-119.ec2.internal,},FirstTimestamp:2026-09-11 09:13:02.876418662 +0000 UTC m=+0.310651411,LastTimestamp:2026-09-11 09:13:02.876418662 +0000 UTC m=+0.310651411,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-119.ec2.internal,}" Sep 11 09:13:02.887553 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.887526 2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-k6lvv" Sep 11 09:13:02.887623 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.887398 2575 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 11 09:13:02.887623 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.887623 2575 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 11 09:13:02.887748 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.887706 2575 reconstruct.go:97] "Volume reconstruction finished" Sep 11 09:13:02.887748 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.887717 2575 reconciler.go:29] "Reconciler: start to sync state" Sep 11 09:13:02.888327 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.888312 2575 factory.go:153] Registering CRI-O factory Sep 11 09:13:02.888379 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.888333 2575 factory.go:223] Registration of the crio container factory successfully Sep 11 09:13:02.888408 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.888400 2575 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 11 09:13:02.888408 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.888407 2575 factory.go:55] Registering systemd factory Sep 11 09:13:02.888492 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.888413 2575 factory.go:223] Registration of the systemd container factory successfully Sep 11 09:13:02.888492 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.888426 2575 factory.go:103] Registering Raw factory Sep 11 09:13:02.888492 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.888435 2575 manager.go:1196] Started watching for new ooms in manager Sep 11 09:13:02.888788 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.888777 2575 manager.go:349] Starting recovery of all containers Sep 11 09:13:02.890007 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:02.889983 2575 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 11 09:13:02.890090 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:02.890057 2575 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-119.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 11 09:13:02.898787 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.898771 2575 manager.go:354] Recovery completed Sep 11 09:13:02.904808 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.904658 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 09:13:02.906758 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.906739 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-119.ec2.internal" event="NodeHasSufficientMemory" Sep 11 09:13:02.906829 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.906778 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-119.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 09:13:02.906829 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.906794 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-119.ec2.internal" event="NodeHasSufficientPID" Sep 11 09:13:02.907374 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.907357 2575 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 11 09:13:02.907374 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.907370 2575 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 11 09:13:02.907374 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.907374 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 09:13:02.907518 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.907384 2575 state_mem.go:36] "Initialized new in-memory state store" Sep 11 09:13:02.908136 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.908119 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-119.ec2.internal" event="NodeHasSufficientMemory" Sep 11 09:13:02.908198 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.908146 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-119.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 09:13:02.908198 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.908164 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-119.ec2.internal" event="NodeHasSufficientPID" Sep 11 09:13:02.909398 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.909381 2575 policy_none.go:49] "None policy: Start" Sep 11 09:13:02.909398 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.909400 2575 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 11 09:13:02.909485 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.909411 2575 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 11 09:13:02.911446 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.911433 2575 policy_none.go:47] "Start" Sep 11 09:13:02.973277 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.947627 2575 manager.go:337] "Starting Device Plugin manager" Sep 11 09:13:02.973277 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:02.947676 2575 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 11 09:13:02.973277 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.947691 2575 server.go:87] "Starting device plugin registration server" Sep 11 09:13:02.973277 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.948112 2575 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 11 09:13:02.973277 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.948127 2575 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 11 09:13:02.973277 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.948223 2575 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 11 09:13:02.973277 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.948312 2575 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 11 09:13:02.973277 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:02.948320 2575 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 11 09:13:02.973277 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:02.949202 2575 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 11 09:13:02.973277 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:02.949253 2575 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-119.ec2.internal\" not found" Sep 11 09:13:03.013125 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:03.013087 2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 11 09:13:03.014516 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:03.014496 2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 11 09:13:03.014610 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:03.014565 2575 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 11 09:13:03.014610 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:03.014603 2575 kubelet.go:2448] "Starting kubelet main sync loop" Sep 11 09:13:03.014737 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:03.014721 2575 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 11 09:13:03.017163 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:03.017146 2575 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 09:13:03.048464 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:03.048434 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 09:13:03.049509 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:03.049483 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-119.ec2.internal" event="NodeHasSufficientMemory" Sep 11 09:13:03.049617 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:03.049526 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-119.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 09:13:03.049617 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:03.049540 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-119.ec2.internal" event="NodeHasSufficientPID" Sep 11 09:13:03.049617 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:03.049570 2575 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-119.ec2.internal" Sep 11 09:13:03.058315 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:03.058242 2575 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-135-119.ec2.internal" Sep 11 09:13:03.058315 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:03.058271 2575 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-119.ec2.internal\": node \"ip-10-0-135-119.ec2.internal\" not found" Sep 11 09:13:03.075594 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:03.075561 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-119.ec2.internal\" not found" Sep 11 09:13:03.115033 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:03.114996 2575 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-119.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-119.ec2.internal"] Sep 11 09:13:03.115178 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:03.115079 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 09:13:03.117051 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:03.117032 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-119.ec2.internal" event="NodeHasSufficientMemory" Sep 11 09:13:03.117161 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:03.117065 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-119.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 09:13:03.117161 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:03.117077 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-119.ec2.internal" event="NodeHasSufficientPID" Sep 11 09:13:03.119296 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:03.119280 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 09:13:03.119428 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:03.119411 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-119.ec2.internal" Sep 11 09:13:03.119480 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:03.119449 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 09:13:03.120092 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:03.120074 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-119.ec2.internal" event="NodeHasSufficientMemory" Sep 11 09:13:03.120092 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:03.120077 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-119.ec2.internal" event="NodeHasSufficientMemory" Sep 11 09:13:03.120247 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:03.120120 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-119.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 09:13:03.120247 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:03.120132 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-119.ec2.internal" event="NodeHasSufficientPID" Sep 11 09:13:03.120247 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:03.120096 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-119.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 09:13:03.120247 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:03.120173 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-119.ec2.internal" event="NodeHasSufficientPID" Sep 11 09:13:03.122348 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:03.122332 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-119.ec2.internal" Sep 11 09:13:03.122426 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:03.122360 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 09:13:03.123072 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:03.123056 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-119.ec2.internal" event="NodeHasSufficientMemory" Sep 11 09:13:03.123169 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:03.123083 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-119.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 09:13:03.123169 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:03.123094 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-119.ec2.internal" event="NodeHasSufficientPID" Sep 11 09:13:03.145633 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:03.145607 2575 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-119.ec2.internal\" not found" node="ip-10-0-135-119.ec2.internal" Sep 11 09:13:03.150177 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:03.150159 2575 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-119.ec2.internal\" not found" node="ip-10-0-135-119.ec2.internal" Sep 11 09:13:03.176588 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:03.176560 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-119.ec2.internal\" not found" Sep 11 09:13:03.189626 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:03.189601 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ce93b5afb1de6b9a80d64df7da81462d-config\") pod \"kube-apiserver-proxy-ip-10-0-135-119.ec2.internal\" (UID: \"ce93b5afb1de6b9a80d64df7da81462d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-119.ec2.internal" Sep 11 09:13:03.189712 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:03.189630 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/db42fcd462e8e18de823d8d0adf7cc0a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-119.ec2.internal\" (UID: \"db42fcd462e8e18de823d8d0adf7cc0a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-119.ec2.internal" Sep 11 09:13:03.189712 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:03.189650 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/db42fcd462e8e18de823d8d0adf7cc0a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-119.ec2.internal\" (UID: \"db42fcd462e8e18de823d8d0adf7cc0a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-119.ec2.internal" Sep 11 09:13:03.277030 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:03.276999 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-119.ec2.internal\" not found" Sep 11 09:13:03.290386 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:03.290362 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/db42fcd462e8e18de823d8d0adf7cc0a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-119.ec2.internal\" (UID: \"db42fcd462e8e18de823d8d0adf7cc0a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-119.ec2.internal" Sep 11 09:13:03.290475 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:03.290394 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/db42fcd462e8e18de823d8d0adf7cc0a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-119.ec2.internal\" (UID: \"db42fcd462e8e18de823d8d0adf7cc0a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-119.ec2.internal" Sep 11 09:13:03.290475 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:03.290420 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ce93b5afb1de6b9a80d64df7da81462d-config\") pod \"kube-apiserver-proxy-ip-10-0-135-119.ec2.internal\" (UID: \"ce93b5afb1de6b9a80d64df7da81462d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-119.ec2.internal" Sep 11 09:13:03.290567 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:03.290485 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/db42fcd462e8e18de823d8d0adf7cc0a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-119.ec2.internal\" (UID: \"db42fcd462e8e18de823d8d0adf7cc0a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-119.ec2.internal" Sep 11 09:13:03.290567 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:03.290505 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ce93b5afb1de6b9a80d64df7da81462d-config\") pod \"kube-apiserver-proxy-ip-10-0-135-119.ec2.internal\" (UID: \"ce93b5afb1de6b9a80d64df7da81462d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-119.ec2.internal" Sep 11 09:13:03.290567 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:03.290485 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/db42fcd462e8e18de823d8d0adf7cc0a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-119.ec2.internal\" (UID: \"db42fcd462e8e18de823d8d0adf7cc0a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-119.ec2.internal" Sep 11 09:13:03.377878 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:03.377770 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-119.ec2.internal\" not found" Sep 11 09:13:04.541909 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:04.541871 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-119.ec2.internal" Sep 11 09:13:04.547476 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:04.547364 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-119.ec2.internal" Sep 11 09:13:04.572460 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:04.572435 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-119.ec2.internal\" not found" Sep 11 09:13:04.672948 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:04.672901 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-119.ec2.internal\" not found" Sep 11 09:13:04.773495 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:04.773386 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-119.ec2.internal\" not found" Sep 11 09:13:04.873918 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:04.873870 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-119.ec2.internal\" not found" Sep 11 09:13:04.917439 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:04.917406 2575 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 11 09:13:04.917944 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:04.917595 2575 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 11 09:13:04.974902 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:04.974848 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-119.ec2.internal\" not found" Sep 11 09:13:04.986980 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:04.986944 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-10 09:08:03 +0000 UTC" deadline="2028-06-26 18:14:29.33938742 +0000 UTC" Sep 11 09:13:04.986980 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:04.986974 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15705h1m24.352416505s" Sep 11 09:13:05.056919 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:05.056834 2575 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 09:13:05.081568 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:05.081547 2575 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-119.ec2.internal" Sep 11 09:13:05.094508 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:05.094484 2575 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 11 09:13:05.095929 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:05.095913 2575 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-119.ec2.internal" Sep 11 09:13:05.105718 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:05.105696 2575 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 11 09:13:05.157722 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:05.157692 2575 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 09:13:05.222877 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:05.222829 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce93b5afb1de6b9a80d64df7da81462d.slice/crio-7ed018af667b5a2b7a4977308558031db76ff64d944c1f21bb15bb7a251da263 WatchSource:0}: Error finding container 7ed018af667b5a2b7a4977308558031db76ff64d944c1f21bb15bb7a251da263: Status 404 returned error can't find the container with id 7ed018af667b5a2b7a4977308558031db76ff64d944c1f21bb15bb7a251da263 Sep 11 09:13:05.223211 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:05.223190 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb42fcd462e8e18de823d8d0adf7cc0a.slice/crio-e79c76f9564878a63ab6c97d325effc95f68792d18a395d69a10d1d234b30166 WatchSource:0}: Error finding container e79c76f9564878a63ab6c97d325effc95f68792d18a395d69a10d1d234b30166: Status 404 returned error can't find the container with id e79c76f9564878a63ab6c97d325effc95f68792d18a395d69a10d1d234b30166 Sep 11 09:13:05.227804 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:05.227789 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 09:13:05.339149 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:05.339055 2575 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 09:13:05.777009 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:05.776935 2575 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 09:13:05.959712 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:05.959674 2575 apiserver.go:52] "Watching apiserver" Sep 11 09:13:05.971965 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:05.970694 2575 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 11 09:13:05.971965 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:05.971743 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-pq626","kube-system/kube-apiserver-proxy-ip-10-0-135-119.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l46kr","openshift-cluster-node-tuning-operator/tuned-9zgvm","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-119.ec2.internal","openshift-multus/multus-dvpdv","openshift-multus/network-metrics-daemon-rlftv","openshift-network-diagnostics/network-check-target-bb6x9","kube-system/global-pull-secret-syncer-8ng6h","openshift-dns/node-resolver-g6t8z","openshift-image-registry/node-ca-skn2c","openshift-multus/multus-additional-cni-plugins-8z7wv","openshift-network-operator/iptables-alerter-5xwxg","openshift-ovn-kubernetes/ovnkube-node-wgrlw"] Sep 11 09:13:05.974570 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:05.974540 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rlftv" Sep 11 09:13:05.974717 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:05.974636 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rlftv" podUID="abeb037b-2e12-4485-974d-35280f982f4a" Sep 11 09:13:05.977524 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:05.977008 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-skn2c" Sep 11 09:13:05.981668 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:05.980174 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l46kr" Sep 11 09:13:05.981668 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:05.981052 2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 11 09:13:05.981668 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:05.981302 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 11 09:13:05.981668 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:05.981387 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 11 09:13:05.981668 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:05.981394 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 11 09:13:05.981668 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:05.981303 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-h5mth\"" Sep 11 09:13:05.981668 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:05.981597 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-9zgvm" Sep 11 09:13:05.984125 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:05.983689 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-wgrlw" Sep 11 09:13:05.986323 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:05.984841 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 11 09:13:05.986323 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:05.985126 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 11 09:13:05.986323 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:05.985368 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 11 09:13:05.986323 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:05.985564 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 11 09:13:05.986323 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:05.985790 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-c2mzm\"" Sep 11 09:13:05.986323 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:05.985995 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 11 09:13:05.986323 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:05.986183 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-6rtss\"" Sep 11 09:13:05.986780 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:05.986487 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-dvpdv" Sep 11 09:13:05.987686 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:05.987219 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-f7h44\"" Sep 11 09:13:05.987686 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:05.987314 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 11 09:13:05.987686 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:05.987359 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 11 09:13:05.990240 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:05.988243 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 11 09:13:05.990240 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:05.988461 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 11 09:13:05.990240 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:05.988848 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 11 09:13:05.990240 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:05.989023 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 11 09:13:05.990240 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:05.989528 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 11 09:13:05.990240 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:05.989871 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 11 09:13:05.990240 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:05.990106 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 11 09:13:05.990668 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:05.990274 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 11 09:13:05.991088 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:05.990821 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8ng6h" Sep 11 09:13:05.991088 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:05.990890 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8ng6h" podUID="7eddd8ad-0ac3-421d-9649-95dc4fadfd7b" Sep 11 09:13:05.991256 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:05.991149 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-z4rm2\"" Sep 11 09:13:05.992149 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:05.992130 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-5xwxg" Sep 11 09:13:05.992767 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:05.992749 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3c0627e0-f086-4fc4-b2d7-724b67c685d4-run-openvswitch\") pod \"ovnkube-node-wgrlw\" (UID: \"3c0627e0-f086-4fc4-b2d7-724b67c685d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgrlw" Sep 11 09:13:05.992988 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:05.992972 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3c0627e0-f086-4fc4-b2d7-724b67c685d4-env-overrides\") pod \"ovnkube-node-wgrlw\" (UID: \"3c0627e0-f086-4fc4-b2d7-724b67c685d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgrlw" Sep 11 09:13:05.993160 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:05.993143 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abeb037b-2e12-4485-974d-35280f982f4a-metrics-certs\") pod \"network-metrics-daemon-rlftv\" (UID: \"abeb037b-2e12-4485-974d-35280f982f4a\") " pod="openshift-multus/network-metrics-daemon-rlftv" Sep 11 09:13:05.993276 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:05.993262 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3c0627e0-f086-4fc4-b2d7-724b67c685d4-host-slash\") pod \"ovnkube-node-wgrlw\" (UID: \"3c0627e0-f086-4fc4-b2d7-724b67c685d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgrlw" Sep 11 09:13:05.993376 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:05.993363 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3c0627e0-f086-4fc4-b2d7-724b67c685d4-run-systemd\") pod \"ovnkube-node-wgrlw\" (UID: \"3c0627e0-f086-4fc4-b2d7-724b67c685d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgrlw" Sep 11 09:13:05.993508 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:05.993490 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hd6lj\" (UniqueName: \"kubernetes.io/projected/08d48976-3c61-4273-88da-fa30ba4057bc-kube-api-access-hd6lj\") pod \"node-ca-skn2c\" (UID: \"08d48976-3c61-4273-88da-fa30ba4057bc\") " pod="openshift-image-registry/node-ca-skn2c" Sep 11 09:13:05.993613 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:05.993598 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fcb16b38-81cb-41f7-81d2-730c64ee3a59-etc-modprobe-d\") pod \"tuned-9zgvm\" (UID: \"fcb16b38-81cb-41f7-81d2-730c64ee3a59\") " pod="openshift-cluster-node-tuning-operator/tuned-9zgvm" Sep 11 09:13:05.993762 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:05.993736 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fcb16b38-81cb-41f7-81d2-730c64ee3a59-run\") pod \"tuned-9zgvm\" (UID: \"fcb16b38-81cb-41f7-81d2-730c64ee3a59\") " pod="openshift-cluster-node-tuning-operator/tuned-9zgvm" Sep 11 09:13:05.993882 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:05.993844 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3c0627e0-f086-4fc4-b2d7-724b67c685d4-systemd-units\") pod \"ovnkube-node-wgrlw\" (UID: \"3c0627e0-f086-4fc4-b2d7-724b67c685d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgrlw" Sep 11 09:13:05.993955 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:05.993887 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3c0627e0-f086-4fc4-b2d7-724b67c685d4-ovnkube-config\") pod \"ovnkube-node-wgrlw\" (UID: \"3c0627e0-f086-4fc4-b2d7-724b67c685d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgrlw" Sep 11 09:13:05.993955 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:05.993912 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fcb16b38-81cb-41f7-81d2-730c64ee3a59-etc-sysctl-conf\") pod \"tuned-9zgvm\" (UID: \"fcb16b38-81cb-41f7-81d2-730c64ee3a59\") " pod="openshift-cluster-node-tuning-operator/tuned-9zgvm" Sep 11 09:13:05.994063 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:05.993993 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fcb16b38-81cb-41f7-81d2-730c64ee3a59-etc-tuned\") pod \"tuned-9zgvm\" (UID: \"fcb16b38-81cb-41f7-81d2-730c64ee3a59\") " pod="openshift-cluster-node-tuning-operator/tuned-9zgvm" Sep 11 09:13:05.994063 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:05.994022 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1f74babf-697e-4234-b22e-f7a18ba4a753-etc-selinux\") pod \"aws-ebs-csi-driver-node-l46kr\" (UID: \"1f74babf-697e-4234-b22e-f7a18ba4a753\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l46kr" Sep 11 09:13:05.994159 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:05.994067 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1f74babf-697e-4234-b22e-f7a18ba4a753-kubelet-dir\") pod \"aws-ebs-csi-driver-node-l46kr\" (UID: \"1f74babf-697e-4234-b22e-f7a18ba4a753\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l46kr" Sep 11 09:13:05.994159 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:05.994091 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1f74babf-697e-4234-b22e-f7a18ba4a753-socket-dir\") pod \"aws-ebs-csi-driver-node-l46kr\" (UID: \"1f74babf-697e-4234-b22e-f7a18ba4a753\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l46kr" Sep 11 09:13:05.994159 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:05.994121 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fcb16b38-81cb-41f7-81d2-730c64ee3a59-etc-sysconfig\") pod \"tuned-9zgvm\" (UID: \"fcb16b38-81cb-41f7-81d2-730c64ee3a59\") " pod="openshift-cluster-node-tuning-operator/tuned-9zgvm" Sep 11 09:13:05.994961 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:05.994489 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3c0627e0-f086-4fc4-b2d7-724b67c685d4-host-kubelet\") pod \"ovnkube-node-wgrlw\" (UID: \"3c0627e0-f086-4fc4-b2d7-724b67c685d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgrlw" Sep 11 09:13:05.994961 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:05.994576 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3c0627e0-f086-4fc4-b2d7-724b67c685d4-etc-openvswitch\") pod \"ovnkube-node-wgrlw\" (UID: \"3c0627e0-f086-4fc4-b2d7-724b67c685d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgrlw" Sep 11 09:13:05.994961 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:05.994601 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/08d48976-3c61-4273-88da-fa30ba4057bc-serviceca\") pod \"node-ca-skn2c\" (UID: \"08d48976-3c61-4273-88da-fa30ba4057bc\") " pod="openshift-image-registry/node-ca-skn2c" Sep 11 09:13:05.994961 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:05.994625 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3c0627e0-f086-4fc4-b2d7-724b67c685d4-run-ovn\") pod \"ovnkube-node-wgrlw\" (UID: \"3c0627e0-f086-4fc4-b2d7-724b67c685d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgrlw" Sep 11 09:13:05.994961 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:05.994646 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3c0627e0-f086-4fc4-b2d7-724b67c685d4-log-socket\") pod \"ovnkube-node-wgrlw\" (UID: \"3c0627e0-f086-4fc4-b2d7-724b67c685d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgrlw" Sep 11 09:13:05.994961 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:05.994669 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3c0627e0-f086-4fc4-b2d7-724b67c685d4-host-run-ovn-kubernetes\") pod \"ovnkube-node-wgrlw\" (UID: \"3c0627e0-f086-4fc4-b2d7-724b67c685d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgrlw" Sep 11 09:13:05.994961 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:05.994695 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3c0627e0-f086-4fc4-b2d7-724b67c685d4-host-cni-bin\") pod \"ovnkube-node-wgrlw\" (UID: \"3c0627e0-f086-4fc4-b2d7-724b67c685d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgrlw" Sep 11 09:13:05.994961 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:05.994745 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3c0627e0-f086-4fc4-b2d7-724b67c685d4-host-cni-netd\") pod \"ovnkube-node-wgrlw\" (UID: \"3c0627e0-f086-4fc4-b2d7-724b67c685d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgrlw" Sep 11 09:13:05.994961 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:05.994771 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3c0627e0-f086-4fc4-b2d7-724b67c685d4-ovn-node-metrics-cert\") pod \"ovnkube-node-wgrlw\" (UID: \"3c0627e0-f086-4fc4-b2d7-724b67c685d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgrlw" Sep 11 09:13:05.994961 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:05.994796 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3c0627e0-f086-4fc4-b2d7-724b67c685d4-ovnkube-script-lib\") pod \"ovnkube-node-wgrlw\" (UID: \"3c0627e0-f086-4fc4-b2d7-724b67c685d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgrlw" Sep 11 09:13:05.994961 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:05.994819 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1f74babf-697e-4234-b22e-f7a18ba4a753-device-dir\") pod \"aws-ebs-csi-driver-node-l46kr\" (UID: \"1f74babf-697e-4234-b22e-f7a18ba4a753\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l46kr" Sep 11 09:13:05.994961 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:05.994884 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvd2w\" (UniqueName: \"kubernetes.io/projected/abeb037b-2e12-4485-974d-35280f982f4a-kube-api-access-nvd2w\") pod \"network-metrics-daemon-rlftv\" (UID: \"abeb037b-2e12-4485-974d-35280f982f4a\") " pod="openshift-multus/network-metrics-daemon-rlftv" Sep 11 09:13:05.994961 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:05.994909 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fcb16b38-81cb-41f7-81d2-730c64ee3a59-etc-kubernetes\") pod \"tuned-9zgvm\" (UID: \"fcb16b38-81cb-41f7-81d2-730c64ee3a59\") " pod="openshift-cluster-node-tuning-operator/tuned-9zgvm" Sep 11 09:13:05.994961 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:05.994931 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3c0627e0-f086-4fc4-b2d7-724b67c685d4-host-run-netns\") pod \"ovnkube-node-wgrlw\" (UID: \"3c0627e0-f086-4fc4-b2d7-724b67c685d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgrlw" Sep 11 09:13:05.995817 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:05.994965 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 11 09:13:05.995817 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:05.995183 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 11 09:13:05.995817 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:05.994972 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3c0627e0-f086-4fc4-b2d7-724b67c685d4-node-log\") pod \"ovnkube-node-wgrlw\" (UID: \"3c0627e0-f086-4fc4-b2d7-724b67c685d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgrlw" Sep 11 09:13:05.995817 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:05.995337 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fcb16b38-81cb-41f7-81d2-730c64ee3a59-var-lib-kubelet\") pod \"tuned-9zgvm\" (UID: \"fcb16b38-81cb-41f7-81d2-730c64ee3a59\") " pod="openshift-cluster-node-tuning-operator/tuned-9zgvm" Sep 11 09:13:05.995817 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:05.995365 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3c0627e0-f086-4fc4-b2d7-724b67c685d4-var-lib-openvswitch\") pod \"ovnkube-node-wgrlw\" (UID: \"3c0627e0-f086-4fc4-b2d7-724b67c685d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgrlw" Sep 11 09:13:05.995817 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:05.995391 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3c0627e0-f086-4fc4-b2d7-724b67c685d4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wgrlw\" (UID: \"3c0627e0-f086-4fc4-b2d7-724b67c685d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgrlw" Sep 11 09:13:05.995817 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:05.995416 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cm5xp\" (UniqueName: \"kubernetes.io/projected/3c0627e0-f086-4fc4-b2d7-724b67c685d4-kube-api-access-cm5xp\") pod \"ovnkube-node-wgrlw\" (UID: \"3c0627e0-f086-4fc4-b2d7-724b67c685d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgrlw" Sep 11 09:13:05.995817 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:05.995442 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fcb16b38-81cb-41f7-81d2-730c64ee3a59-etc-sysctl-d\") pod \"tuned-9zgvm\" (UID: \"fcb16b38-81cb-41f7-81d2-730c64ee3a59\") " pod="openshift-cluster-node-tuning-operator/tuned-9zgvm" Sep 11 09:13:05.995817 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:05.995488 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fcb16b38-81cb-41f7-81d2-730c64ee3a59-etc-systemd\") pod \"tuned-9zgvm\" (UID: \"fcb16b38-81cb-41f7-81d2-730c64ee3a59\") " pod="openshift-cluster-node-tuning-operator/tuned-9zgvm" Sep 11 09:13:05.995817 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:05.995519 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fcb16b38-81cb-41f7-81d2-730c64ee3a59-sys\") pod \"tuned-9zgvm\" (UID: \"fcb16b38-81cb-41f7-81d2-730c64ee3a59\") " pod="openshift-cluster-node-tuning-operator/tuned-9zgvm" Sep 11 09:13:05.995817 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:05.995608 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fcb16b38-81cb-41f7-81d2-730c64ee3a59-lib-modules\") pod \"tuned-9zgvm\" (UID: \"fcb16b38-81cb-41f7-81d2-730c64ee3a59\") " pod="openshift-cluster-node-tuning-operator/tuned-9zgvm" Sep 11 09:13:05.995817 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:05.995632 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fcb16b38-81cb-41f7-81d2-730c64ee3a59-host\") pod \"tuned-9zgvm\" (UID: \"fcb16b38-81cb-41f7-81d2-730c64ee3a59\") " pod="openshift-cluster-node-tuning-operator/tuned-9zgvm" Sep 11 09:13:05.995817 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:05.995655 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/08d48976-3c61-4273-88da-fa30ba4057bc-host\") pod \"node-ca-skn2c\" (UID: \"08d48976-3c61-4273-88da-fa30ba4057bc\") " pod="openshift-image-registry/node-ca-skn2c" Sep 11 09:13:05.995817 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:05.995692 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1f74babf-697e-4234-b22e-f7a18ba4a753-registration-dir\") pod \"aws-ebs-csi-driver-node-l46kr\" (UID: \"1f74babf-697e-4234-b22e-f7a18ba4a753\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l46kr" Sep 11 09:13:05.995817 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:05.995715 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1f74babf-697e-4234-b22e-f7a18ba4a753-sys-fs\") pod \"aws-ebs-csi-driver-node-l46kr\" (UID: \"1f74babf-697e-4234-b22e-f7a18ba4a753\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l46kr" Sep 11 09:13:05.995817 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:05.995737 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fcb16b38-81cb-41f7-81d2-730c64ee3a59-tmp\") pod \"tuned-9zgvm\" (UID: \"fcb16b38-81cb-41f7-81d2-730c64ee3a59\") " pod="openshift-cluster-node-tuning-operator/tuned-9zgvm" Sep 11 09:13:05.995817 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:05.995762 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r98q2\" (UniqueName: \"kubernetes.io/projected/1f74babf-697e-4234-b22e-f7a18ba4a753-kube-api-access-r98q2\") pod \"aws-ebs-csi-driver-node-l46kr\" (UID: \"1f74babf-697e-4234-b22e-f7a18ba4a753\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l46kr" Sep 11 09:13:05.995817 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:05.995812 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-fp7vx\"" Sep 11 09:13:05.996635 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:05.995822 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hh6cm\" (UniqueName: \"kubernetes.io/projected/fcb16b38-81cb-41f7-81d2-730c64ee3a59-kube-api-access-hh6cm\") pod \"tuned-9zgvm\" (UID: \"fcb16b38-81cb-41f7-81d2-730c64ee3a59\") " pod="openshift-cluster-node-tuning-operator/tuned-9zgvm" Sep 11 09:13:05.996635 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:05.996028 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 11 09:13:05.996635 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:05.996271 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bb6x9" Sep 11 09:13:05.996635 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:05.996329 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bb6x9" podUID="92eb1c40-b8d0-428c-955a-c005e197f70a" Sep 11 09:13:05.997196 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:05.997048 2575 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 11 09:13:05.998560 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:05.998009 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-g6t8z" Sep 11 09:13:06.000820 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.000418 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 11 09:13:06.000820 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.000643 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 11 09:13:06.001069 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.000893 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-pq626" Sep 11 09:13:06.001069 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.001002 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-d6n7g\"" Sep 11 09:13:06.002784 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.002743 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-8z7wv" Sep 11 09:13:06.006300 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.006278 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 11 09:13:06.006399 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.006312 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 11 09:13:06.006399 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.006317 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 11 09:13:06.006596 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.006579 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-sw5xm\"" Sep 11 09:13:06.006797 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.006777 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-c4dqx\"" Sep 11 09:13:06.006984 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.006969 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 11 09:13:06.018875 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.018846 2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-kthx4" Sep 11 09:13:06.031390 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.031334 2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-kthx4" Sep 11 09:13:06.082231 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.082205 2575 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 11 09:13:06.096259 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.096224 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3c0627e0-f086-4fc4-b2d7-724b67c685d4-env-overrides\") pod \"ovnkube-node-wgrlw\" (UID: \"3c0627e0-f086-4fc4-b2d7-724b67c685d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgrlw" Sep 11 09:13:06.096423 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.096276 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/27736c8f-eddf-4460-94fd-b211c07ae744-system-cni-dir\") pod \"multus-dvpdv\" (UID: \"27736c8f-eddf-4460-94fd-b211c07ae744\") " pod="openshift-multus/multus-dvpdv" Sep 11 09:13:06.096423 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.096354 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fcb16b38-81cb-41f7-81d2-730c64ee3a59-lib-modules\") pod \"tuned-9zgvm\" (UID: \"fcb16b38-81cb-41f7-81d2-730c64ee3a59\") " pod="openshift-cluster-node-tuning-operator/tuned-9zgvm" Sep 11 09:13:06.096423 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.096381 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/08d48976-3c61-4273-88da-fa30ba4057bc-host\") pod \"node-ca-skn2c\" (UID: \"08d48976-3c61-4273-88da-fa30ba4057bc\") " pod="openshift-image-registry/node-ca-skn2c" Sep 11 09:13:06.096423 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.096400 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/27736c8f-eddf-4460-94fd-b211c07ae744-multus-daemon-config\") pod \"multus-dvpdv\" (UID: \"27736c8f-eddf-4460-94fd-b211c07ae744\") " pod="openshift-multus/multus-dvpdv" Sep 11 09:13:06.096640 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.096421 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/27736c8f-eddf-4460-94fd-b211c07ae744-host-run-multus-certs\") pod \"multus-dvpdv\" (UID: \"27736c8f-eddf-4460-94fd-b211c07ae744\") " pod="openshift-multus/multus-dvpdv" Sep 11 09:13:06.096640 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.096449 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mshgs\" (UniqueName: \"kubernetes.io/projected/65ac2ff5-03f0-4143-871f-061152802913-kube-api-access-mshgs\") pod \"iptables-alerter-5xwxg\" (UID: \"65ac2ff5-03f0-4143-871f-061152802913\") " pod="openshift-network-operator/iptables-alerter-5xwxg" Sep 11 09:13:06.096640 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.096488 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1f74babf-697e-4234-b22e-f7a18ba4a753-sys-fs\") pod \"aws-ebs-csi-driver-node-l46kr\" (UID: \"1f74babf-697e-4234-b22e-f7a18ba4a753\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l46kr" Sep 11 09:13:06.096640 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.096502 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/08d48976-3c61-4273-88da-fa30ba4057bc-host\") pod \"node-ca-skn2c\" (UID: \"08d48976-3c61-4273-88da-fa30ba4057bc\") " pod="openshift-image-registry/node-ca-skn2c" Sep 11 09:13:06.096640 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.096510 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hh6cm\" (UniqueName: \"kubernetes.io/projected/fcb16b38-81cb-41f7-81d2-730c64ee3a59-kube-api-access-hh6cm\") pod \"tuned-9zgvm\" (UID: \"fcb16b38-81cb-41f7-81d2-730c64ee3a59\") " pod="openshift-cluster-node-tuning-operator/tuned-9zgvm" Sep 11 09:13:06.096640 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.096567 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1f74babf-697e-4234-b22e-f7a18ba4a753-sys-fs\") pod \"aws-ebs-csi-driver-node-l46kr\" (UID: \"1f74babf-697e-4234-b22e-f7a18ba4a753\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l46kr" Sep 11 09:13:06.096923 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.096610 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/27736c8f-eddf-4460-94fd-b211c07ae744-host-run-netns\") pod \"multus-dvpdv\" (UID: \"27736c8f-eddf-4460-94fd-b211c07ae744\") " pod="openshift-multus/multus-dvpdv" Sep 11 09:13:06.096923 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.096699 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7eddd8ad-0ac3-421d-9649-95dc4fadfd7b-kubelet-config\") pod \"global-pull-secret-syncer-8ng6h\" (UID: \"7eddd8ad-0ac3-421d-9649-95dc4fadfd7b\") " pod="kube-system/global-pull-secret-syncer-8ng6h" Sep 11 09:13:06.096923 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.096721 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abeb037b-2e12-4485-974d-35280f982f4a-metrics-certs\") pod \"network-metrics-daemon-rlftv\" (UID: \"abeb037b-2e12-4485-974d-35280f982f4a\") " pod="openshift-multus/network-metrics-daemon-rlftv" Sep 11 09:13:06.096923 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.096737 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3c0627e0-f086-4fc4-b2d7-724b67c685d4-host-slash\") pod \"ovnkube-node-wgrlw\" (UID: \"3c0627e0-f086-4fc4-b2d7-724b67c685d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgrlw" Sep 11 09:13:06.096923 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.096752 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3c0627e0-f086-4fc4-b2d7-724b67c685d4-env-overrides\") pod \"ovnkube-node-wgrlw\" (UID: \"3c0627e0-f086-4fc4-b2d7-724b67c685d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgrlw" Sep 11 09:13:06.096923 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.096783 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3c0627e0-f086-4fc4-b2d7-724b67c685d4-run-systemd\") pod \"ovnkube-node-wgrlw\" (UID: \"3c0627e0-f086-4fc4-b2d7-724b67c685d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgrlw" Sep 11 09:13:06.096923 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.096637 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fcb16b38-81cb-41f7-81d2-730c64ee3a59-lib-modules\") pod \"tuned-9zgvm\" (UID: \"fcb16b38-81cb-41f7-81d2-730c64ee3a59\") " pod="openshift-cluster-node-tuning-operator/tuned-9zgvm" Sep 11 09:13:06.096923 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.096756 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3c0627e0-f086-4fc4-b2d7-724b67c685d4-run-systemd\") pod \"ovnkube-node-wgrlw\" (UID: \"3c0627e0-f086-4fc4-b2d7-724b67c685d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgrlw" Sep 11 09:13:06.096923 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.096831 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/27736c8f-eddf-4460-94fd-b211c07ae744-host-run-k8s-cni-cncf-io\") pod \"multus-dvpdv\" (UID: \"27736c8f-eddf-4460-94fd-b211c07ae744\") " pod="openshift-multus/multus-dvpdv" Sep 11 09:13:06.096923 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.096848 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8chkf\" (UniqueName: \"kubernetes.io/projected/27736c8f-eddf-4460-94fd-b211c07ae744-kube-api-access-8chkf\") pod \"multus-dvpdv\" (UID: \"27736c8f-eddf-4460-94fd-b211c07ae744\") " pod="openshift-multus/multus-dvpdv" Sep 11 09:13:06.096923 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.096868 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fcb16b38-81cb-41f7-81d2-730c64ee3a59-run\") pod \"tuned-9zgvm\" (UID: \"fcb16b38-81cb-41f7-81d2-730c64ee3a59\") " pod="openshift-cluster-node-tuning-operator/tuned-9zgvm" Sep 11 09:13:06.096923 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:06.096878 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 09:13:06.096923 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.096894 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3c0627e0-f086-4fc4-b2d7-724b67c685d4-systemd-units\") pod \"ovnkube-node-wgrlw\" (UID: \"3c0627e0-f086-4fc4-b2d7-724b67c685d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgrlw" Sep 11 09:13:06.096923 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.096917 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/27736c8f-eddf-4460-94fd-b211c07ae744-multus-cni-dir\") pod \"multus-dvpdv\" (UID: \"27736c8f-eddf-4460-94fd-b211c07ae744\") " pod="openshift-multus/multus-dvpdv" Sep 11 09:13:06.097612 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:06.096938 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/abeb037b-2e12-4485-974d-35280f982f4a-metrics-certs podName:abeb037b-2e12-4485-974d-35280f982f4a nodeName:}" failed. No retries permitted until 2026-09-11 09:13:06.596916185 +0000 UTC m=+2.937526392 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/abeb037b-2e12-4485-974d-35280f982f4a-metrics-certs") pod "network-metrics-daemon-rlftv" (UID: "abeb037b-2e12-4485-974d-35280f982f4a") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 09:13:06.097612 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.096949 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fcb16b38-81cb-41f7-81d2-730c64ee3a59-run\") pod \"tuned-9zgvm\" (UID: \"fcb16b38-81cb-41f7-81d2-730c64ee3a59\") " pod="openshift-cluster-node-tuning-operator/tuned-9zgvm" Sep 11 09:13:06.097612 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.096965 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3c0627e0-f086-4fc4-b2d7-724b67c685d4-systemd-units\") pod \"ovnkube-node-wgrlw\" (UID: \"3c0627e0-f086-4fc4-b2d7-724b67c685d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgrlw" Sep 11 09:13:06.097612 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.096969 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3c0627e0-f086-4fc4-b2d7-724b67c685d4-host-slash\") pod \"ovnkube-node-wgrlw\" (UID: \"3c0627e0-f086-4fc4-b2d7-724b67c685d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgrlw" Sep 11 09:13:06.097612 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.096995 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fc69178e-852e-4853-bb53-01b784cd8a3a-konnectivity-ca\") pod \"konnectivity-agent-pq626\" (UID: \"fc69178e-852e-4853-bb53-01b784cd8a3a\") " pod="kube-system/konnectivity-agent-pq626" Sep 11 09:13:06.097612 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.097016 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/34c95ff5-365b-414d-9968-2db918292b7f-cnibin\") pod \"multus-additional-cni-plugins-8z7wv\" (UID: \"34c95ff5-365b-414d-9968-2db918292b7f\") " pod="openshift-multus/multus-additional-cni-plugins-8z7wv" Sep 11 09:13:06.097612 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.097031 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fcb16b38-81cb-41f7-81d2-730c64ee3a59-etc-tuned\") pod \"tuned-9zgvm\" (UID: \"fcb16b38-81cb-41f7-81d2-730c64ee3a59\") " pod="openshift-cluster-node-tuning-operator/tuned-9zgvm" Sep 11 09:13:06.097612 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.097050 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1f74babf-697e-4234-b22e-f7a18ba4a753-etc-selinux\") pod \"aws-ebs-csi-driver-node-l46kr\" (UID: \"1f74babf-697e-4234-b22e-f7a18ba4a753\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l46kr" Sep 11 09:13:06.097612 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.097074 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1f74babf-697e-4234-b22e-f7a18ba4a753-kubelet-dir\") pod \"aws-ebs-csi-driver-node-l46kr\" (UID: \"1f74babf-697e-4234-b22e-f7a18ba4a753\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l46kr" Sep 11 09:13:06.097612 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.097096 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fcb16b38-81cb-41f7-81d2-730c64ee3a59-etc-sysconfig\") pod \"tuned-9zgvm\" (UID: \"fcb16b38-81cb-41f7-81d2-730c64ee3a59\") " pod="openshift-cluster-node-tuning-operator/tuned-9zgvm" Sep 11 09:13:06.097612 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.097121 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1f74babf-697e-4234-b22e-f7a18ba4a753-etc-selinux\") pod \"aws-ebs-csi-driver-node-l46kr\" (UID: \"1f74babf-697e-4234-b22e-f7a18ba4a753\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l46kr" Sep 11 09:13:06.097612 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.097150 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fcb16b38-81cb-41f7-81d2-730c64ee3a59-etc-sysconfig\") pod \"tuned-9zgvm\" (UID: \"fcb16b38-81cb-41f7-81d2-730c64ee3a59\") " pod="openshift-cluster-node-tuning-operator/tuned-9zgvm" Sep 11 09:13:06.097612 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.097171 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1f74babf-697e-4234-b22e-f7a18ba4a753-kubelet-dir\") pod \"aws-ebs-csi-driver-node-l46kr\" (UID: \"1f74babf-697e-4234-b22e-f7a18ba4a753\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l46kr" Sep 11 09:13:06.097612 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.097195 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3c0627e0-f086-4fc4-b2d7-724b67c685d4-etc-openvswitch\") pod \"ovnkube-node-wgrlw\" (UID: \"3c0627e0-f086-4fc4-b2d7-724b67c685d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgrlw" Sep 11 09:13:06.097612 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.097244 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3c0627e0-f086-4fc4-b2d7-724b67c685d4-run-ovn\") pod \"ovnkube-node-wgrlw\" (UID: \"3c0627e0-f086-4fc4-b2d7-724b67c685d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgrlw" Sep 11 09:13:06.097612 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.097266 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3c0627e0-f086-4fc4-b2d7-724b67c685d4-host-cni-netd\") pod \"ovnkube-node-wgrlw\" (UID: \"3c0627e0-f086-4fc4-b2d7-724b67c685d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgrlw" Sep 11 09:13:06.098357 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.097299 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3c0627e0-f086-4fc4-b2d7-724b67c685d4-etc-openvswitch\") pod \"ovnkube-node-wgrlw\" (UID: \"3c0627e0-f086-4fc4-b2d7-724b67c685d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgrlw" Sep 11 09:13:06.098357 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.097307 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3c0627e0-f086-4fc4-b2d7-724b67c685d4-run-ovn\") pod \"ovnkube-node-wgrlw\" (UID: \"3c0627e0-f086-4fc4-b2d7-724b67c685d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgrlw" Sep 11 09:13:06.098357 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.097329 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3c0627e0-f086-4fc4-b2d7-724b67c685d4-ovn-node-metrics-cert\") pod \"ovnkube-node-wgrlw\" (UID: \"3c0627e0-f086-4fc4-b2d7-724b67c685d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgrlw" Sep 11 09:13:06.098357 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.097304 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3c0627e0-f086-4fc4-b2d7-724b67c685d4-host-cni-netd\") pod \"ovnkube-node-wgrlw\" (UID: \"3c0627e0-f086-4fc4-b2d7-724b67c685d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgrlw" Sep 11 09:13:06.098357 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.097372 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/08d48976-3c61-4273-88da-fa30ba4057bc-serviceca\") pod \"node-ca-skn2c\" (UID: \"08d48976-3c61-4273-88da-fa30ba4057bc\") " pod="openshift-image-registry/node-ca-skn2c" Sep 11 09:13:06.098357 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.097393 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3c0627e0-f086-4fc4-b2d7-724b67c685d4-log-socket\") pod \"ovnkube-node-wgrlw\" (UID: \"3c0627e0-f086-4fc4-b2d7-724b67c685d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgrlw" Sep 11 09:13:06.098357 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.097414 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3c0627e0-f086-4fc4-b2d7-724b67c685d4-ovnkube-script-lib\") pod \"ovnkube-node-wgrlw\" (UID: \"3c0627e0-f086-4fc4-b2d7-724b67c685d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgrlw" Sep 11 09:13:06.098357 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.097441 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/27736c8f-eddf-4460-94fd-b211c07ae744-cni-binary-copy\") pod \"multus-dvpdv\" (UID: \"27736c8f-eddf-4460-94fd-b211c07ae744\") " pod="openshift-multus/multus-dvpdv" Sep 11 09:13:06.098357 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.097479 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/27736c8f-eddf-4460-94fd-b211c07ae744-host-var-lib-cni-bin\") pod \"multus-dvpdv\" (UID: \"27736c8f-eddf-4460-94fd-b211c07ae744\") " pod="openshift-multus/multus-dvpdv" Sep 11 09:13:06.098357 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.097506 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/27736c8f-eddf-4460-94fd-b211c07ae744-multus-conf-dir\") pod \"multus-dvpdv\" (UID: \"27736c8f-eddf-4460-94fd-b211c07ae744\") " pod="openshift-multus/multus-dvpdv" Sep 11 09:13:06.098357 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.097506 2575 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 11 09:13:06.098357 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.097528 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7eddd8ad-0ac3-421d-9649-95dc4fadfd7b-dbus\") pod \"global-pull-secret-syncer-8ng6h\" (UID: \"7eddd8ad-0ac3-421d-9649-95dc4fadfd7b\") " pod="kube-system/global-pull-secret-syncer-8ng6h" Sep 11 09:13:06.098357 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.097558 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/65ac2ff5-03f0-4143-871f-061152802913-iptables-alerter-script\") pod \"iptables-alerter-5xwxg\" (UID: \"65ac2ff5-03f0-4143-871f-061152802913\") " pod="openshift-network-operator/iptables-alerter-5xwxg" Sep 11 09:13:06.098357 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.097583 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1f74babf-697e-4234-b22e-f7a18ba4a753-device-dir\") pod \"aws-ebs-csi-driver-node-l46kr\" (UID: \"1f74babf-697e-4234-b22e-f7a18ba4a753\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l46kr" Sep 11 09:13:06.098357 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.097604 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/27736c8f-eddf-4460-94fd-b211c07ae744-host-var-lib-kubelet\") pod \"multus-dvpdv\" (UID: \"27736c8f-eddf-4460-94fd-b211c07ae744\") " pod="openshift-multus/multus-dvpdv" Sep 11 09:13:06.098357 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.097629 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8vct\" (UniqueName: \"kubernetes.io/projected/34c95ff5-365b-414d-9968-2db918292b7f-kube-api-access-m8vct\") pod \"multus-additional-cni-plugins-8z7wv\" (UID: \"34c95ff5-365b-414d-9968-2db918292b7f\") " pod="openshift-multus/multus-additional-cni-plugins-8z7wv" Sep 11 09:13:06.098357 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.097652 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fcb16b38-81cb-41f7-81d2-730c64ee3a59-var-lib-kubelet\") pod \"tuned-9zgvm\" (UID: \"fcb16b38-81cb-41f7-81d2-730c64ee3a59\") " pod="openshift-cluster-node-tuning-operator/tuned-9zgvm" Sep 11 09:13:06.099176 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.097690 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/27736c8f-eddf-4460-94fd-b211c07ae744-cnibin\") pod \"multus-dvpdv\" (UID: \"27736c8f-eddf-4460-94fd-b211c07ae744\") " pod="openshift-multus/multus-dvpdv" Sep 11 09:13:06.099176 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.097711 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slfls\" (UniqueName: \"kubernetes.io/projected/92eb1c40-b8d0-428c-955a-c005e197f70a-kube-api-access-slfls\") pod \"network-check-target-bb6x9\" (UID: \"92eb1c40-b8d0-428c-955a-c005e197f70a\") " pod="openshift-network-diagnostics/network-check-target-bb6x9" Sep 11 09:13:06.099176 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.097741 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r98q2\" (UniqueName: \"kubernetes.io/projected/1f74babf-697e-4234-b22e-f7a18ba4a753-kube-api-access-r98q2\") pod \"aws-ebs-csi-driver-node-l46kr\" (UID: \"1f74babf-697e-4234-b22e-f7a18ba4a753\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l46kr" Sep 11 09:13:06.099176 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.097765 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/27736c8f-eddf-4460-94fd-b211c07ae744-multus-socket-dir-parent\") pod \"multus-dvpdv\" (UID: \"27736c8f-eddf-4460-94fd-b211c07ae744\") " pod="openshift-multus/multus-dvpdv" Sep 11 09:13:06.099176 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.097788 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fcb16b38-81cb-41f7-81d2-730c64ee3a59-etc-sysctl-d\") pod \"tuned-9zgvm\" (UID: \"fcb16b38-81cb-41f7-81d2-730c64ee3a59\") " pod="openshift-cluster-node-tuning-operator/tuned-9zgvm" Sep 11 09:13:06.099176 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.097806 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/08d48976-3c61-4273-88da-fa30ba4057bc-serviceca\") pod \"node-ca-skn2c\" (UID: \"08d48976-3c61-4273-88da-fa30ba4057bc\") " pod="openshift-image-registry/node-ca-skn2c" Sep 11 09:13:06.099176 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.097879 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fcb16b38-81cb-41f7-81d2-730c64ee3a59-etc-systemd\") pod \"tuned-9zgvm\" (UID: \"fcb16b38-81cb-41f7-81d2-730c64ee3a59\") " pod="openshift-cluster-node-tuning-operator/tuned-9zgvm" Sep 11 09:13:06.099176 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.097810 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fcb16b38-81cb-41f7-81d2-730c64ee3a59-etc-systemd\") pod \"tuned-9zgvm\" (UID: \"fcb16b38-81cb-41f7-81d2-730c64ee3a59\") " pod="openshift-cluster-node-tuning-operator/tuned-9zgvm" Sep 11 09:13:06.099176 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.097917 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3c0627e0-f086-4fc4-b2d7-724b67c685d4-ovnkube-script-lib\") pod \"ovnkube-node-wgrlw\" (UID: \"3c0627e0-f086-4fc4-b2d7-724b67c685d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgrlw" Sep 11 09:13:06.099176 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.097922 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fcb16b38-81cb-41f7-81d2-730c64ee3a59-sys\") pod \"tuned-9zgvm\" (UID: \"fcb16b38-81cb-41f7-81d2-730c64ee3a59\") " pod="openshift-cluster-node-tuning-operator/tuned-9zgvm" Sep 11 09:13:06.099176 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.097958 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fcb16b38-81cb-41f7-81d2-730c64ee3a59-sys\") pod \"tuned-9zgvm\" (UID: \"fcb16b38-81cb-41f7-81d2-730c64ee3a59\") " pod="openshift-cluster-node-tuning-operator/tuned-9zgvm" Sep 11 09:13:06.099176 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.097961 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fcb16b38-81cb-41f7-81d2-730c64ee3a59-host\") pod \"tuned-9zgvm\" (UID: \"fcb16b38-81cb-41f7-81d2-730c64ee3a59\") " pod="openshift-cluster-node-tuning-operator/tuned-9zgvm" Sep 11 09:13:06.099176 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.098009 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1f74babf-697e-4234-b22e-f7a18ba4a753-registration-dir\") pod \"aws-ebs-csi-driver-node-l46kr\" (UID: \"1f74babf-697e-4234-b22e-f7a18ba4a753\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l46kr" Sep 11 09:13:06.099176 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.098016 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1f74babf-697e-4234-b22e-f7a18ba4a753-device-dir\") pod \"aws-ebs-csi-driver-node-l46kr\" (UID: \"1f74babf-697e-4234-b22e-f7a18ba4a753\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l46kr" Sep 11 09:13:06.099176 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.098033 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fcb16b38-81cb-41f7-81d2-730c64ee3a59-tmp\") pod \"tuned-9zgvm\" (UID: \"fcb16b38-81cb-41f7-81d2-730c64ee3a59\") " pod="openshift-cluster-node-tuning-operator/tuned-9zgvm" Sep 11 09:13:06.099176 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.098060 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fcb16b38-81cb-41f7-81d2-730c64ee3a59-var-lib-kubelet\") pod \"tuned-9zgvm\" (UID: \"fcb16b38-81cb-41f7-81d2-730c64ee3a59\") " pod="openshift-cluster-node-tuning-operator/tuned-9zgvm" Sep 11 09:13:06.099176 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.098061 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3c0627e0-f086-4fc4-b2d7-724b67c685d4-run-openvswitch\") pod \"ovnkube-node-wgrlw\" (UID: \"3c0627e0-f086-4fc4-b2d7-724b67c685d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgrlw" Sep 11 09:13:06.099920 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.098128 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/27736c8f-eddf-4460-94fd-b211c07ae744-hostroot\") pod \"multus-dvpdv\" (UID: \"27736c8f-eddf-4460-94fd-b211c07ae744\") " pod="openshift-multus/multus-dvpdv" Sep 11 09:13:06.099920 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.098149 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3c0627e0-f086-4fc4-b2d7-724b67c685d4-run-openvswitch\") pod \"ovnkube-node-wgrlw\" (UID: \"3c0627e0-f086-4fc4-b2d7-724b67c685d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgrlw" Sep 11 09:13:06.099920 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.098152 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/27736c8f-eddf-4460-94fd-b211c07ae744-etc-kubernetes\") pod \"multus-dvpdv\" (UID: \"27736c8f-eddf-4460-94fd-b211c07ae744\") " pod="openshift-multus/multus-dvpdv" Sep 11 09:13:06.099920 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.098187 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdzgt\" (UniqueName: \"kubernetes.io/projected/5a29489a-506a-4070-b30d-7366366058dc-kube-api-access-qdzgt\") pod \"node-resolver-g6t8z\" (UID: \"5a29489a-506a-4070-b30d-7366366058dc\") " pod="openshift-dns/node-resolver-g6t8z" Sep 11 09:13:06.099920 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.098194 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1f74babf-697e-4234-b22e-f7a18ba4a753-registration-dir\") pod \"aws-ebs-csi-driver-node-l46kr\" (UID: \"1f74babf-697e-4234-b22e-f7a18ba4a753\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l46kr" Sep 11 09:13:06.099920 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.098223 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fc69178e-852e-4853-bb53-01b784cd8a3a-agent-certs\") pod \"konnectivity-agent-pq626\" (UID: \"fc69178e-852e-4853-bb53-01b784cd8a3a\") " pod="kube-system/konnectivity-agent-pq626" Sep 11 09:13:06.099920 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.098249 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/34c95ff5-365b-414d-9968-2db918292b7f-system-cni-dir\") pod \"multus-additional-cni-plugins-8z7wv\" (UID: \"34c95ff5-365b-414d-9968-2db918292b7f\") " pod="openshift-multus/multus-additional-cni-plugins-8z7wv" Sep 11 09:13:06.099920 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.098273 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/34c95ff5-365b-414d-9968-2db918292b7f-cni-binary-copy\") pod \"multus-additional-cni-plugins-8z7wv\" (UID: \"34c95ff5-365b-414d-9968-2db918292b7f\") " pod="openshift-multus/multus-additional-cni-plugins-8z7wv" Sep 11 09:13:06.099920 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.098279 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fcb16b38-81cb-41f7-81d2-730c64ee3a59-etc-sysctl-d\") pod \"tuned-9zgvm\" (UID: \"fcb16b38-81cb-41f7-81d2-730c64ee3a59\") " pod="openshift-cluster-node-tuning-operator/tuned-9zgvm" Sep 11 09:13:06.099920 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.098302 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hd6lj\" (UniqueName: \"kubernetes.io/projected/08d48976-3c61-4273-88da-fa30ba4057bc-kube-api-access-hd6lj\") pod \"node-ca-skn2c\" (UID: \"08d48976-3c61-4273-88da-fa30ba4057bc\") " pod="openshift-image-registry/node-ca-skn2c" Sep 11 09:13:06.099920 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.098327 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fcb16b38-81cb-41f7-81d2-730c64ee3a59-etc-modprobe-d\") pod \"tuned-9zgvm\" (UID: \"fcb16b38-81cb-41f7-81d2-730c64ee3a59\") " pod="openshift-cluster-node-tuning-operator/tuned-9zgvm" Sep 11 09:13:06.099920 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.098322 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3c0627e0-f086-4fc4-b2d7-724b67c685d4-log-socket\") pod \"ovnkube-node-wgrlw\" (UID: \"3c0627e0-f086-4fc4-b2d7-724b67c685d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgrlw" Sep 11 09:13:06.099920 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.098357 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3c0627e0-f086-4fc4-b2d7-724b67c685d4-ovnkube-config\") pod \"ovnkube-node-wgrlw\" (UID: \"3c0627e0-f086-4fc4-b2d7-724b67c685d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgrlw" Sep 11 09:13:06.099920 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.098359 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fcb16b38-81cb-41f7-81d2-730c64ee3a59-host\") pod \"tuned-9zgvm\" (UID: \"fcb16b38-81cb-41f7-81d2-730c64ee3a59\") " pod="openshift-cluster-node-tuning-operator/tuned-9zgvm" Sep 11 09:13:06.099920 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.098384 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/65ac2ff5-03f0-4143-871f-061152802913-host-slash\") pod \"iptables-alerter-5xwxg\" (UID: \"65ac2ff5-03f0-4143-871f-061152802913\") " pod="openshift-network-operator/iptables-alerter-5xwxg" Sep 11 09:13:06.099920 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.098454 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/34c95ff5-365b-414d-9968-2db918292b7f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8z7wv\" (UID: \"34c95ff5-365b-414d-9968-2db918292b7f\") " pod="openshift-multus/multus-additional-cni-plugins-8z7wv" Sep 11 09:13:06.099920 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.098531 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fcb16b38-81cb-41f7-81d2-730c64ee3a59-etc-modprobe-d\") pod \"tuned-9zgvm\" (UID: \"fcb16b38-81cb-41f7-81d2-730c64ee3a59\") " pod="openshift-cluster-node-tuning-operator/tuned-9zgvm" Sep 11 09:13:06.100697 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.098584 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fcb16b38-81cb-41f7-81d2-730c64ee3a59-etc-sysctl-conf\") pod \"tuned-9zgvm\" (UID: \"fcb16b38-81cb-41f7-81d2-730c64ee3a59\") " pod="openshift-cluster-node-tuning-operator/tuned-9zgvm" Sep 11 09:13:06.100697 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.098626 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1f74babf-697e-4234-b22e-f7a18ba4a753-socket-dir\") pod \"aws-ebs-csi-driver-node-l46kr\" (UID: \"1f74babf-697e-4234-b22e-f7a18ba4a753\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l46kr" Sep 11 09:13:06.100697 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.098665 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3c0627e0-f086-4fc4-b2d7-724b67c685d4-host-kubelet\") pod \"ovnkube-node-wgrlw\" (UID: \"3c0627e0-f086-4fc4-b2d7-724b67c685d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgrlw" Sep 11 09:13:06.100697 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.098681 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fcb16b38-81cb-41f7-81d2-730c64ee3a59-etc-sysctl-conf\") pod \"tuned-9zgvm\" (UID: \"fcb16b38-81cb-41f7-81d2-730c64ee3a59\") " pod="openshift-cluster-node-tuning-operator/tuned-9zgvm" Sep 11 09:13:06.100697 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.098692 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3c0627e0-f086-4fc4-b2d7-724b67c685d4-host-run-ovn-kubernetes\") pod \"ovnkube-node-wgrlw\" (UID: \"3c0627e0-f086-4fc4-b2d7-724b67c685d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgrlw" Sep 11 09:13:06.100697 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.098721 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cm5xp\" (UniqueName: \"kubernetes.io/projected/3c0627e0-f086-4fc4-b2d7-724b67c685d4-kube-api-access-cm5xp\") pod \"ovnkube-node-wgrlw\" (UID: \"3c0627e0-f086-4fc4-b2d7-724b67c685d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgrlw" Sep 11 09:13:06.100697 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.098729 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3c0627e0-f086-4fc4-b2d7-724b67c685d4-host-kubelet\") pod \"ovnkube-node-wgrlw\" (UID: \"3c0627e0-f086-4fc4-b2d7-724b67c685d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgrlw" Sep 11 09:13:06.100697 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.098750 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a29489a-506a-4070-b30d-7366366058dc-tmp-dir\") pod \"node-resolver-g6t8z\" (UID: \"5a29489a-506a-4070-b30d-7366366058dc\") " pod="openshift-dns/node-resolver-g6t8z" Sep 11 09:13:06.100697 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.098836 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3c0627e0-f086-4fc4-b2d7-724b67c685d4-ovnkube-config\") pod \"ovnkube-node-wgrlw\" (UID: \"3c0627e0-f086-4fc4-b2d7-724b67c685d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgrlw" Sep 11 09:13:06.100697 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.098860 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1f74babf-697e-4234-b22e-f7a18ba4a753-socket-dir\") pod \"aws-ebs-csi-driver-node-l46kr\" (UID: \"1f74babf-697e-4234-b22e-f7a18ba4a753\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l46kr" Sep 11 09:13:06.100697 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.098876 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/34c95ff5-365b-414d-9968-2db918292b7f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8z7wv\" (UID: \"34c95ff5-365b-414d-9968-2db918292b7f\") " pod="openshift-multus/multus-additional-cni-plugins-8z7wv" Sep 11 09:13:06.100697 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.098905 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3c0627e0-f086-4fc4-b2d7-724b67c685d4-host-run-ovn-kubernetes\") pod \"ovnkube-node-wgrlw\" (UID: \"3c0627e0-f086-4fc4-b2d7-724b67c685d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgrlw" Sep 11 09:13:06.100697 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.098946 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3c0627e0-f086-4fc4-b2d7-724b67c685d4-host-cni-bin\") pod \"ovnkube-node-wgrlw\" (UID: \"3c0627e0-f086-4fc4-b2d7-724b67c685d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgrlw" Sep 11 09:13:06.100697 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.098988 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/34c95ff5-365b-414d-9968-2db918292b7f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8z7wv\" (UID: \"34c95ff5-365b-414d-9968-2db918292b7f\") " pod="openshift-multus/multus-additional-cni-plugins-8z7wv" Sep 11 09:13:06.100697 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.099009 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3c0627e0-f086-4fc4-b2d7-724b67c685d4-host-cni-bin\") pod \"ovnkube-node-wgrlw\" (UID: \"3c0627e0-f086-4fc4-b2d7-724b67c685d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgrlw" Sep 11 09:13:06.100697 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.099048 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvd2w\" (UniqueName: \"kubernetes.io/projected/abeb037b-2e12-4485-974d-35280f982f4a-kube-api-access-nvd2w\") pod \"network-metrics-daemon-rlftv\" (UID: \"abeb037b-2e12-4485-974d-35280f982f4a\") " pod="openshift-multus/network-metrics-daemon-rlftv" Sep 11 09:13:06.100697 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.099076 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fcb16b38-81cb-41f7-81d2-730c64ee3a59-etc-kubernetes\") pod \"tuned-9zgvm\" (UID: \"fcb16b38-81cb-41f7-81d2-730c64ee3a59\") " pod="openshift-cluster-node-tuning-operator/tuned-9zgvm" Sep 11 09:13:06.101394 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.099124 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3c0627e0-f086-4fc4-b2d7-724b67c685d4-host-run-netns\") pod \"ovnkube-node-wgrlw\" (UID: \"3c0627e0-f086-4fc4-b2d7-724b67c685d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgrlw" Sep 11 09:13:06.101394 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.099172 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fcb16b38-81cb-41f7-81d2-730c64ee3a59-etc-kubernetes\") pod \"tuned-9zgvm\" (UID: \"fcb16b38-81cb-41f7-81d2-730c64ee3a59\") " pod="openshift-cluster-node-tuning-operator/tuned-9zgvm" Sep 11 09:13:06.101394 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.099185 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3c0627e0-f086-4fc4-b2d7-724b67c685d4-node-log\") pod \"ovnkube-node-wgrlw\" (UID: \"3c0627e0-f086-4fc4-b2d7-724b67c685d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgrlw" Sep 11 09:13:06.101394 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.099210 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3c0627e0-f086-4fc4-b2d7-724b67c685d4-host-run-netns\") pod \"ovnkube-node-wgrlw\" (UID: \"3c0627e0-f086-4fc4-b2d7-724b67c685d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgrlw" Sep 11 09:13:06.101394 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.099221 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7eddd8ad-0ac3-421d-9649-95dc4fadfd7b-original-pull-secret\") pod \"global-pull-secret-syncer-8ng6h\" (UID: \"7eddd8ad-0ac3-421d-9649-95dc4fadfd7b\") " pod="kube-system/global-pull-secret-syncer-8ng6h" Sep 11 09:13:06.101394 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.099252 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3c0627e0-f086-4fc4-b2d7-724b67c685d4-node-log\") pod \"ovnkube-node-wgrlw\" (UID: \"3c0627e0-f086-4fc4-b2d7-724b67c685d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgrlw" Sep 11 09:13:06.101394 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.099251 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3c0627e0-f086-4fc4-b2d7-724b67c685d4-var-lib-openvswitch\") pod \"ovnkube-node-wgrlw\" (UID: \"3c0627e0-f086-4fc4-b2d7-724b67c685d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgrlw" Sep 11 09:13:06.101394 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.099288 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3c0627e0-f086-4fc4-b2d7-724b67c685d4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wgrlw\" (UID: \"3c0627e0-f086-4fc4-b2d7-724b67c685d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgrlw" Sep 11 09:13:06.101394 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.099292 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3c0627e0-f086-4fc4-b2d7-724b67c685d4-var-lib-openvswitch\") pod \"ovnkube-node-wgrlw\" (UID: \"3c0627e0-f086-4fc4-b2d7-724b67c685d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgrlw" Sep 11 09:13:06.101394 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.099326 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/27736c8f-eddf-4460-94fd-b211c07ae744-host-var-lib-cni-multus\") pod \"multus-dvpdv\" (UID: \"27736c8f-eddf-4460-94fd-b211c07ae744\") " pod="openshift-multus/multus-dvpdv" Sep 11 09:13:06.101394 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.099356 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3c0627e0-f086-4fc4-b2d7-724b67c685d4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wgrlw\" (UID: \"3c0627e0-f086-4fc4-b2d7-724b67c685d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgrlw" Sep 11 09:13:06.101394 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.099409 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5a29489a-506a-4070-b30d-7366366058dc-hosts-file\") pod \"node-resolver-g6t8z\" (UID: \"5a29489a-506a-4070-b30d-7366366058dc\") " pod="openshift-dns/node-resolver-g6t8z" Sep 11 09:13:06.102006 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.101679 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fcb16b38-81cb-41f7-81d2-730c64ee3a59-etc-tuned\") pod \"tuned-9zgvm\" (UID: \"fcb16b38-81cb-41f7-81d2-730c64ee3a59\") " pod="openshift-cluster-node-tuning-operator/tuned-9zgvm" Sep 11 09:13:06.102068 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.102050 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fcb16b38-81cb-41f7-81d2-730c64ee3a59-tmp\") pod \"tuned-9zgvm\" (UID: \"fcb16b38-81cb-41f7-81d2-730c64ee3a59\") " pod="openshift-cluster-node-tuning-operator/tuned-9zgvm" Sep 11 09:13:06.102702 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.102573 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3c0627e0-f086-4fc4-b2d7-724b67c685d4-ovn-node-metrics-cert\") pod \"ovnkube-node-wgrlw\" (UID: \"3c0627e0-f086-4fc4-b2d7-724b67c685d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgrlw" Sep 11 09:13:06.107086 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.107063 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hh6cm\" (UniqueName: \"kubernetes.io/projected/fcb16b38-81cb-41f7-81d2-730c64ee3a59-kube-api-access-hh6cm\") pod \"tuned-9zgvm\" (UID: \"fcb16b38-81cb-41f7-81d2-730c64ee3a59\") " pod="openshift-cluster-node-tuning-operator/tuned-9zgvm" Sep 11 09:13:06.111891 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.111864 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cm5xp\" (UniqueName: \"kubernetes.io/projected/3c0627e0-f086-4fc4-b2d7-724b67c685d4-kube-api-access-cm5xp\") pod \"ovnkube-node-wgrlw\" (UID: \"3c0627e0-f086-4fc4-b2d7-724b67c685d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgrlw" Sep 11 09:13:06.116047 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.115981 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvd2w\" (UniqueName: \"kubernetes.io/projected/abeb037b-2e12-4485-974d-35280f982f4a-kube-api-access-nvd2w\") pod \"network-metrics-daemon-rlftv\" (UID: \"abeb037b-2e12-4485-974d-35280f982f4a\") " pod="openshift-multus/network-metrics-daemon-rlftv" Sep 11 09:13:06.116257 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.116235 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hd6lj\" (UniqueName: \"kubernetes.io/projected/08d48976-3c61-4273-88da-fa30ba4057bc-kube-api-access-hd6lj\") pod \"node-ca-skn2c\" (UID: \"08d48976-3c61-4273-88da-fa30ba4057bc\") " pod="openshift-image-registry/node-ca-skn2c" Sep 11 09:13:06.116401 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.116275 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-119.ec2.internal" event={"ID":"ce93b5afb1de6b9a80d64df7da81462d","Type":"ContainerStarted","Data":"7ed018af667b5a2b7a4977308558031db76ff64d944c1f21bb15bb7a251da263"} Sep 11 09:13:06.117300 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.117247 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r98q2\" (UniqueName: \"kubernetes.io/projected/1f74babf-697e-4234-b22e-f7a18ba4a753-kube-api-access-r98q2\") pod \"aws-ebs-csi-driver-node-l46kr\" (UID: \"1f74babf-697e-4234-b22e-f7a18ba4a753\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l46kr" Sep 11 09:13:06.118671 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.118646 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-119.ec2.internal" event={"ID":"db42fcd462e8e18de823d8d0adf7cc0a","Type":"ContainerStarted","Data":"e79c76f9564878a63ab6c97d325effc95f68792d18a395d69a10d1d234b30166"} Sep 11 09:13:06.200388 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.200356 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/27736c8f-eddf-4460-94fd-b211c07ae744-host-var-lib-cni-bin\") pod \"multus-dvpdv\" (UID: \"27736c8f-eddf-4460-94fd-b211c07ae744\") " pod="openshift-multus/multus-dvpdv" Sep 11 09:13:06.200388 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.200390 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/27736c8f-eddf-4460-94fd-b211c07ae744-multus-conf-dir\") pod \"multus-dvpdv\" (UID: \"27736c8f-eddf-4460-94fd-b211c07ae744\") " pod="openshift-multus/multus-dvpdv" Sep 11 09:13:06.200617 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.200412 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7eddd8ad-0ac3-421d-9649-95dc4fadfd7b-dbus\") pod \"global-pull-secret-syncer-8ng6h\" (UID: \"7eddd8ad-0ac3-421d-9649-95dc4fadfd7b\") " pod="kube-system/global-pull-secret-syncer-8ng6h" Sep 11 09:13:06.200617 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.200438 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/65ac2ff5-03f0-4143-871f-061152802913-iptables-alerter-script\") pod \"iptables-alerter-5xwxg\" (UID: \"65ac2ff5-03f0-4143-871f-061152802913\") " pod="openshift-network-operator/iptables-alerter-5xwxg" Sep 11 09:13:06.200617 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.200440 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/27736c8f-eddf-4460-94fd-b211c07ae744-host-var-lib-cni-bin\") pod \"multus-dvpdv\" (UID: \"27736c8f-eddf-4460-94fd-b211c07ae744\") " pod="openshift-multus/multus-dvpdv" Sep 11 09:13:06.200617 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.200463 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/27736c8f-eddf-4460-94fd-b211c07ae744-host-var-lib-kubelet\") pod \"multus-dvpdv\" (UID: \"27736c8f-eddf-4460-94fd-b211c07ae744\") " pod="openshift-multus/multus-dvpdv" Sep 11 09:13:06.200617 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.200443 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/27736c8f-eddf-4460-94fd-b211c07ae744-multus-conf-dir\") pod \"multus-dvpdv\" (UID: \"27736c8f-eddf-4460-94fd-b211c07ae744\") " pod="openshift-multus/multus-dvpdv" Sep 11 09:13:06.200617 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.200499 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8vct\" (UniqueName: \"kubernetes.io/projected/34c95ff5-365b-414d-9968-2db918292b7f-kube-api-access-m8vct\") pod \"multus-additional-cni-plugins-8z7wv\" (UID: \"34c95ff5-365b-414d-9968-2db918292b7f\") " pod="openshift-multus/multus-additional-cni-plugins-8z7wv" Sep 11 09:13:06.200617 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.200524 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/27736c8f-eddf-4460-94fd-b211c07ae744-cnibin\") pod \"multus-dvpdv\" (UID: \"27736c8f-eddf-4460-94fd-b211c07ae744\") " pod="openshift-multus/multus-dvpdv" Sep 11 09:13:06.200617 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.200526 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/27736c8f-eddf-4460-94fd-b211c07ae744-host-var-lib-kubelet\") pod \"multus-dvpdv\" (UID: \"27736c8f-eddf-4460-94fd-b211c07ae744\") " pod="openshift-multus/multus-dvpdv" Sep 11 09:13:06.200617 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.200549 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slfls\" (UniqueName: \"kubernetes.io/projected/92eb1c40-b8d0-428c-955a-c005e197f70a-kube-api-access-slfls\") pod \"network-check-target-bb6x9\" (UID: \"92eb1c40-b8d0-428c-955a-c005e197f70a\") " pod="openshift-network-diagnostics/network-check-target-bb6x9" Sep 11 09:13:06.200617 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.200558 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7eddd8ad-0ac3-421d-9649-95dc4fadfd7b-dbus\") pod \"global-pull-secret-syncer-8ng6h\" (UID: \"7eddd8ad-0ac3-421d-9649-95dc4fadfd7b\") " pod="kube-system/global-pull-secret-syncer-8ng6h" Sep 11 09:13:06.200617 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.200574 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/27736c8f-eddf-4460-94fd-b211c07ae744-multus-socket-dir-parent\") pod \"multus-dvpdv\" (UID: \"27736c8f-eddf-4460-94fd-b211c07ae744\") " pod="openshift-multus/multus-dvpdv" Sep 11 09:13:06.200617 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.200603 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/27736c8f-eddf-4460-94fd-b211c07ae744-hostroot\") pod \"multus-dvpdv\" (UID: \"27736c8f-eddf-4460-94fd-b211c07ae744\") " pod="openshift-multus/multus-dvpdv" Sep 11 09:13:06.200617 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.200623 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/27736c8f-eddf-4460-94fd-b211c07ae744-etc-kubernetes\") pod \"multus-dvpdv\" (UID: \"27736c8f-eddf-4460-94fd-b211c07ae744\") " pod="openshift-multus/multus-dvpdv" Sep 11 09:13:06.201268 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.200624 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/27736c8f-eddf-4460-94fd-b211c07ae744-cnibin\") pod \"multus-dvpdv\" (UID: \"27736c8f-eddf-4460-94fd-b211c07ae744\") " pod="openshift-multus/multus-dvpdv" Sep 11 09:13:06.201268 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.200646 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdzgt\" (UniqueName: \"kubernetes.io/projected/5a29489a-506a-4070-b30d-7366366058dc-kube-api-access-qdzgt\") pod \"node-resolver-g6t8z\" (UID: \"5a29489a-506a-4070-b30d-7366366058dc\") " pod="openshift-dns/node-resolver-g6t8z" Sep 11 09:13:06.201268 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.200670 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fc69178e-852e-4853-bb53-01b784cd8a3a-agent-certs\") pod \"konnectivity-agent-pq626\" (UID: \"fc69178e-852e-4853-bb53-01b784cd8a3a\") " pod="kube-system/konnectivity-agent-pq626" Sep 11 09:13:06.201268 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.200693 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/34c95ff5-365b-414d-9968-2db918292b7f-system-cni-dir\") pod \"multus-additional-cni-plugins-8z7wv\" (UID: \"34c95ff5-365b-414d-9968-2db918292b7f\") " pod="openshift-multus/multus-additional-cni-plugins-8z7wv" Sep 11 09:13:06.201268 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.200701 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/27736c8f-eddf-4460-94fd-b211c07ae744-multus-socket-dir-parent\") pod \"multus-dvpdv\" (UID: \"27736c8f-eddf-4460-94fd-b211c07ae744\") " pod="openshift-multus/multus-dvpdv" Sep 11 09:13:06.201268 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.200717 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/34c95ff5-365b-414d-9968-2db918292b7f-cni-binary-copy\") pod \"multus-additional-cni-plugins-8z7wv\" (UID: \"34c95ff5-365b-414d-9968-2db918292b7f\") " pod="openshift-multus/multus-additional-cni-plugins-8z7wv" Sep 11 09:13:06.201268 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.200745 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/65ac2ff5-03f0-4143-871f-061152802913-host-slash\") pod \"iptables-alerter-5xwxg\" (UID: \"65ac2ff5-03f0-4143-871f-061152802913\") " pod="openshift-network-operator/iptables-alerter-5xwxg" Sep 11 09:13:06.201268 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.200768 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/34c95ff5-365b-414d-9968-2db918292b7f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8z7wv\" (UID: \"34c95ff5-365b-414d-9968-2db918292b7f\") " pod="openshift-multus/multus-additional-cni-plugins-8z7wv" Sep 11 09:13:06.201268 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.200783 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/27736c8f-eddf-4460-94fd-b211c07ae744-hostroot\") pod \"multus-dvpdv\" (UID: \"27736c8f-eddf-4460-94fd-b211c07ae744\") " pod="openshift-multus/multus-dvpdv" Sep 11 09:13:06.201268 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.200795 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a29489a-506a-4070-b30d-7366366058dc-tmp-dir\") pod \"node-resolver-g6t8z\" (UID: \"5a29489a-506a-4070-b30d-7366366058dc\") " pod="openshift-dns/node-resolver-g6t8z" Sep 11 09:13:06.201268 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.200746 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/27736c8f-eddf-4460-94fd-b211c07ae744-etc-kubernetes\") pod \"multus-dvpdv\" (UID: \"27736c8f-eddf-4460-94fd-b211c07ae744\") " pod="openshift-multus/multus-dvpdv" Sep 11 09:13:06.201268 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.200820 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/34c95ff5-365b-414d-9968-2db918292b7f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8z7wv\" (UID: \"34c95ff5-365b-414d-9968-2db918292b7f\") " pod="openshift-multus/multus-additional-cni-plugins-8z7wv" Sep 11 09:13:06.201268 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.200855 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/34c95ff5-365b-414d-9968-2db918292b7f-system-cni-dir\") pod \"multus-additional-cni-plugins-8z7wv\" (UID: \"34c95ff5-365b-414d-9968-2db918292b7f\") " pod="openshift-multus/multus-additional-cni-plugins-8z7wv" Sep 11 09:13:06.201268 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.200848 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/34c95ff5-365b-414d-9968-2db918292b7f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8z7wv\" (UID: \"34c95ff5-365b-414d-9968-2db918292b7f\") " pod="openshift-multus/multus-additional-cni-plugins-8z7wv" Sep 11 09:13:06.201268 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.200915 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/65ac2ff5-03f0-4143-871f-061152802913-host-slash\") pod \"iptables-alerter-5xwxg\" (UID: \"65ac2ff5-03f0-4143-871f-061152802913\") " pod="openshift-network-operator/iptables-alerter-5xwxg" Sep 11 09:13:06.201268 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.201007 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/34c95ff5-365b-414d-9968-2db918292b7f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8z7wv\" (UID: \"34c95ff5-365b-414d-9968-2db918292b7f\") " pod="openshift-multus/multus-additional-cni-plugins-8z7wv" Sep 11 09:13:06.201268 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.201039 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/65ac2ff5-03f0-4143-871f-061152802913-iptables-alerter-script\") pod \"iptables-alerter-5xwxg\" (UID: \"65ac2ff5-03f0-4143-871f-061152802913\") " pod="openshift-network-operator/iptables-alerter-5xwxg" Sep 11 09:13:06.202088 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.201101 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7eddd8ad-0ac3-421d-9649-95dc4fadfd7b-original-pull-secret\") pod \"global-pull-secret-syncer-8ng6h\" (UID: \"7eddd8ad-0ac3-421d-9649-95dc4fadfd7b\") " pod="kube-system/global-pull-secret-syncer-8ng6h" Sep 11 09:13:06.202088 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.201152 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/27736c8f-eddf-4460-94fd-b211c07ae744-host-var-lib-cni-multus\") pod \"multus-dvpdv\" (UID: \"27736c8f-eddf-4460-94fd-b211c07ae744\") " pod="openshift-multus/multus-dvpdv" Sep 11 09:13:06.202088 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.201149 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a29489a-506a-4070-b30d-7366366058dc-tmp-dir\") pod \"node-resolver-g6t8z\" (UID: \"5a29489a-506a-4070-b30d-7366366058dc\") " pod="openshift-dns/node-resolver-g6t8z" Sep 11 09:13:06.202088 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.201222 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/27736c8f-eddf-4460-94fd-b211c07ae744-host-var-lib-cni-multus\") pod \"multus-dvpdv\" (UID: \"27736c8f-eddf-4460-94fd-b211c07ae744\") " pod="openshift-multus/multus-dvpdv" Sep 11 09:13:06.202088 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:06.201221 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 09:13:06.202088 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.201272 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5a29489a-506a-4070-b30d-7366366058dc-hosts-file\") pod \"node-resolver-g6t8z\" (UID: \"5a29489a-506a-4070-b30d-7366366058dc\") " pod="openshift-dns/node-resolver-g6t8z" Sep 11 09:13:06.202088 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:06.201313 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7eddd8ad-0ac3-421d-9649-95dc4fadfd7b-original-pull-secret podName:7eddd8ad-0ac3-421d-9649-95dc4fadfd7b nodeName:}" failed. No retries permitted until 2026-09-11 09:13:06.701292634 +0000 UTC m=+3.041902833 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7eddd8ad-0ac3-421d-9649-95dc4fadfd7b-original-pull-secret") pod "global-pull-secret-syncer-8ng6h" (UID: "7eddd8ad-0ac3-421d-9649-95dc4fadfd7b") : object "kube-system"/"original-pull-secret" not registered Sep 11 09:13:06.202088 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.201333 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/27736c8f-eddf-4460-94fd-b211c07ae744-system-cni-dir\") pod \"multus-dvpdv\" (UID: \"27736c8f-eddf-4460-94fd-b211c07ae744\") " pod="openshift-multus/multus-dvpdv" Sep 11 09:13:06.202088 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.201343 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5a29489a-506a-4070-b30d-7366366058dc-hosts-file\") pod \"node-resolver-g6t8z\" (UID: \"5a29489a-506a-4070-b30d-7366366058dc\") " pod="openshift-dns/node-resolver-g6t8z" Sep 11 09:13:06.202088 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.201363 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/27736c8f-eddf-4460-94fd-b211c07ae744-multus-daemon-config\") pod \"multus-dvpdv\" (UID: \"27736c8f-eddf-4460-94fd-b211c07ae744\") " pod="openshift-multus/multus-dvpdv" Sep 11 09:13:06.202088 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.201388 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/27736c8f-eddf-4460-94fd-b211c07ae744-host-run-multus-certs\") pod \"multus-dvpdv\" (UID: \"27736c8f-eddf-4460-94fd-b211c07ae744\") " pod="openshift-multus/multus-dvpdv" Sep 11 09:13:06.202088 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.201389 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/34c95ff5-365b-414d-9968-2db918292b7f-cni-binary-copy\") pod \"multus-additional-cni-plugins-8z7wv\" (UID: \"34c95ff5-365b-414d-9968-2db918292b7f\") " pod="openshift-multus/multus-additional-cni-plugins-8z7wv" Sep 11 09:13:06.202088 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.201402 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/34c95ff5-365b-414d-9968-2db918292b7f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8z7wv\" (UID: \"34c95ff5-365b-414d-9968-2db918292b7f\") " pod="openshift-multus/multus-additional-cni-plugins-8z7wv" Sep 11 09:13:06.202088 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.201413 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mshgs\" (UniqueName: \"kubernetes.io/projected/65ac2ff5-03f0-4143-871f-061152802913-kube-api-access-mshgs\") pod \"iptables-alerter-5xwxg\" (UID: \"65ac2ff5-03f0-4143-871f-061152802913\") " pod="openshift-network-operator/iptables-alerter-5xwxg" Sep 11 09:13:06.202088 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.201398 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/27736c8f-eddf-4460-94fd-b211c07ae744-system-cni-dir\") pod \"multus-dvpdv\" (UID: \"27736c8f-eddf-4460-94fd-b211c07ae744\") " pod="openshift-multus/multus-dvpdv" Sep 11 09:13:06.202088 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.201440 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/27736c8f-eddf-4460-94fd-b211c07ae744-host-run-multus-certs\") pod \"multus-dvpdv\" (UID: \"27736c8f-eddf-4460-94fd-b211c07ae744\") " pod="openshift-multus/multus-dvpdv" Sep 11 09:13:06.202088 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.201526 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/27736c8f-eddf-4460-94fd-b211c07ae744-host-run-netns\") pod \"multus-dvpdv\" (UID: \"27736c8f-eddf-4460-94fd-b211c07ae744\") " pod="openshift-multus/multus-dvpdv" Sep 11 09:13:06.202810 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.201556 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7eddd8ad-0ac3-421d-9649-95dc4fadfd7b-kubelet-config\") pod \"global-pull-secret-syncer-8ng6h\" (UID: \"7eddd8ad-0ac3-421d-9649-95dc4fadfd7b\") " pod="kube-system/global-pull-secret-syncer-8ng6h" Sep 11 09:13:06.202810 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.201596 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/27736c8f-eddf-4460-94fd-b211c07ae744-host-run-k8s-cni-cncf-io\") pod \"multus-dvpdv\" (UID: \"27736c8f-eddf-4460-94fd-b211c07ae744\") " pod="openshift-multus/multus-dvpdv" Sep 11 09:13:06.202810 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.201625 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8chkf\" (UniqueName: \"kubernetes.io/projected/27736c8f-eddf-4460-94fd-b211c07ae744-kube-api-access-8chkf\") pod \"multus-dvpdv\" (UID: \"27736c8f-eddf-4460-94fd-b211c07ae744\") " pod="openshift-multus/multus-dvpdv" Sep 11 09:13:06.202810 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.201652 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/27736c8f-eddf-4460-94fd-b211c07ae744-multus-cni-dir\") pod \"multus-dvpdv\" (UID: \"27736c8f-eddf-4460-94fd-b211c07ae744\") " pod="openshift-multus/multus-dvpdv" Sep 11 09:13:06.202810 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.201676 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fc69178e-852e-4853-bb53-01b784cd8a3a-konnectivity-ca\") pod \"konnectivity-agent-pq626\" (UID: \"fc69178e-852e-4853-bb53-01b784cd8a3a\") " pod="kube-system/konnectivity-agent-pq626" Sep 11 09:13:06.202810 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.201701 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/34c95ff5-365b-414d-9968-2db918292b7f-cnibin\") pod \"multus-additional-cni-plugins-8z7wv\" (UID: \"34c95ff5-365b-414d-9968-2db918292b7f\") " pod="openshift-multus/multus-additional-cni-plugins-8z7wv" Sep 11 09:13:06.202810 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.201734 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/27736c8f-eddf-4460-94fd-b211c07ae744-host-run-k8s-cni-cncf-io\") pod \"multus-dvpdv\" (UID: \"27736c8f-eddf-4460-94fd-b211c07ae744\") " pod="openshift-multus/multus-dvpdv" Sep 11 09:13:06.202810 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.201739 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/27736c8f-eddf-4460-94fd-b211c07ae744-cni-binary-copy\") pod \"multus-dvpdv\" (UID: \"27736c8f-eddf-4460-94fd-b211c07ae744\") " pod="openshift-multus/multus-dvpdv" Sep 11 09:13:06.202810 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.201839 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/27736c8f-eddf-4460-94fd-b211c07ae744-host-run-netns\") pod \"multus-dvpdv\" (UID: \"27736c8f-eddf-4460-94fd-b211c07ae744\") " pod="openshift-multus/multus-dvpdv" Sep 11 09:13:06.202810 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.201891 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7eddd8ad-0ac3-421d-9649-95dc4fadfd7b-kubelet-config\") pod \"global-pull-secret-syncer-8ng6h\" (UID: \"7eddd8ad-0ac3-421d-9649-95dc4fadfd7b\") " pod="kube-system/global-pull-secret-syncer-8ng6h" Sep 11 09:13:06.202810 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.201912 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/27736c8f-eddf-4460-94fd-b211c07ae744-multus-daemon-config\") pod \"multus-dvpdv\" (UID: \"27736c8f-eddf-4460-94fd-b211c07ae744\") " pod="openshift-multus/multus-dvpdv" Sep 11 09:13:06.202810 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.201974 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/27736c8f-eddf-4460-94fd-b211c07ae744-multus-cni-dir\") pod \"multus-dvpdv\" (UID: \"27736c8f-eddf-4460-94fd-b211c07ae744\") " pod="openshift-multus/multus-dvpdv" Sep 11 09:13:06.202810 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.202140 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/34c95ff5-365b-414d-9968-2db918292b7f-cnibin\") pod \"multus-additional-cni-plugins-8z7wv\" (UID: \"34c95ff5-365b-414d-9968-2db918292b7f\") " pod="openshift-multus/multus-additional-cni-plugins-8z7wv" Sep 11 09:13:06.202810 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.202186 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/27736c8f-eddf-4460-94fd-b211c07ae744-cni-binary-copy\") pod \"multus-dvpdv\" (UID: \"27736c8f-eddf-4460-94fd-b211c07ae744\") " pod="openshift-multus/multus-dvpdv" Sep 11 09:13:06.202810 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.202441 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fc69178e-852e-4853-bb53-01b784cd8a3a-konnectivity-ca\") pod \"konnectivity-agent-pq626\" (UID: \"fc69178e-852e-4853-bb53-01b784cd8a3a\") " pod="kube-system/konnectivity-agent-pq626" Sep 11 09:13:06.202810 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.202594 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/34c95ff5-365b-414d-9968-2db918292b7f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8z7wv\" (UID: \"34c95ff5-365b-414d-9968-2db918292b7f\") " pod="openshift-multus/multus-additional-cni-plugins-8z7wv" Sep 11 09:13:06.203502 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.203481 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fc69178e-852e-4853-bb53-01b784cd8a3a-agent-certs\") pod \"konnectivity-agent-pq626\" (UID: \"fc69178e-852e-4853-bb53-01b784cd8a3a\") " pod="kube-system/konnectivity-agent-pq626" Sep 11 09:13:06.207821 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:06.207803 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 09:13:06.207918 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:06.207825 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 09:13:06.207918 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:06.207838 2575 projected.go:196] Error preparing data for projected volume kube-api-access-slfls for pod openshift-network-diagnostics/network-check-target-bb6x9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 09:13:06.207918 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:06.207913 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/92eb1c40-b8d0-428c-955a-c005e197f70a-kube-api-access-slfls podName:92eb1c40-b8d0-428c-955a-c005e197f70a nodeName:}" failed. No retries permitted until 2026-09-11 09:13:06.707896143 +0000 UTC m=+3.048506332 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-slfls" (UniqueName: "kubernetes.io/projected/92eb1c40-b8d0-428c-955a-c005e197f70a-kube-api-access-slfls") pod "network-check-target-bb6x9" (UID: "92eb1c40-b8d0-428c-955a-c005e197f70a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 09:13:06.210079 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.210053 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdzgt\" (UniqueName: \"kubernetes.io/projected/5a29489a-506a-4070-b30d-7366366058dc-kube-api-access-qdzgt\") pod \"node-resolver-g6t8z\" (UID: \"5a29489a-506a-4070-b30d-7366366058dc\") " pod="openshift-dns/node-resolver-g6t8z" Sep 11 09:13:06.210172 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.210146 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8vct\" (UniqueName: \"kubernetes.io/projected/34c95ff5-365b-414d-9968-2db918292b7f-kube-api-access-m8vct\") pod \"multus-additional-cni-plugins-8z7wv\" (UID: \"34c95ff5-365b-414d-9968-2db918292b7f\") " pod="openshift-multus/multus-additional-cni-plugins-8z7wv" Sep 11 09:13:06.212639 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.212618 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8chkf\" (UniqueName: \"kubernetes.io/projected/27736c8f-eddf-4460-94fd-b211c07ae744-kube-api-access-8chkf\") pod \"multus-dvpdv\" (UID: \"27736c8f-eddf-4460-94fd-b211c07ae744\") " pod="openshift-multus/multus-dvpdv" Sep 11 09:13:06.213080 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.213059 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mshgs\" (UniqueName: \"kubernetes.io/projected/65ac2ff5-03f0-4143-871f-061152802913-kube-api-access-mshgs\") pod \"iptables-alerter-5xwxg\" (UID: \"65ac2ff5-03f0-4143-871f-061152802913\") " pod="openshift-network-operator/iptables-alerter-5xwxg" Sep 11 09:13:06.294127 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.294036 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-skn2c" Sep 11 09:13:06.306430 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.306403 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l46kr" Sep 11 09:13:06.321040 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.321013 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-9zgvm" Sep 11 09:13:06.327912 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.327889 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-wgrlw" Sep 11 09:13:06.336562 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.336536 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-dvpdv" Sep 11 09:13:06.349137 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.349106 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-5xwxg" Sep 11 09:13:06.354857 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.354829 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-g6t8z" Sep 11 09:13:06.365491 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.365453 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-pq626" Sep 11 09:13:06.372148 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.372129 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-8z7wv" Sep 11 09:13:06.604243 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.604160 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abeb037b-2e12-4485-974d-35280f982f4a-metrics-certs\") pod \"network-metrics-daemon-rlftv\" (UID: \"abeb037b-2e12-4485-974d-35280f982f4a\") " pod="openshift-multus/network-metrics-daemon-rlftv" Sep 11 09:13:06.604385 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:06.604298 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 09:13:06.604385 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:06.604358 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/abeb037b-2e12-4485-974d-35280f982f4a-metrics-certs podName:abeb037b-2e12-4485-974d-35280f982f4a nodeName:}" failed. No retries permitted until 2026-09-11 09:13:07.604338816 +0000 UTC m=+3.944949018 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/abeb037b-2e12-4485-974d-35280f982f4a-metrics-certs") pod "network-metrics-daemon-rlftv" (UID: "abeb037b-2e12-4485-974d-35280f982f4a") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 09:13:06.705125 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.705081 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7eddd8ad-0ac3-421d-9649-95dc4fadfd7b-original-pull-secret\") pod \"global-pull-secret-syncer-8ng6h\" (UID: \"7eddd8ad-0ac3-421d-9649-95dc4fadfd7b\") " pod="kube-system/global-pull-secret-syncer-8ng6h" Sep 11 09:13:06.705297 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:06.705215 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 09:13:06.705297 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:06.705285 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7eddd8ad-0ac3-421d-9649-95dc4fadfd7b-original-pull-secret podName:7eddd8ad-0ac3-421d-9649-95dc4fadfd7b nodeName:}" failed. No retries permitted until 2026-09-11 09:13:07.705264604 +0000 UTC m=+4.045874789 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7eddd8ad-0ac3-421d-9649-95dc4fadfd7b-original-pull-secret") pod "global-pull-secret-syncer-8ng6h" (UID: "7eddd8ad-0ac3-421d-9649-95dc4fadfd7b") : object "kube-system"/"original-pull-secret" not registered Sep 11 09:13:06.806023 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:06.805981 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slfls\" (UniqueName: \"kubernetes.io/projected/92eb1c40-b8d0-428c-955a-c005e197f70a-kube-api-access-slfls\") pod \"network-check-target-bb6x9\" (UID: \"92eb1c40-b8d0-428c-955a-c005e197f70a\") " pod="openshift-network-diagnostics/network-check-target-bb6x9" Sep 11 09:13:06.806218 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:06.806129 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 09:13:06.806218 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:06.806148 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 09:13:06.806218 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:06.806161 2575 projected.go:196] Error preparing data for projected volume kube-api-access-slfls for pod openshift-network-diagnostics/network-check-target-bb6x9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 09:13:06.806370 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:06.806232 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/92eb1c40-b8d0-428c-955a-c005e197f70a-kube-api-access-slfls podName:92eb1c40-b8d0-428c-955a-c005e197f70a nodeName:}" failed. No retries permitted until 2026-09-11 09:13:07.806206196 +0000 UTC m=+4.146816389 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-slfls" (UniqueName: "kubernetes.io/projected/92eb1c40-b8d0-428c-955a-c005e197f70a-kube-api-access-slfls") pod "network-check-target-bb6x9" (UID: "92eb1c40-b8d0-428c-955a-c005e197f70a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 09:13:07.035187 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:07.035078 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 09:08:06 +0000 UTC" deadline="2028-02-10 05:36:07.702057344 +0000 UTC" Sep 11 09:13:07.035187 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:07.035132 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12404h23m0.66693059s" Sep 11 09:13:07.108766 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:07.108729 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bb6x9" Sep 11 09:13:07.108963 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:07.108863 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bb6x9" podUID="92eb1c40-b8d0-428c-955a-c005e197f70a" Sep 11 09:13:07.187567 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:07.187339 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcb16b38_81cb_41f7_81d2_730c64ee3a59.slice/crio-7229d729cd5f878016db3243e156dabd2be02fae43fb86ef319815448b5cf570 WatchSource:0}: Error finding container 7229d729cd5f878016db3243e156dabd2be02fae43fb86ef319815448b5cf570: Status 404 returned error can't find the container with id 7229d729cd5f878016db3243e156dabd2be02fae43fb86ef319815448b5cf570 Sep 11 09:13:07.188500 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:07.188449 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc69178e_852e_4853_bb53_01b784cd8a3a.slice/crio-df9be27071d20f989dab5209aad756b2e1cbcca85ee209eea29e39847f60ad39 WatchSource:0}: Error finding container df9be27071d20f989dab5209aad756b2e1cbcca85ee209eea29e39847f60ad39: Status 404 returned error can't find the container with id df9be27071d20f989dab5209aad756b2e1cbcca85ee209eea29e39847f60ad39 Sep 11 09:13:07.192580 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:07.192557 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a29489a_506a_4070_b30d_7366366058dc.slice/crio-889dbacede6697f9003f3598bbcc0507ff144e68b64691fee2fc7b3f0f670dcc WatchSource:0}: Error finding container 889dbacede6697f9003f3598bbcc0507ff144e68b64691fee2fc7b3f0f670dcc: Status 404 returned error can't find the container with id 889dbacede6697f9003f3598bbcc0507ff144e68b64691fee2fc7b3f0f670dcc Sep 11 09:13:07.193757 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:07.193734 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27736c8f_eddf_4460_94fd_b211c07ae744.slice/crio-a2615958032e1203c69671677d573dc910535537fc8f019aadb069bcc08739da WatchSource:0}: Error finding container a2615958032e1203c69671677d573dc910535537fc8f019aadb069bcc08739da: Status 404 returned error can't find the container with id a2615958032e1203c69671677d573dc910535537fc8f019aadb069bcc08739da Sep 11 09:13:07.194613 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:07.194591 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34c95ff5_365b_414d_9968_2db918292b7f.slice/crio-cbbd25ae0abea11567d9a21bb924bed4a6024ed1c3ba24f6917fc153fa949f3c WatchSource:0}: Error finding container cbbd25ae0abea11567d9a21bb924bed4a6024ed1c3ba24f6917fc153fa949f3c: Status 404 returned error can't find the container with id cbbd25ae0abea11567d9a21bb924bed4a6024ed1c3ba24f6917fc153fa949f3c Sep 11 09:13:07.195938 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:07.195913 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08d48976_3c61_4273_88da_fa30ba4057bc.slice/crio-274b99982f374f10c76479d7ea9b67aac49896242cd5f76f27f341bbc4f108a8 WatchSource:0}: Error finding container 274b99982f374f10c76479d7ea9b67aac49896242cd5f76f27f341bbc4f108a8: Status 404 returned error can't find the container with id 274b99982f374f10c76479d7ea9b67aac49896242cd5f76f27f341bbc4f108a8 Sep 11 09:13:07.216611 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:07.216585 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f74babf_697e_4234_b22e_f7a18ba4a753.slice/crio-cd5e8fde804583901ec422c2940f91b2aa691f3288eedf70fc1779a6b8804bae WatchSource:0}: Error finding container cd5e8fde804583901ec422c2940f91b2aa691f3288eedf70fc1779a6b8804bae: Status 404 returned error can't find the container with id cd5e8fde804583901ec422c2940f91b2aa691f3288eedf70fc1779a6b8804bae Sep 11 09:13:07.217372 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:07.217341 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65ac2ff5_03f0_4143_871f_061152802913.slice/crio-852bcec0f665c528585e72d46cb7173a796845ab8b77dd72730f4d5de3ba6b5b WatchSource:0}: Error finding container 852bcec0f665c528585e72d46cb7173a796845ab8b77dd72730f4d5de3ba6b5b: Status 404 returned error can't find the container with id 852bcec0f665c528585e72d46cb7173a796845ab8b77dd72730f4d5de3ba6b5b Sep 11 09:13:07.220512 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:07.219749 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c0627e0_f086_4fc4_b2d7_724b67c685d4.slice/crio-129864c7b4bc65a4c2f09b2b6676cc12df44d4bfbb599d0df90ce3117d52d179 WatchSource:0}: Error finding container 129864c7b4bc65a4c2f09b2b6676cc12df44d4bfbb599d0df90ce3117d52d179: Status 404 returned error can't find the container with id 129864c7b4bc65a4c2f09b2b6676cc12df44d4bfbb599d0df90ce3117d52d179 Sep 11 09:13:07.613101 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:07.613011 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abeb037b-2e12-4485-974d-35280f982f4a-metrics-certs\") pod \"network-metrics-daemon-rlftv\" (UID: \"abeb037b-2e12-4485-974d-35280f982f4a\") " pod="openshift-multus/network-metrics-daemon-rlftv" Sep 11 09:13:07.613260 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:07.613185 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 09:13:07.613333 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:07.613262 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/abeb037b-2e12-4485-974d-35280f982f4a-metrics-certs podName:abeb037b-2e12-4485-974d-35280f982f4a nodeName:}" failed. No retries permitted until 2026-09-11 09:13:09.613242516 +0000 UTC m=+5.953852715 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/abeb037b-2e12-4485-974d-35280f982f4a-metrics-certs") pod "network-metrics-daemon-rlftv" (UID: "abeb037b-2e12-4485-974d-35280f982f4a") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 09:13:07.714634 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:07.714004 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7eddd8ad-0ac3-421d-9649-95dc4fadfd7b-original-pull-secret\") pod \"global-pull-secret-syncer-8ng6h\" (UID: \"7eddd8ad-0ac3-421d-9649-95dc4fadfd7b\") " pod="kube-system/global-pull-secret-syncer-8ng6h" Sep 11 09:13:07.714634 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:07.714212 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 09:13:07.714634 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:07.714276 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7eddd8ad-0ac3-421d-9649-95dc4fadfd7b-original-pull-secret podName:7eddd8ad-0ac3-421d-9649-95dc4fadfd7b nodeName:}" failed. No retries permitted until 2026-09-11 09:13:09.714258851 +0000 UTC m=+6.054869049 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7eddd8ad-0ac3-421d-9649-95dc4fadfd7b-original-pull-secret") pod "global-pull-secret-syncer-8ng6h" (UID: "7eddd8ad-0ac3-421d-9649-95dc4fadfd7b") : object "kube-system"/"original-pull-secret" not registered Sep 11 09:13:07.816104 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:07.815377 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slfls\" (UniqueName: \"kubernetes.io/projected/92eb1c40-b8d0-428c-955a-c005e197f70a-kube-api-access-slfls\") pod \"network-check-target-bb6x9\" (UID: \"92eb1c40-b8d0-428c-955a-c005e197f70a\") " pod="openshift-network-diagnostics/network-check-target-bb6x9" Sep 11 09:13:07.816104 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:07.815613 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 09:13:07.816104 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:07.815638 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 09:13:07.816104 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:07.815651 2575 projected.go:196] Error preparing data for projected volume kube-api-access-slfls for pod openshift-network-diagnostics/network-check-target-bb6x9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 09:13:07.816104 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:07.815713 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/92eb1c40-b8d0-428c-955a-c005e197f70a-kube-api-access-slfls podName:92eb1c40-b8d0-428c-955a-c005e197f70a nodeName:}" failed. No retries permitted until 2026-09-11 09:13:09.815695898 +0000 UTC m=+6.156306087 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-slfls" (UniqueName: "kubernetes.io/projected/92eb1c40-b8d0-428c-955a-c005e197f70a-kube-api-access-slfls") pod "network-check-target-bb6x9" (UID: "92eb1c40-b8d0-428c-955a-c005e197f70a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 09:13:08.035590 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:08.035445 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 09:08:06 +0000 UTC" deadline="2028-03-23 17:39:32.159670323 +0000 UTC" Sep 11 09:13:08.035590 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:08.035514 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13424h26m24.124161377s" Sep 11 09:13:08.109762 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:08.109573 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8ng6h" Sep 11 09:13:08.110212 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:08.110039 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rlftv" Sep 11 09:13:08.110212 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:08.110157 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rlftv" podUID="abeb037b-2e12-4485-974d-35280f982f4a" Sep 11 09:13:08.110422 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:08.109702 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8ng6h" podUID="7eddd8ad-0ac3-421d-9649-95dc4fadfd7b" Sep 11 09:13:08.138420 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:08.138381 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-5xwxg" event={"ID":"65ac2ff5-03f0-4143-871f-061152802913","Type":"ContainerStarted","Data":"852bcec0f665c528585e72d46cb7173a796845ab8b77dd72730f4d5de3ba6b5b"} Sep 11 09:13:08.148188 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:08.148148 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l46kr" event={"ID":"1f74babf-697e-4234-b22e-f7a18ba4a753","Type":"ContainerStarted","Data":"cd5e8fde804583901ec422c2940f91b2aa691f3288eedf70fc1779a6b8804bae"} Sep 11 09:13:08.156839 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:08.156800 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-skn2c" event={"ID":"08d48976-3c61-4273-88da-fa30ba4057bc","Type":"ContainerStarted","Data":"274b99982f374f10c76479d7ea9b67aac49896242cd5f76f27f341bbc4f108a8"} Sep 11 09:13:08.164764 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:08.164723 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8z7wv" event={"ID":"34c95ff5-365b-414d-9968-2db918292b7f","Type":"ContainerStarted","Data":"cbbd25ae0abea11567d9a21bb924bed4a6024ed1c3ba24f6917fc153fa949f3c"} Sep 11 09:13:08.172120 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:08.172082 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-pq626" event={"ID":"fc69178e-852e-4853-bb53-01b784cd8a3a","Type":"ContainerStarted","Data":"df9be27071d20f989dab5209aad756b2e1cbcca85ee209eea29e39847f60ad39"} Sep 11 09:13:08.186240 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:08.186192 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-9zgvm" event={"ID":"fcb16b38-81cb-41f7-81d2-730c64ee3a59","Type":"ContainerStarted","Data":"7229d729cd5f878016db3243e156dabd2be02fae43fb86ef319815448b5cf570"} Sep 11 09:13:08.209154 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:08.208370 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-119.ec2.internal" event={"ID":"ce93b5afb1de6b9a80d64df7da81462d","Type":"ContainerStarted","Data":"689a48c556c8dd98015916f3a83651d7a4c0c43f709402e1ef63b0dc60e1bc82"} Sep 11 09:13:08.218826 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:08.218775 2575 generic.go:356] "Generic (PLEG): container finished" podID="db42fcd462e8e18de823d8d0adf7cc0a" containerID="8a860a4ecab6f596c1f7b5282e0fe7dd1e38bd21774c83d9875eb108996429a7" exitCode=0 Sep 11 09:13:08.218999 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:08.218892 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-119.ec2.internal" event={"ID":"db42fcd462e8e18de823d8d0adf7cc0a","Type":"ContainerDied","Data":"8a860a4ecab6f596c1f7b5282e0fe7dd1e38bd21774c83d9875eb108996429a7"} Sep 11 09:13:08.223822 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:08.223767 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-119.ec2.internal" podStartSLOduration=3.223745634 podStartE2EDuration="3.223745634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 09:13:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 09:13:08.222560558 +0000 UTC m=+4.563170765" watchObservedRunningTime="2026-09-11 09:13:08.223745634 +0000 UTC m=+4.564355841" Sep 11 09:13:08.241336 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:08.241226 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wgrlw" event={"ID":"3c0627e0-f086-4fc4-b2d7-724b67c685d4","Type":"ContainerStarted","Data":"129864c7b4bc65a4c2f09b2b6676cc12df44d4bfbb599d0df90ce3117d52d179"} Sep 11 09:13:08.258883 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:08.258796 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-dvpdv" event={"ID":"27736c8f-eddf-4460-94fd-b211c07ae744","Type":"ContainerStarted","Data":"a2615958032e1203c69671677d573dc910535537fc8f019aadb069bcc08739da"} Sep 11 09:13:08.271659 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:08.271612 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-g6t8z" event={"ID":"5a29489a-506a-4070-b30d-7366366058dc","Type":"ContainerStarted","Data":"889dbacede6697f9003f3598bbcc0507ff144e68b64691fee2fc7b3f0f670dcc"} Sep 11 09:13:09.109605 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:09.109144 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bb6x9" Sep 11 09:13:09.109605 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:09.109257 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bb6x9" podUID="92eb1c40-b8d0-428c-955a-c005e197f70a" Sep 11 09:13:09.296651 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:09.296613 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-119.ec2.internal" event={"ID":"db42fcd462e8e18de823d8d0adf7cc0a","Type":"ContainerStarted","Data":"8d4fb3c0664acb6c2f6fefd33c362d432f6ae9455b95df85a263fdddb79d3a52"} Sep 11 09:13:09.313558 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:09.313486 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-119.ec2.internal" podStartSLOduration=4.313444228 podStartE2EDuration="4.313444228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 09:13:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 09:13:09.312144648 +0000 UTC m=+5.652754854" watchObservedRunningTime="2026-09-11 09:13:09.313444228 +0000 UTC m=+5.654054434" Sep 11 09:13:09.632203 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:09.632168 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abeb037b-2e12-4485-974d-35280f982f4a-metrics-certs\") pod \"network-metrics-daemon-rlftv\" (UID: \"abeb037b-2e12-4485-974d-35280f982f4a\") " pod="openshift-multus/network-metrics-daemon-rlftv" Sep 11 09:13:09.632390 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:09.632304 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 09:13:09.632390 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:09.632365 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/abeb037b-2e12-4485-974d-35280f982f4a-metrics-certs podName:abeb037b-2e12-4485-974d-35280f982f4a nodeName:}" failed. No retries permitted until 2026-09-11 09:13:13.632347779 +0000 UTC m=+9.972957969 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/abeb037b-2e12-4485-974d-35280f982f4a-metrics-certs") pod "network-metrics-daemon-rlftv" (UID: "abeb037b-2e12-4485-974d-35280f982f4a") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 09:13:09.733194 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:09.733158 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7eddd8ad-0ac3-421d-9649-95dc4fadfd7b-original-pull-secret\") pod \"global-pull-secret-syncer-8ng6h\" (UID: \"7eddd8ad-0ac3-421d-9649-95dc4fadfd7b\") " pod="kube-system/global-pull-secret-syncer-8ng6h" Sep 11 09:13:09.733381 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:09.733332 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 09:13:09.733439 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:09.733390 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7eddd8ad-0ac3-421d-9649-95dc4fadfd7b-original-pull-secret podName:7eddd8ad-0ac3-421d-9649-95dc4fadfd7b nodeName:}" failed. No retries permitted until 2026-09-11 09:13:13.733373617 +0000 UTC m=+10.073983801 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7eddd8ad-0ac3-421d-9649-95dc4fadfd7b-original-pull-secret") pod "global-pull-secret-syncer-8ng6h" (UID: "7eddd8ad-0ac3-421d-9649-95dc4fadfd7b") : object "kube-system"/"original-pull-secret" not registered Sep 11 09:13:09.833659 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:09.833624 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slfls\" (UniqueName: \"kubernetes.io/projected/92eb1c40-b8d0-428c-955a-c005e197f70a-kube-api-access-slfls\") pod \"network-check-target-bb6x9\" (UID: \"92eb1c40-b8d0-428c-955a-c005e197f70a\") " pod="openshift-network-diagnostics/network-check-target-bb6x9" Sep 11 09:13:09.833825 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:09.833793 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 09:13:09.833825 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:09.833811 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 09:13:09.833825 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:09.833823 2575 projected.go:196] Error preparing data for projected volume kube-api-access-slfls for pod openshift-network-diagnostics/network-check-target-bb6x9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 09:13:09.833991 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:09.833885 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/92eb1c40-b8d0-428c-955a-c005e197f70a-kube-api-access-slfls podName:92eb1c40-b8d0-428c-955a-c005e197f70a nodeName:}" failed. No retries permitted until 2026-09-11 09:13:13.833867952 +0000 UTC m=+10.174478157 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-slfls" (UniqueName: "kubernetes.io/projected/92eb1c40-b8d0-428c-955a-c005e197f70a-kube-api-access-slfls") pod "network-check-target-bb6x9" (UID: "92eb1c40-b8d0-428c-955a-c005e197f70a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 09:13:10.109949 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:10.109592 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8ng6h" Sep 11 09:13:10.109949 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:10.109721 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8ng6h" podUID="7eddd8ad-0ac3-421d-9649-95dc4fadfd7b" Sep 11 09:13:10.109949 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:10.109729 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rlftv" Sep 11 09:13:10.109949 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:10.109840 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rlftv" podUID="abeb037b-2e12-4485-974d-35280f982f4a" Sep 11 09:13:11.109511 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:11.109462 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bb6x9" Sep 11 09:13:11.109707 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:11.109612 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bb6x9" podUID="92eb1c40-b8d0-428c-955a-c005e197f70a" Sep 11 09:13:12.109760 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:12.109451 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rlftv" Sep 11 09:13:12.109760 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:12.109612 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rlftv" podUID="abeb037b-2e12-4485-974d-35280f982f4a" Sep 11 09:13:12.109760 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:12.109451 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8ng6h" Sep 11 09:13:12.109760 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:12.109725 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8ng6h" podUID="7eddd8ad-0ac3-421d-9649-95dc4fadfd7b" Sep 11 09:13:13.109299 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:13.109223 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bb6x9" Sep 11 09:13:13.109493 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:13.109361 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bb6x9" podUID="92eb1c40-b8d0-428c-955a-c005e197f70a" Sep 11 09:13:13.664933 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:13.664889 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abeb037b-2e12-4485-974d-35280f982f4a-metrics-certs\") pod \"network-metrics-daemon-rlftv\" (UID: \"abeb037b-2e12-4485-974d-35280f982f4a\") " pod="openshift-multus/network-metrics-daemon-rlftv" Sep 11 09:13:13.665344 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:13.665016 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 09:13:13.665344 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:13.665076 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/abeb037b-2e12-4485-974d-35280f982f4a-metrics-certs podName:abeb037b-2e12-4485-974d-35280f982f4a nodeName:}" failed. No retries permitted until 2026-09-11 09:13:21.665058051 +0000 UTC m=+18.005668253 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/abeb037b-2e12-4485-974d-35280f982f4a-metrics-certs") pod "network-metrics-daemon-rlftv" (UID: "abeb037b-2e12-4485-974d-35280f982f4a") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 09:13:13.766590 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:13.765955 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7eddd8ad-0ac3-421d-9649-95dc4fadfd7b-original-pull-secret\") pod \"global-pull-secret-syncer-8ng6h\" (UID: \"7eddd8ad-0ac3-421d-9649-95dc4fadfd7b\") " pod="kube-system/global-pull-secret-syncer-8ng6h" Sep 11 09:13:13.766590 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:13.766163 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 09:13:13.766590 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:13.766229 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7eddd8ad-0ac3-421d-9649-95dc4fadfd7b-original-pull-secret podName:7eddd8ad-0ac3-421d-9649-95dc4fadfd7b nodeName:}" failed. No retries permitted until 2026-09-11 09:13:21.76621293 +0000 UTC m=+18.106823121 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7eddd8ad-0ac3-421d-9649-95dc4fadfd7b-original-pull-secret") pod "global-pull-secret-syncer-8ng6h" (UID: "7eddd8ad-0ac3-421d-9649-95dc4fadfd7b") : object "kube-system"/"original-pull-secret" not registered Sep 11 09:13:13.867131 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:13.866366 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slfls\" (UniqueName: \"kubernetes.io/projected/92eb1c40-b8d0-428c-955a-c005e197f70a-kube-api-access-slfls\") pod \"network-check-target-bb6x9\" (UID: \"92eb1c40-b8d0-428c-955a-c005e197f70a\") " pod="openshift-network-diagnostics/network-check-target-bb6x9" Sep 11 09:13:13.867131 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:13.866599 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 09:13:13.867131 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:13.866624 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 09:13:13.867131 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:13.866636 2575 projected.go:196] Error preparing data for projected volume kube-api-access-slfls for pod openshift-network-diagnostics/network-check-target-bb6x9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 09:13:13.867131 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:13.866698 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/92eb1c40-b8d0-428c-955a-c005e197f70a-kube-api-access-slfls podName:92eb1c40-b8d0-428c-955a-c005e197f70a nodeName:}" failed. No retries permitted until 2026-09-11 09:13:21.866679963 +0000 UTC m=+18.207290151 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-slfls" (UniqueName: "kubernetes.io/projected/92eb1c40-b8d0-428c-955a-c005e197f70a-kube-api-access-slfls") pod "network-check-target-bb6x9" (UID: "92eb1c40-b8d0-428c-955a-c005e197f70a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 09:13:14.109691 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:14.109609 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rlftv" Sep 11 09:13:14.109846 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:14.109713 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rlftv" podUID="abeb037b-2e12-4485-974d-35280f982f4a" Sep 11 09:13:14.110198 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:14.109621 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8ng6h" Sep 11 09:13:14.110198 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:14.110146 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8ng6h" podUID="7eddd8ad-0ac3-421d-9649-95dc4fadfd7b" Sep 11 09:13:15.109222 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:15.109158 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bb6x9" Sep 11 09:13:15.109739 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:15.109349 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bb6x9" podUID="92eb1c40-b8d0-428c-955a-c005e197f70a" Sep 11 09:13:16.108732 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:16.108699 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rlftv" Sep 11 09:13:16.108923 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:16.108706 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8ng6h" Sep 11 09:13:16.108923 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:16.108823 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rlftv" podUID="abeb037b-2e12-4485-974d-35280f982f4a" Sep 11 09:13:16.109046 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:16.108931 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8ng6h" podUID="7eddd8ad-0ac3-421d-9649-95dc4fadfd7b" Sep 11 09:13:17.108524 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:17.108495 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bb6x9" Sep 11 09:13:17.108898 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:17.108613 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bb6x9" podUID="92eb1c40-b8d0-428c-955a-c005e197f70a" Sep 11 09:13:18.111525 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:18.111451 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8ng6h" Sep 11 09:13:18.111933 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:18.111457 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rlftv" Sep 11 09:13:18.111933 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:18.111566 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8ng6h" podUID="7eddd8ad-0ac3-421d-9649-95dc4fadfd7b" Sep 11 09:13:18.111933 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:18.111695 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rlftv" podUID="abeb037b-2e12-4485-974d-35280f982f4a" Sep 11 09:13:19.108655 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:19.108615 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bb6x9" Sep 11 09:13:19.108857 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:19.108737 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bb6x9" podUID="92eb1c40-b8d0-428c-955a-c005e197f70a" Sep 11 09:13:20.108812 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:20.108775 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rlftv" Sep 11 09:13:20.109266 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:20.108824 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8ng6h" Sep 11 09:13:20.109266 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:20.108941 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rlftv" podUID="abeb037b-2e12-4485-974d-35280f982f4a" Sep 11 09:13:20.109266 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:20.109082 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8ng6h" podUID="7eddd8ad-0ac3-421d-9649-95dc4fadfd7b" Sep 11 09:13:21.108952 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:21.108907 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bb6x9" Sep 11 09:13:21.109449 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:21.109046 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bb6x9" podUID="92eb1c40-b8d0-428c-955a-c005e197f70a" Sep 11 09:13:21.727304 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:21.727204 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abeb037b-2e12-4485-974d-35280f982f4a-metrics-certs\") pod \"network-metrics-daemon-rlftv\" (UID: \"abeb037b-2e12-4485-974d-35280f982f4a\") " pod="openshift-multus/network-metrics-daemon-rlftv" Sep 11 09:13:21.727482 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:21.727347 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 09:13:21.727482 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:21.727426 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/abeb037b-2e12-4485-974d-35280f982f4a-metrics-certs podName:abeb037b-2e12-4485-974d-35280f982f4a nodeName:}" failed. No retries permitted until 2026-09-11 09:13:37.727404213 +0000 UTC m=+34.068014397 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/abeb037b-2e12-4485-974d-35280f982f4a-metrics-certs") pod "network-metrics-daemon-rlftv" (UID: "abeb037b-2e12-4485-974d-35280f982f4a") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 09:13:21.828146 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:21.828106 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7eddd8ad-0ac3-421d-9649-95dc4fadfd7b-original-pull-secret\") pod \"global-pull-secret-syncer-8ng6h\" (UID: \"7eddd8ad-0ac3-421d-9649-95dc4fadfd7b\") " pod="kube-system/global-pull-secret-syncer-8ng6h" Sep 11 09:13:21.828330 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:21.828272 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 09:13:21.828389 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:21.828347 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7eddd8ad-0ac3-421d-9649-95dc4fadfd7b-original-pull-secret podName:7eddd8ad-0ac3-421d-9649-95dc4fadfd7b nodeName:}" failed. No retries permitted until 2026-09-11 09:13:37.828325851 +0000 UTC m=+34.168936060 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7eddd8ad-0ac3-421d-9649-95dc4fadfd7b-original-pull-secret") pod "global-pull-secret-syncer-8ng6h" (UID: "7eddd8ad-0ac3-421d-9649-95dc4fadfd7b") : object "kube-system"/"original-pull-secret" not registered Sep 11 09:13:21.929282 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:21.929244 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slfls\" (UniqueName: \"kubernetes.io/projected/92eb1c40-b8d0-428c-955a-c005e197f70a-kube-api-access-slfls\") pod \"network-check-target-bb6x9\" (UID: \"92eb1c40-b8d0-428c-955a-c005e197f70a\") " pod="openshift-network-diagnostics/network-check-target-bb6x9" Sep 11 09:13:21.929455 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:21.929431 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 09:13:21.929531 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:21.929458 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 09:13:21.929531 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:21.929487 2575 projected.go:196] Error preparing data for projected volume kube-api-access-slfls for pod openshift-network-diagnostics/network-check-target-bb6x9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 09:13:21.929643 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:21.929560 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/92eb1c40-b8d0-428c-955a-c005e197f70a-kube-api-access-slfls podName:92eb1c40-b8d0-428c-955a-c005e197f70a nodeName:}" failed. No retries permitted until 2026-09-11 09:13:37.929540376 +0000 UTC m=+34.270150564 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-slfls" (UniqueName: "kubernetes.io/projected/92eb1c40-b8d0-428c-955a-c005e197f70a-kube-api-access-slfls") pod "network-check-target-bb6x9" (UID: "92eb1c40-b8d0-428c-955a-c005e197f70a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 09:13:22.109036 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:22.108954 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rlftv" Sep 11 09:13:22.109036 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:22.109000 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8ng6h" Sep 11 09:13:22.109506 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:22.109106 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rlftv" podUID="abeb037b-2e12-4485-974d-35280f982f4a" Sep 11 09:13:22.109506 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:22.109227 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8ng6h" podUID="7eddd8ad-0ac3-421d-9649-95dc4fadfd7b" Sep 11 09:13:23.108779 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:23.108567 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bb6x9" Sep 11 09:13:23.108969 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:23.108879 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bb6x9" podUID="92eb1c40-b8d0-428c-955a-c005e197f70a" Sep 11 09:13:24.111993 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:24.111962 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8ng6h" Sep 11 09:13:24.112400 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:24.112099 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8ng6h" podUID="7eddd8ad-0ac3-421d-9649-95dc4fadfd7b" Sep 11 09:13:24.112400 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:24.112138 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rlftv" Sep 11 09:13:24.112400 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:24.112241 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rlftv" podUID="abeb037b-2e12-4485-974d-35280f982f4a" Sep 11 09:13:25.109135 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:25.108953 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bb6x9" Sep 11 09:13:25.109258 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:25.109210 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bb6x9" podUID="92eb1c40-b8d0-428c-955a-c005e197f70a" Sep 11 09:13:25.328767 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:25.328739 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wgrlw" event={"ID":"3c0627e0-f086-4fc4-b2d7-724b67c685d4","Type":"ContainerStarted","Data":"f038d8c590306ec4c41c7fd62406c0ee7afe8607fdf7aed446e4ce7e2b4f9bb6"} Sep 11 09:13:25.329396 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:25.328813 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wgrlw" event={"ID":"3c0627e0-f086-4fc4-b2d7-724b67c685d4","Type":"ContainerStarted","Data":"3a252611d7b23e1e83423d957cb6c5a8f19195e770c4c31e1b90066e1fa433e8"} Sep 11 09:13:25.329396 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:25.328848 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wgrlw" event={"ID":"3c0627e0-f086-4fc4-b2d7-724b67c685d4","Type":"ContainerStarted","Data":"eebb3dd34c6f7226f104e36e8d4b0f85825dfb3391003f4b391e15b3c18fc636"} Sep 11 09:13:25.329396 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:25.328881 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wgrlw" event={"ID":"3c0627e0-f086-4fc4-b2d7-724b67c685d4","Type":"ContainerStarted","Data":"cf50f87dbe8ba9eb6d80b45359316a88f067b550d8ac1af052e2b2f9afe98309"} Sep 11 09:13:25.329396 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:25.328908 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wgrlw" event={"ID":"3c0627e0-f086-4fc4-b2d7-724b67c685d4","Type":"ContainerStarted","Data":"1d6b2ffeb4f429b21be3dccf6e14864c69482bebc6b9fef589e41d62ff8c84be"} Sep 11 09:13:25.330146 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:25.330126 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-dvpdv" event={"ID":"27736c8f-eddf-4460-94fd-b211c07ae744","Type":"ContainerStarted","Data":"90df5a9842ba187e66e1fd961518bf9a66e979a7123e49bb77b57d9dd89f4de5"} Sep 11 09:13:25.331570 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:25.331551 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-g6t8z" event={"ID":"5a29489a-506a-4070-b30d-7366366058dc","Type":"ContainerStarted","Data":"0b837664a5d407dd03b260c1c86f80a03a1584a0d8636277e9068c503e1f3b31"} Sep 11 09:13:25.332923 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:25.332904 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l46kr" event={"ID":"1f74babf-697e-4234-b22e-f7a18ba4a753","Type":"ContainerStarted","Data":"0d888baf87f62f135d67978d0a2b5994ee5818c6501911ad305bed4e7ce75b5f"} Sep 11 09:13:25.334235 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:25.334217 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-skn2c" event={"ID":"08d48976-3c61-4273-88da-fa30ba4057bc","Type":"ContainerStarted","Data":"76f32f14aeb7967e6c00ebaba3d4f2a759abffb681b37c7b0c9bc7ebf79014ef"} Sep 11 09:13:25.337670 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:25.337652 2575 generic.go:356] "Generic (PLEG): container finished" podID="34c95ff5-365b-414d-9968-2db918292b7f" containerID="93889c71acca52c99c245b22d4eed58351eb9d643ef7d0388f4c8a9e5c112c65" exitCode=0 Sep 11 09:13:25.337743 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:25.337715 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8z7wv" event={"ID":"34c95ff5-365b-414d-9968-2db918292b7f","Type":"ContainerDied","Data":"93889c71acca52c99c245b22d4eed58351eb9d643ef7d0388f4c8a9e5c112c65"} Sep 11 09:13:25.339628 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:25.339318 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-pq626" event={"ID":"fc69178e-852e-4853-bb53-01b784cd8a3a","Type":"ContainerStarted","Data":"fe65024bccc9e26a1d53cd0c7ad2e4090a0d14d435a735ea82830005d6a37d9d"} Sep 11 09:13:25.340920 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:25.340897 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-9zgvm" event={"ID":"fcb16b38-81cb-41f7-81d2-730c64ee3a59","Type":"ContainerStarted","Data":"aee684be1e44ed87bdf962e0e5edc8b8a1f1d98950330c32dcf48fa610d58aed"} Sep 11 09:13:25.358694 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:25.358642 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-dvpdv" podStartSLOduration=3.584899232 podStartE2EDuration="21.35862788s" totalImagesPullingTime="17.773728648s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 09:13:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 09:13:25.35834146 +0000 UTC m=+21.698951665" watchObservedRunningTime="2026-09-11 09:13:25.35862788 +0000 UTC m=+21.699238124" Sep 11 09:13:25.376374 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:25.376335 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-skn2c" podStartSLOduration=3.836768343 podStartE2EDuration="21.376322224s" totalImagesPullingTime="17.539553881s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 09:13:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 09:13:25.376081718 +0000 UTC m=+21.716691925" watchObservedRunningTime="2026-09-11 09:13:25.376322224 +0000 UTC m=+21.716932429" Sep 11 09:13:25.397521 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:25.397480 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-pq626" podStartSLOduration=3.833276531 podStartE2EDuration="21.397450349s" totalImagesPullingTime="17.564173818s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 09:13:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 09:13:25.393044041 +0000 UTC m=+21.733654247" watchObservedRunningTime="2026-09-11 09:13:25.397450349 +0000 UTC m=+21.738060556" Sep 11 09:13:25.432071 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:25.432013 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-9zgvm" podStartSLOduration=3.847543909 podStartE2EDuration="21.431991356s" totalImagesPullingTime="17.584447447s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 09:13:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 09:13:25.431770399 +0000 UTC m=+21.772380605" watchObservedRunningTime="2026-09-11 09:13:25.431991356 +0000 UTC m=+21.772601586" Sep 11 09:13:25.446616 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:25.446572 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-g6t8z" podStartSLOduration=3.886321977 podStartE2EDuration="21.446555842s" totalImagesPullingTime="17.560233865s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 09:13:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 09:13:25.446517667 +0000 UTC m=+21.787127874" watchObservedRunningTime="2026-09-11 09:13:25.446555842 +0000 UTC m=+21.787166047" Sep 11 09:13:25.777639 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:25.777602 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-pq626" Sep 11 09:13:25.778929 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:25.778907 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-pq626" Sep 11 09:13:26.108688 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:26.108602 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8ng6h" Sep 11 09:13:26.108845 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:26.108603 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rlftv" Sep 11 09:13:26.108845 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:26.108746 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8ng6h" podUID="7eddd8ad-0ac3-421d-9649-95dc4fadfd7b" Sep 11 09:13:26.108845 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:26.108818 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rlftv" podUID="abeb037b-2e12-4485-974d-35280f982f4a" Sep 11 09:13:26.346877 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:26.346845 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wgrlw" event={"ID":"3c0627e0-f086-4fc4-b2d7-724b67c685d4","Type":"ContainerStarted","Data":"de83cf8e546ca73d1144a96ff2c24ed4c8b9bfac7e658dd634793bdfb132e36e"} Sep 11 09:13:26.348341 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:26.348299 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-5xwxg" event={"ID":"65ac2ff5-03f0-4143-871f-061152802913","Type":"ContainerStarted","Data":"0b4dfa6e939cbf4711e07fe2ce4aced81daa5dfab0598450d9846c5923563e17"} Sep 11 09:13:26.349148 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:26.349126 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-pq626" Sep 11 09:13:26.349697 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:26.349680 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-pq626" Sep 11 09:13:26.368675 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:26.368622 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-5xwxg" podStartSLOduration=4.814961867 podStartE2EDuration="22.368597277s" totalImagesPullingTime="17.55363541s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 09:13:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 09:13:26.368459693 +0000 UTC m=+22.709069901" watchObservedRunningTime="2026-09-11 09:13:26.368597277 +0000 UTC m=+22.709207483" Sep 11 09:13:26.384264 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:26.384237 2575 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 11 09:13:27.054805 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:27.054683 2575 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-11T09:13:26.384258478Z","UUID":"55a881dc-e40d-4836-bb51-d5d12dc48e40","Handler":null,"Name":"","Endpoint":""} Sep 11 09:13:27.056444 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:27.056421 2575 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 11 09:13:27.056601 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:27.056456 2575 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 11 09:13:27.109380 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:27.109343 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bb6x9" Sep 11 09:13:27.109583 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:27.109461 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bb6x9" podUID="92eb1c40-b8d0-428c-955a-c005e197f70a" Sep 11 09:13:27.353581 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:27.353495 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l46kr" event={"ID":"1f74babf-697e-4234-b22e-f7a18ba4a753","Type":"ContainerStarted","Data":"57aa35b195a0937f8e943995f6ae55c0a5fac18e2ef555370aa1bfc9452af22e"} Sep 11 09:13:28.112685 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:28.112653 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8ng6h" Sep 11 09:13:28.112887 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:28.112790 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8ng6h" podUID="7eddd8ad-0ac3-421d-9649-95dc4fadfd7b" Sep 11 09:13:28.112887 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:28.112831 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rlftv" Sep 11 09:13:28.112996 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:28.112970 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rlftv" podUID="abeb037b-2e12-4485-974d-35280f982f4a" Sep 11 09:13:28.358532 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:28.358490 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wgrlw" event={"ID":"3c0627e0-f086-4fc4-b2d7-724b67c685d4","Type":"ContainerStarted","Data":"4710a74d2ed56beb598ddadd190b2cd533496a4c8a5d2f2af5a54d4b45739232"} Sep 11 09:13:28.361216 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:28.361190 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l46kr" event={"ID":"1f74babf-697e-4234-b22e-f7a18ba4a753","Type":"ContainerStarted","Data":"3c6099b3295fc46d11753ebb28a447cdbfc4fa7e64b8208776121a6cf2672fd6"} Sep 11 09:13:28.381961 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:28.381868 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l46kr" podStartSLOduration=3.716711847 podStartE2EDuration="24.381852768s" totalImagesPullingTime="20.665140921s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 09:13:04 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 09:13:28.381581772 +0000 UTC m=+24.722191981" watchObservedRunningTime="2026-09-11 09:13:28.381852768 +0000 UTC m=+24.722462970" Sep 11 09:13:29.109297 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:29.109267 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bb6x9" Sep 11 09:13:29.109537 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:29.109388 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bb6x9" podUID="92eb1c40-b8d0-428c-955a-c005e197f70a" Sep 11 09:13:30.113493 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:30.113405 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8ng6h" Sep 11 09:13:30.113913 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:30.113536 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8ng6h" podUID="7eddd8ad-0ac3-421d-9649-95dc4fadfd7b" Sep 11 09:13:30.113913 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:30.113597 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rlftv" Sep 11 09:13:30.113913 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:30.113746 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rlftv" podUID="abeb037b-2e12-4485-974d-35280f982f4a" Sep 11 09:13:31.109011 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:31.108847 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bb6x9" Sep 11 09:13:31.109110 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:31.109092 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bb6x9" podUID="92eb1c40-b8d0-428c-955a-c005e197f70a" Sep 11 09:13:31.369819 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:31.369783 2575 generic.go:356] "Generic (PLEG): container finished" podID="34c95ff5-365b-414d-9968-2db918292b7f" containerID="88a48e714e17f24d0db10e5069672d48ad62a8bc4f20101d6a31082daf10519a" exitCode=0 Sep 11 09:13:31.370267 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:31.369867 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8z7wv" event={"ID":"34c95ff5-365b-414d-9968-2db918292b7f","Type":"ContainerDied","Data":"88a48e714e17f24d0db10e5069672d48ad62a8bc4f20101d6a31082daf10519a"} Sep 11 09:13:31.373761 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:31.373736 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wgrlw" event={"ID":"3c0627e0-f086-4fc4-b2d7-724b67c685d4","Type":"ContainerStarted","Data":"5ef181786130d1f433315fd8cc6b37dfe3dba5bbd0d5588ff75e535f60653826"} Sep 11 09:13:31.373974 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:31.373954 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wgrlw" Sep 11 09:13:31.374068 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:31.374017 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wgrlw" Sep 11 09:13:31.388764 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:31.388736 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wgrlw" Sep 11 09:13:31.416618 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:31.416569 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-wgrlw" podStartSLOduration=9.82882785 podStartE2EDuration="27.416553226s" totalImagesPullingTime="17.587725376s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 09:13:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 09:13:31.415216573 +0000 UTC m=+27.755826773" watchObservedRunningTime="2026-09-11 09:13:31.416553226 +0000 UTC m=+27.757163437" Sep 11 09:13:32.109426 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:32.109390 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rlftv" Sep 11 09:13:32.109708 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:32.109425 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8ng6h" Sep 11 09:13:32.109708 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:32.109549 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rlftv" podUID="abeb037b-2e12-4485-974d-35280f982f4a" Sep 11 09:13:32.109708 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:32.109667 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8ng6h" podUID="7eddd8ad-0ac3-421d-9649-95dc4fadfd7b" Sep 11 09:13:32.376037 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:32.375953 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wgrlw" Sep 11 09:13:32.390147 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:32.390122 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wgrlw" Sep 11 09:13:33.109280 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:33.109242 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bb6x9" Sep 11 09:13:33.109457 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:33.109348 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bb6x9" podUID="92eb1c40-b8d0-428c-955a-c005e197f70a" Sep 11 09:13:33.380300 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:33.380216 2575 generic.go:356] "Generic (PLEG): container finished" podID="34c95ff5-365b-414d-9968-2db918292b7f" containerID="88a5e88799a11f3ee3ef0759168a82a910d7cb3b00c821bba133ecb1b4847a6f" exitCode=0 Sep 11 09:13:33.380300 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:33.380284 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8z7wv" event={"ID":"34c95ff5-365b-414d-9968-2db918292b7f","Type":"ContainerDied","Data":"88a5e88799a11f3ee3ef0759168a82a910d7cb3b00c821bba133ecb1b4847a6f"} Sep 11 09:13:34.109655 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:34.109615 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8ng6h" Sep 11 09:13:34.109847 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:34.109720 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8ng6h" podUID="7eddd8ad-0ac3-421d-9649-95dc4fadfd7b" Sep 11 09:13:34.109847 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:34.109766 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rlftv" Sep 11 09:13:34.109957 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:34.109863 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rlftv" podUID="abeb037b-2e12-4485-974d-35280f982f4a" Sep 11 09:13:35.108697 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:35.108660 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bb6x9" Sep 11 09:13:35.109157 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:35.108769 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bb6x9" podUID="92eb1c40-b8d0-428c-955a-c005e197f70a" Sep 11 09:13:35.387309 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:35.387213 2575 generic.go:356] "Generic (PLEG): container finished" podID="34c95ff5-365b-414d-9968-2db918292b7f" containerID="5aa52e430e65bc2e9dbaf60ee453bd498b5c584ade880d9f4b21e350aa0212f2" exitCode=0 Sep 11 09:13:35.387309 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:35.387261 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8z7wv" event={"ID":"34c95ff5-365b-414d-9968-2db918292b7f","Type":"ContainerDied","Data":"5aa52e430e65bc2e9dbaf60ee453bd498b5c584ade880d9f4b21e350aa0212f2"} Sep 11 09:13:36.108608 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:36.108564 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rlftv" Sep 11 09:13:36.108803 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:36.108586 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8ng6h" Sep 11 09:13:36.108803 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:36.108706 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rlftv" podUID="abeb037b-2e12-4485-974d-35280f982f4a" Sep 11 09:13:36.108803 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:36.108746 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8ng6h" podUID="7eddd8ad-0ac3-421d-9649-95dc4fadfd7b" Sep 11 09:13:37.108625 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:37.108592 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bb6x9" Sep 11 09:13:37.108800 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:37.108708 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bb6x9" podUID="92eb1c40-b8d0-428c-955a-c005e197f70a" Sep 11 09:13:37.748695 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:37.748641 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abeb037b-2e12-4485-974d-35280f982f4a-metrics-certs\") pod \"network-metrics-daemon-rlftv\" (UID: \"abeb037b-2e12-4485-974d-35280f982f4a\") " pod="openshift-multus/network-metrics-daemon-rlftv" Sep 11 09:13:37.749259 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:37.748805 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 09:13:37.749259 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:37.748881 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/abeb037b-2e12-4485-974d-35280f982f4a-metrics-certs podName:abeb037b-2e12-4485-974d-35280f982f4a nodeName:}" failed. No retries permitted until 2026-09-11 09:14:09.748862862 +0000 UTC m=+66.089473066 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/abeb037b-2e12-4485-974d-35280f982f4a-metrics-certs") pod "network-metrics-daemon-rlftv" (UID: "abeb037b-2e12-4485-974d-35280f982f4a") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 09:13:37.849052 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:37.849013 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7eddd8ad-0ac3-421d-9649-95dc4fadfd7b-original-pull-secret\") pod \"global-pull-secret-syncer-8ng6h\" (UID: \"7eddd8ad-0ac3-421d-9649-95dc4fadfd7b\") " pod="kube-system/global-pull-secret-syncer-8ng6h" Sep 11 09:13:37.849223 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:37.849189 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 09:13:37.849279 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:37.849272 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7eddd8ad-0ac3-421d-9649-95dc4fadfd7b-original-pull-secret podName:7eddd8ad-0ac3-421d-9649-95dc4fadfd7b nodeName:}" failed. No retries permitted until 2026-09-11 09:14:09.849251245 +0000 UTC m=+66.189861452 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7eddd8ad-0ac3-421d-9649-95dc4fadfd7b-original-pull-secret") pod "global-pull-secret-syncer-8ng6h" (UID: "7eddd8ad-0ac3-421d-9649-95dc4fadfd7b") : object "kube-system"/"original-pull-secret" not registered Sep 11 09:13:37.949457 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:37.949415 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slfls\" (UniqueName: \"kubernetes.io/projected/92eb1c40-b8d0-428c-955a-c005e197f70a-kube-api-access-slfls\") pod \"network-check-target-bb6x9\" (UID: \"92eb1c40-b8d0-428c-955a-c005e197f70a\") " pod="openshift-network-diagnostics/network-check-target-bb6x9" Sep 11 09:13:37.949634 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:37.949612 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 09:13:37.949705 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:37.949643 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 09:13:37.949705 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:37.949657 2575 projected.go:196] Error preparing data for projected volume kube-api-access-slfls for pod openshift-network-diagnostics/network-check-target-bb6x9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 09:13:37.949805 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:37.949715 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/92eb1c40-b8d0-428c-955a-c005e197f70a-kube-api-access-slfls podName:92eb1c40-b8d0-428c-955a-c005e197f70a nodeName:}" failed. No retries permitted until 2026-09-11 09:14:09.949702456 +0000 UTC m=+66.290312641 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-slfls" (UniqueName: "kubernetes.io/projected/92eb1c40-b8d0-428c-955a-c005e197f70a-kube-api-access-slfls") pod "network-check-target-bb6x9" (UID: "92eb1c40-b8d0-428c-955a-c005e197f70a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 09:13:38.109415 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:38.109362 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8ng6h" Sep 11 09:13:38.109624 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:38.109512 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8ng6h" podUID="7eddd8ad-0ac3-421d-9649-95dc4fadfd7b" Sep 11 09:13:38.109624 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:38.109561 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rlftv" Sep 11 09:13:38.109738 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:38.109677 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rlftv" podUID="abeb037b-2e12-4485-974d-35280f982f4a" Sep 11 09:13:39.109367 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:39.109331 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bb6x9" Sep 11 09:13:39.109815 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:39.109440 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bb6x9" podUID="92eb1c40-b8d0-428c-955a-c005e197f70a" Sep 11 09:13:40.109018 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:40.108990 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8ng6h" Sep 11 09:13:40.109140 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:40.109120 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8ng6h" podUID="7eddd8ad-0ac3-421d-9649-95dc4fadfd7b" Sep 11 09:13:40.109188 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:40.109160 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rlftv" Sep 11 09:13:40.109265 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:40.109249 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rlftv" podUID="abeb037b-2e12-4485-974d-35280f982f4a" Sep 11 09:13:40.402294 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:40.402192 2575 generic.go:356] "Generic (PLEG): container finished" podID="34c95ff5-365b-414d-9968-2db918292b7f" containerID="ab386d6899d4323e187a62ddd420fdf7aa16c85725dccbc06a81251916ec7afe" exitCode=0 Sep 11 09:13:40.402294 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:40.402253 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8z7wv" event={"ID":"34c95ff5-365b-414d-9968-2db918292b7f","Type":"ContainerDied","Data":"ab386d6899d4323e187a62ddd420fdf7aa16c85725dccbc06a81251916ec7afe"} Sep 11 09:13:41.109250 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:41.109217 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bb6x9" Sep 11 09:13:41.109427 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:41.109325 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bb6x9" podUID="92eb1c40-b8d0-428c-955a-c005e197f70a" Sep 11 09:13:41.408285 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:41.408200 2575 generic.go:356] "Generic (PLEG): container finished" podID="34c95ff5-365b-414d-9968-2db918292b7f" containerID="1635659e21eb04a79aed41fe23377ae14956bdb6f8d8593ac6536f7ab3c4b573" exitCode=0 Sep 11 09:13:41.408285 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:41.408247 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8z7wv" event={"ID":"34c95ff5-365b-414d-9968-2db918292b7f","Type":"ContainerDied","Data":"1635659e21eb04a79aed41fe23377ae14956bdb6f8d8593ac6536f7ab3c4b573"} Sep 11 09:13:42.109069 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:42.108881 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rlftv" Sep 11 09:13:42.109263 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:42.108935 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8ng6h" Sep 11 09:13:42.109263 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:42.109149 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rlftv" podUID="abeb037b-2e12-4485-974d-35280f982f4a" Sep 11 09:13:42.109263 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:42.109226 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8ng6h" podUID="7eddd8ad-0ac3-421d-9649-95dc4fadfd7b" Sep 11 09:13:42.413665 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:42.413585 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8z7wv" event={"ID":"34c95ff5-365b-414d-9968-2db918292b7f","Type":"ContainerStarted","Data":"59d12ef68a9392e2c989f90c7d4a308a90b55ed71eb4151915ba7e6d0aa9fa14"} Sep 11 09:13:42.437340 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:42.437289 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-8z7wv" podStartSLOduration=8.343229214 podStartE2EDuration="38.437273828s" totalImagesPullingTime="30.094044614s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 09:13:04 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 09:13:42.436727555 +0000 UTC m=+38.777337760" watchObservedRunningTime="2026-09-11 09:13:42.437273828 +0000 UTC m=+38.777884033" Sep 11 09:13:43.108662 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:43.108627 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bb6x9" Sep 11 09:13:43.108842 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:43.108735 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bb6x9" podUID="92eb1c40-b8d0-428c-955a-c005e197f70a" Sep 11 09:13:44.110094 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:44.110052 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rlftv" Sep 11 09:13:44.110509 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:44.110150 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rlftv" podUID="abeb037b-2e12-4485-974d-35280f982f4a" Sep 11 09:13:44.110509 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:44.110234 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8ng6h" Sep 11 09:13:44.110509 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:44.110345 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8ng6h" podUID="7eddd8ad-0ac3-421d-9649-95dc4fadfd7b" Sep 11 09:13:45.109083 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:45.109048 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bb6x9" Sep 11 09:13:45.109262 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:45.109146 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bb6x9" podUID="92eb1c40-b8d0-428c-955a-c005e197f70a" Sep 11 09:13:46.109127 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:46.109081 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rlftv" Sep 11 09:13:46.109127 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:46.109100 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8ng6h" Sep 11 09:13:46.109693 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:46.109193 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rlftv" podUID="abeb037b-2e12-4485-974d-35280f982f4a" Sep 11 09:13:46.109693 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:46.109314 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8ng6h" podUID="7eddd8ad-0ac3-421d-9649-95dc4fadfd7b" Sep 11 09:13:47.109168 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:47.109136 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bb6x9" Sep 11 09:13:47.109573 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:47.109246 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bb6x9" podUID="92eb1c40-b8d0-428c-955a-c005e197f70a" Sep 11 09:13:48.108733 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:48.108699 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8ng6h" Sep 11 09:13:48.108919 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:48.108739 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rlftv" Sep 11 09:13:48.108919 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:48.108833 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8ng6h" podUID="7eddd8ad-0ac3-421d-9649-95dc4fadfd7b" Sep 11 09:13:48.109030 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:48.108964 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rlftv" podUID="abeb037b-2e12-4485-974d-35280f982f4a" Sep 11 09:13:49.108792 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:49.108754 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bb6x9" Sep 11 09:13:49.109241 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:49.108873 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bb6x9" podUID="92eb1c40-b8d0-428c-955a-c005e197f70a" Sep 11 09:13:49.152261 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:49.152175 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-bb6x9"] Sep 11 09:13:49.158731 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:49.158700 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-rlftv"] Sep 11 09:13:49.158880 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:49.158809 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rlftv" Sep 11 09:13:49.158971 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:49.158942 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rlftv" podUID="abeb037b-2e12-4485-974d-35280f982f4a" Sep 11 09:13:49.163306 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:49.163281 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-8ng6h"] Sep 11 09:13:49.163408 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:49.163375 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8ng6h" Sep 11 09:13:49.163518 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:49.163495 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8ng6h" podUID="7eddd8ad-0ac3-421d-9649-95dc4fadfd7b" Sep 11 09:13:49.428941 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:49.428858 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bb6x9" Sep 11 09:13:49.429094 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:49.428963 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bb6x9" podUID="92eb1c40-b8d0-428c-955a-c005e197f70a" Sep 11 09:13:51.108480 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:51.108441 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8ng6h" Sep 11 09:13:51.108945 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:51.108508 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bb6x9" Sep 11 09:13:51.108945 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:51.108568 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rlftv" Sep 11 09:13:51.108945 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:51.108593 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8ng6h" podUID="7eddd8ad-0ac3-421d-9649-95dc4fadfd7b" Sep 11 09:13:51.108945 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:51.108643 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rlftv" podUID="abeb037b-2e12-4485-974d-35280f982f4a" Sep 11 09:13:51.108945 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:51.108698 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bb6x9" podUID="92eb1c40-b8d0-428c-955a-c005e197f70a" Sep 11 09:13:53.108612 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:53.108584 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8ng6h" Sep 11 09:13:53.108984 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:53.108583 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rlftv" Sep 11 09:13:53.108984 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:53.108685 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8ng6h" podUID="7eddd8ad-0ac3-421d-9649-95dc4fadfd7b" Sep 11 09:13:53.108984 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:53.108588 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bb6x9" Sep 11 09:13:53.108984 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:53.108764 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rlftv" podUID="abeb037b-2e12-4485-974d-35280f982f4a" Sep 11 09:13:53.108984 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:13:53.108806 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bb6x9" podUID="92eb1c40-b8d0-428c-955a-c005e197f70a" Sep 11 09:13:53.497025 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:53.496851 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-119.ec2.internal" event="NodeReady" Sep 11 09:13:53.497177 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:53.497124 2575 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 11 09:13:53.549071 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:53.549036 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-vb5ck"] Sep 11 09:13:53.554246 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:53.554223 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-znq28"] Sep 11 09:13:53.554374 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:53.554333 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vb5ck" Sep 11 09:13:53.558435 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:53.558408 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 11 09:13:53.558739 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:53.558715 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 11 09:13:53.559003 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:53.558723 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-mk8zg\"" Sep 11 09:13:53.564333 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:53.564302 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-vb5ck"] Sep 11 09:13:53.564493 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:53.564423 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-znq28" Sep 11 09:13:53.564724 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:53.564580 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-znq28"] Sep 11 09:13:53.567935 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:53.567911 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 11 09:13:53.568098 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:53.568083 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 11 09:13:53.568674 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:53.568634 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-fbjv9\"" Sep 11 09:13:53.568764 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:53.568714 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 11 09:13:53.568764 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:53.568723 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 11 09:13:53.646789 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:53.646708 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-bsj85"] Sep 11 09:13:53.654214 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:53.654193 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v85kz\" (UniqueName: \"kubernetes.io/projected/54fff86e-d33b-4b2b-bedf-b00daa2f301c-kube-api-access-v85kz\") pod \"dns-default-vb5ck\" (UID: \"54fff86e-d33b-4b2b-bedf-b00daa2f301c\") " pod="openshift-dns/dns-default-vb5ck" Sep 11 09:13:53.654298 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:53.654221 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/54fff86e-d33b-4b2b-bedf-b00daa2f301c-config-volume\") pod \"dns-default-vb5ck\" (UID: \"54fff86e-d33b-4b2b-bedf-b00daa2f301c\") " pod="openshift-dns/dns-default-vb5ck" Sep 11 09:13:53.654298 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:53.654239 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/31745858-acda-4939-97ef-b5d9f6ff710a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-znq28\" (UID: \"31745858-acda-4939-97ef-b5d9f6ff710a\") " pod="openshift-insights/insights-runtime-extractor-znq28" Sep 11 09:13:53.654298 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:53.654262 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/31745858-acda-4939-97ef-b5d9f6ff710a-crio-socket\") pod \"insights-runtime-extractor-znq28\" (UID: \"31745858-acda-4939-97ef-b5d9f6ff710a\") " pod="openshift-insights/insights-runtime-extractor-znq28" Sep 11 09:13:53.654444 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:53.654345 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/31745858-acda-4939-97ef-b5d9f6ff710a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-znq28\" (UID: \"31745858-acda-4939-97ef-b5d9f6ff710a\") " pod="openshift-insights/insights-runtime-extractor-znq28" Sep 11 09:13:53.654444 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:53.654377 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/31745858-acda-4939-97ef-b5d9f6ff710a-data-volume\") pod \"insights-runtime-extractor-znq28\" (UID: \"31745858-acda-4939-97ef-b5d9f6ff710a\") " pod="openshift-insights/insights-runtime-extractor-znq28" Sep 11 09:13:53.654444 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:53.654431 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/54fff86e-d33b-4b2b-bedf-b00daa2f301c-metrics-tls\") pod \"dns-default-vb5ck\" (UID: \"54fff86e-d33b-4b2b-bedf-b00daa2f301c\") " pod="openshift-dns/dns-default-vb5ck" Sep 11 09:13:53.654586 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:53.654458 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54fff86e-d33b-4b2b-bedf-b00daa2f301c-tmp-dir\") pod \"dns-default-vb5ck\" (UID: \"54fff86e-d33b-4b2b-bedf-b00daa2f301c\") " pod="openshift-dns/dns-default-vb5ck" Sep 11 09:13:53.654586 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:53.654502 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2jnb\" (UniqueName: \"kubernetes.io/projected/31745858-acda-4939-97ef-b5d9f6ff710a-kube-api-access-d2jnb\") pod \"insights-runtime-extractor-znq28\" (UID: \"31745858-acda-4939-97ef-b5d9f6ff710a\") " pod="openshift-insights/insights-runtime-extractor-znq28" Sep 11 09:13:53.660242 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:53.660221 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-bsj85"] Sep 11 09:13:53.660348 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:53.660303 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-bsj85" Sep 11 09:13:53.663202 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:53.663183 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 11 09:13:53.663324 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:53.663200 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-twxj8\"" Sep 11 09:13:53.663324 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:53.663314 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 11 09:13:53.663429 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:53.663364 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 11 09:13:53.754849 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:53.754808 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2jnb\" (UniqueName: \"kubernetes.io/projected/31745858-acda-4939-97ef-b5d9f6ff710a-kube-api-access-d2jnb\") pod \"insights-runtime-extractor-znq28\" (UID: \"31745858-acda-4939-97ef-b5d9f6ff710a\") " pod="openshift-insights/insights-runtime-extractor-znq28" Sep 11 09:13:53.755005 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:53.754859 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4e0aace6-bbd9-4cbb-8759-165ffbaa3b2a-cert\") pod \"ingress-canary-bsj85\" (UID: \"4e0aace6-bbd9-4cbb-8759-165ffbaa3b2a\") " pod="openshift-ingress-canary/ingress-canary-bsj85" Sep 11 09:13:53.755005 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:53.754977 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v85kz\" (UniqueName: \"kubernetes.io/projected/54fff86e-d33b-4b2b-bedf-b00daa2f301c-kube-api-access-v85kz\") pod \"dns-default-vb5ck\" (UID: \"54fff86e-d33b-4b2b-bedf-b00daa2f301c\") " pod="openshift-dns/dns-default-vb5ck" Sep 11 09:13:53.755005 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:53.755002 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/54fff86e-d33b-4b2b-bedf-b00daa2f301c-config-volume\") pod \"dns-default-vb5ck\" (UID: \"54fff86e-d33b-4b2b-bedf-b00daa2f301c\") " pod="openshift-dns/dns-default-vb5ck" Sep 11 09:13:53.755130 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:53.755024 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/31745858-acda-4939-97ef-b5d9f6ff710a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-znq28\" (UID: \"31745858-acda-4939-97ef-b5d9f6ff710a\") " pod="openshift-insights/insights-runtime-extractor-znq28" Sep 11 09:13:53.755269 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:53.755240 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/31745858-acda-4939-97ef-b5d9f6ff710a-crio-socket\") pod \"insights-runtime-extractor-znq28\" (UID: \"31745858-acda-4939-97ef-b5d9f6ff710a\") " pod="openshift-insights/insights-runtime-extractor-znq28" Sep 11 09:13:53.755399 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:53.755306 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/31745858-acda-4939-97ef-b5d9f6ff710a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-znq28\" (UID: \"31745858-acda-4939-97ef-b5d9f6ff710a\") " pod="openshift-insights/insights-runtime-extractor-znq28" Sep 11 09:13:53.755399 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:53.755337 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwd5t\" (UniqueName: \"kubernetes.io/projected/4e0aace6-bbd9-4cbb-8759-165ffbaa3b2a-kube-api-access-cwd5t\") pod \"ingress-canary-bsj85\" (UID: \"4e0aace6-bbd9-4cbb-8759-165ffbaa3b2a\") " pod="openshift-ingress-canary/ingress-canary-bsj85" Sep 11 09:13:53.755399 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:53.755376 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/31745858-acda-4939-97ef-b5d9f6ff710a-data-volume\") pod \"insights-runtime-extractor-znq28\" (UID: \"31745858-acda-4939-97ef-b5d9f6ff710a\") " pod="openshift-insights/insights-runtime-extractor-znq28" Sep 11 09:13:53.755579 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:53.755417 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/54fff86e-d33b-4b2b-bedf-b00daa2f301c-metrics-tls\") pod \"dns-default-vb5ck\" (UID: \"54fff86e-d33b-4b2b-bedf-b00daa2f301c\") " pod="openshift-dns/dns-default-vb5ck" Sep 11 09:13:53.755579 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:53.755440 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54fff86e-d33b-4b2b-bedf-b00daa2f301c-tmp-dir\") pod \"dns-default-vb5ck\" (UID: \"54fff86e-d33b-4b2b-bedf-b00daa2f301c\") " pod="openshift-dns/dns-default-vb5ck" Sep 11 09:13:53.755579 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:53.755539 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/31745858-acda-4939-97ef-b5d9f6ff710a-crio-socket\") pod \"insights-runtime-extractor-znq28\" (UID: \"31745858-acda-4939-97ef-b5d9f6ff710a\") " pod="openshift-insights/insights-runtime-extractor-znq28" Sep 11 09:13:53.755691 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:53.755662 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/31745858-acda-4939-97ef-b5d9f6ff710a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-znq28\" (UID: \"31745858-acda-4939-97ef-b5d9f6ff710a\") " pod="openshift-insights/insights-runtime-extractor-znq28" Sep 11 09:13:53.755746 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:53.755727 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/31745858-acda-4939-97ef-b5d9f6ff710a-data-volume\") pod \"insights-runtime-extractor-znq28\" (UID: \"31745858-acda-4939-97ef-b5d9f6ff710a\") " pod="openshift-insights/insights-runtime-extractor-znq28" Sep 11 09:13:53.755781 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:53.755735 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/54fff86e-d33b-4b2b-bedf-b00daa2f301c-config-volume\") pod \"dns-default-vb5ck\" (UID: \"54fff86e-d33b-4b2b-bedf-b00daa2f301c\") " pod="openshift-dns/dns-default-vb5ck" Sep 11 09:13:53.759276 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:53.759256 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/31745858-acda-4939-97ef-b5d9f6ff710a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-znq28\" (UID: \"31745858-acda-4939-97ef-b5d9f6ff710a\") " pod="openshift-insights/insights-runtime-extractor-znq28" Sep 11 09:13:53.761666 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:53.761646 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54fff86e-d33b-4b2b-bedf-b00daa2f301c-tmp-dir\") pod \"dns-default-vb5ck\" (UID: \"54fff86e-d33b-4b2b-bedf-b00daa2f301c\") " pod="openshift-dns/dns-default-vb5ck" Sep 11 09:13:53.763687 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:53.763668 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2jnb\" (UniqueName: \"kubernetes.io/projected/31745858-acda-4939-97ef-b5d9f6ff710a-kube-api-access-d2jnb\") pod \"insights-runtime-extractor-znq28\" (UID: \"31745858-acda-4939-97ef-b5d9f6ff710a\") " pod="openshift-insights/insights-runtime-extractor-znq28" Sep 11 09:13:53.763831 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:53.763813 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/54fff86e-d33b-4b2b-bedf-b00daa2f301c-metrics-tls\") pod \"dns-default-vb5ck\" (UID: \"54fff86e-d33b-4b2b-bedf-b00daa2f301c\") " pod="openshift-dns/dns-default-vb5ck" Sep 11 09:13:53.763883 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:53.763817 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v85kz\" (UniqueName: \"kubernetes.io/projected/54fff86e-d33b-4b2b-bedf-b00daa2f301c-kube-api-access-v85kz\") pod \"dns-default-vb5ck\" (UID: \"54fff86e-d33b-4b2b-bedf-b00daa2f301c\") " pod="openshift-dns/dns-default-vb5ck" Sep 11 09:13:53.856280 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:53.856246 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwd5t\" (UniqueName: \"kubernetes.io/projected/4e0aace6-bbd9-4cbb-8759-165ffbaa3b2a-kube-api-access-cwd5t\") pod \"ingress-canary-bsj85\" (UID: \"4e0aace6-bbd9-4cbb-8759-165ffbaa3b2a\") " pod="openshift-ingress-canary/ingress-canary-bsj85" Sep 11 09:13:53.856445 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:53.856318 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4e0aace6-bbd9-4cbb-8759-165ffbaa3b2a-cert\") pod \"ingress-canary-bsj85\" (UID: \"4e0aace6-bbd9-4cbb-8759-165ffbaa3b2a\") " pod="openshift-ingress-canary/ingress-canary-bsj85" Sep 11 09:13:53.858641 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:53.858621 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4e0aace6-bbd9-4cbb-8759-165ffbaa3b2a-cert\") pod \"ingress-canary-bsj85\" (UID: \"4e0aace6-bbd9-4cbb-8759-165ffbaa3b2a\") " pod="openshift-ingress-canary/ingress-canary-bsj85" Sep 11 09:13:53.866114 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:53.866092 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vb5ck" Sep 11 09:13:53.868351 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:53.868324 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwd5t\" (UniqueName: \"kubernetes.io/projected/4e0aace6-bbd9-4cbb-8759-165ffbaa3b2a-kube-api-access-cwd5t\") pod \"ingress-canary-bsj85\" (UID: \"4e0aace6-bbd9-4cbb-8759-165ffbaa3b2a\") " pod="openshift-ingress-canary/ingress-canary-bsj85" Sep 11 09:13:53.873993 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:53.873972 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-znq28" Sep 11 09:13:53.968821 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:53.968796 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-bsj85" Sep 11 09:13:54.173774 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:54.173729 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-vb5ck"] Sep 11 09:13:54.174509 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:54.174409 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-znq28"] Sep 11 09:13:54.178138 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:54.178102 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31745858_acda_4939_97ef_b5d9f6ff710a.slice/crio-238387ea4757f84d9367cdba5c1d5b0f63dc0f2bee4c8a068223c26b609af260 WatchSource:0}: Error finding container 238387ea4757f84d9367cdba5c1d5b0f63dc0f2bee4c8a068223c26b609af260: Status 404 returned error can't find the container with id 238387ea4757f84d9367cdba5c1d5b0f63dc0f2bee4c8a068223c26b609af260 Sep 11 09:13:54.255292 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:54.255098 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-bsj85"] Sep 11 09:13:54.260771 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:13:54.260745 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e0aace6_bbd9_4cbb_8759_165ffbaa3b2a.slice/crio-cf950b3f9a7d7d6ea948cd82f0a3d3deb7acedd7d70f2189f653151b00842dfd WatchSource:0}: Error finding container cf950b3f9a7d7d6ea948cd82f0a3d3deb7acedd7d70f2189f653151b00842dfd: Status 404 returned error can't find the container with id cf950b3f9a7d7d6ea948cd82f0a3d3deb7acedd7d70f2189f653151b00842dfd Sep 11 09:13:54.442330 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:54.442294 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-bsj85" event={"ID":"4e0aace6-bbd9-4cbb-8759-165ffbaa3b2a","Type":"ContainerStarted","Data":"cf950b3f9a7d7d6ea948cd82f0a3d3deb7acedd7d70f2189f653151b00842dfd"} Sep 11 09:13:54.443816 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:54.443784 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-znq28" event={"ID":"31745858-acda-4939-97ef-b5d9f6ff710a","Type":"ContainerStarted","Data":"ecc419d964d37ad7da0b37314744b5b05a878d245e82a1ba28a0d7558f7a280c"} Sep 11 09:13:54.443946 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:54.443844 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-znq28" event={"ID":"31745858-acda-4939-97ef-b5d9f6ff710a","Type":"ContainerStarted","Data":"238387ea4757f84d9367cdba5c1d5b0f63dc0f2bee4c8a068223c26b609af260"} Sep 11 09:13:54.444913 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:54.444893 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vb5ck" event={"ID":"54fff86e-d33b-4b2b-bedf-b00daa2f301c","Type":"ContainerStarted","Data":"51e5d5e87712c0635557387d749ea21f4e231df2a9577fce7536255e9498825f"} Sep 11 09:13:55.108485 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:55.108435 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rlftv" Sep 11 09:13:55.108676 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:55.108443 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8ng6h" Sep 11 09:13:55.108676 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:55.108443 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bb6x9" Sep 11 09:13:55.111612 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:55.111588 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 11 09:13:55.112674 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:55.112463 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-ggqqp\"" Sep 11 09:13:55.112674 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:55.112544 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 11 09:13:55.112674 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:55.112564 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 11 09:13:55.112674 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:55.112484 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-qfqbz\"" Sep 11 09:13:55.112674 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:55.112604 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 11 09:13:57.454527 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:57.454406 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-bsj85" event={"ID":"4e0aace6-bbd9-4cbb-8759-165ffbaa3b2a","Type":"ContainerStarted","Data":"c0e4c604f23abce1dd4b73562fa7033f87b1ebf0b39c408d8641da66b2806dad"} Sep 11 09:13:57.456180 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:57.456155 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-znq28" event={"ID":"31745858-acda-4939-97ef-b5d9f6ff710a","Type":"ContainerStarted","Data":"08529fbc75354ea76edf9d02fde37001676724ac1e62509ce06b2024c58c3776"} Sep 11 09:13:57.457811 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:57.457791 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vb5ck" event={"ID":"54fff86e-d33b-4b2b-bedf-b00daa2f301c","Type":"ContainerStarted","Data":"6ce344f3077f4eebf950f34bf3bf746331fdb2cc49b3eea1daa9b014aa46182f"} Sep 11 09:13:57.457898 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:57.457828 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vb5ck" event={"ID":"54fff86e-d33b-4b2b-bedf-b00daa2f301c","Type":"ContainerStarted","Data":"a6440b971eaaf03ceb11843f64fecdd49c854a8589a497eec6a30a847d9db914"} Sep 11 09:13:57.457967 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:57.457946 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-vb5ck" Sep 11 09:13:57.471028 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:57.470977 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-bsj85" podStartSLOduration=1.601915869 podStartE2EDuration="4.470960474s" totalImagesPullingTime="2.869044605s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 09:13:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 09:13:57.470762937 +0000 UTC m=+53.811373141" watchObservedRunningTime="2026-09-11 09:13:57.470960474 +0000 UTC m=+53.811570679" Sep 11 09:13:57.489214 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:57.489162 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-vb5ck" podStartSLOduration=1.558977567 podStartE2EDuration="4.489144879s" totalImagesPullingTime="2.930167312s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 09:13:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 09:13:57.488349734 +0000 UTC m=+53.828959940" watchObservedRunningTime="2026-09-11 09:13:57.489144879 +0000 UTC m=+53.829755082" Sep 11 09:13:59.465700 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:59.465667 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-znq28" event={"ID":"31745858-acda-4939-97ef-b5d9f6ff710a","Type":"ContainerStarted","Data":"91c0f60a0019083093562b88b65617f57f6c56b714d630e796310bd601fded8e"} Sep 11 09:13:59.488789 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:13:59.488738 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-znq28" podStartSLOduration=2.326490715 podStartE2EDuration="6.488720784s" totalImagesPullingTime="4.162230069s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 09:13:53 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 09:13:59.48824866 +0000 UTC m=+55.828858866" watchObservedRunningTime="2026-09-11 09:13:59.488720784 +0000 UTC m=+55.829330990" Sep 11 09:14:01.991890 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:01.991853 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-4xxgk"] Sep 11 09:14:01.996707 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:01.996683 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-4xxgk" Sep 11 09:14:01.999221 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:01.999192 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Sep 11 09:14:01.999221 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:01.999211 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 11 09:14:01.999508 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:01.999493 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 11 09:14:02.000408 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:02.000394 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-5f5r4\"" Sep 11 09:14:02.000546 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:02.000527 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 11 09:14:02.000616 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:02.000601 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Sep 11 09:14:02.007164 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:02.007140 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-4xxgk"] Sep 11 09:14:02.021640 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:02.021618 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/5cc1de09-5625-4d17-a546-1c273a1183ba-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-4xxgk\" (UID: \"5cc1de09-5625-4d17-a546-1c273a1183ba\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-4xxgk" Sep 11 09:14:02.021745 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:02.021673 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5cc1de09-5625-4d17-a546-1c273a1183ba-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-4xxgk\" (UID: \"5cc1de09-5625-4d17-a546-1c273a1183ba\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-4xxgk" Sep 11 09:14:02.021795 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:02.021749 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqsts\" (UniqueName: \"kubernetes.io/projected/5cc1de09-5625-4d17-a546-1c273a1183ba-kube-api-access-wqsts\") pod \"prometheus-operator-68bcd7f987-4xxgk\" (UID: \"5cc1de09-5625-4d17-a546-1c273a1183ba\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-4xxgk" Sep 11 09:14:02.021835 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:02.021814 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5cc1de09-5625-4d17-a546-1c273a1183ba-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-4xxgk\" (UID: \"5cc1de09-5625-4d17-a546-1c273a1183ba\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-4xxgk" Sep 11 09:14:02.122684 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:02.122651 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5cc1de09-5625-4d17-a546-1c273a1183ba-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-4xxgk\" (UID: \"5cc1de09-5625-4d17-a546-1c273a1183ba\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-4xxgk" Sep 11 09:14:02.122867 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:02.122712 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqsts\" (UniqueName: \"kubernetes.io/projected/5cc1de09-5625-4d17-a546-1c273a1183ba-kube-api-access-wqsts\") pod \"prometheus-operator-68bcd7f987-4xxgk\" (UID: \"5cc1de09-5625-4d17-a546-1c273a1183ba\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-4xxgk" Sep 11 09:14:02.122867 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:02.122757 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5cc1de09-5625-4d17-a546-1c273a1183ba-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-4xxgk\" (UID: \"5cc1de09-5625-4d17-a546-1c273a1183ba\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-4xxgk" Sep 11 09:14:02.122867 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:02.122797 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/5cc1de09-5625-4d17-a546-1c273a1183ba-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-4xxgk\" (UID: \"5cc1de09-5625-4d17-a546-1c273a1183ba\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-4xxgk" Sep 11 09:14:02.123457 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:02.123431 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5cc1de09-5625-4d17-a546-1c273a1183ba-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-4xxgk\" (UID: \"5cc1de09-5625-4d17-a546-1c273a1183ba\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-4xxgk" Sep 11 09:14:02.126312 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:02.126290 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/5cc1de09-5625-4d17-a546-1c273a1183ba-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-4xxgk\" (UID: \"5cc1de09-5625-4d17-a546-1c273a1183ba\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-4xxgk" Sep 11 09:14:02.126312 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:02.126301 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5cc1de09-5625-4d17-a546-1c273a1183ba-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-4xxgk\" (UID: \"5cc1de09-5625-4d17-a546-1c273a1183ba\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-4xxgk" Sep 11 09:14:02.131841 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:02.131820 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqsts\" (UniqueName: \"kubernetes.io/projected/5cc1de09-5625-4d17-a546-1c273a1183ba-kube-api-access-wqsts\") pod \"prometheus-operator-68bcd7f987-4xxgk\" (UID: \"5cc1de09-5625-4d17-a546-1c273a1183ba\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-4xxgk" Sep 11 09:14:02.305601 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:02.305571 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-4xxgk" Sep 11 09:14:02.616821 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:02.616752 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-4xxgk"] Sep 11 09:14:02.619843 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:14:02.619814 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cc1de09_5625_4d17_a546_1c273a1183ba.slice/crio-495abe7cbe45070b8ef5b86bdb1bec484c2469cc88d2ff7160bcd034eb5c2055 WatchSource:0}: Error finding container 495abe7cbe45070b8ef5b86bdb1bec484c2469cc88d2ff7160bcd034eb5c2055: Status 404 returned error can't find the container with id 495abe7cbe45070b8ef5b86bdb1bec484c2469cc88d2ff7160bcd034eb5c2055 Sep 11 09:14:03.480360 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:03.480322 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-4xxgk" event={"ID":"5cc1de09-5625-4d17-a546-1c273a1183ba","Type":"ContainerStarted","Data":"495abe7cbe45070b8ef5b86bdb1bec484c2469cc88d2ff7160bcd034eb5c2055"} Sep 11 09:14:04.393805 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:04.393777 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wgrlw" Sep 11 09:14:04.485052 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:04.485017 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-4xxgk" event={"ID":"5cc1de09-5625-4d17-a546-1c273a1183ba","Type":"ContainerStarted","Data":"4874aa89b2e91cc804ea6c8188b2932f736b6701fc2d97f2fcf7eb572489f8c1"} Sep 11 09:14:04.485632 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:04.485073 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-4xxgk" event={"ID":"5cc1de09-5625-4d17-a546-1c273a1183ba","Type":"ContainerStarted","Data":"7c1ab9477d7c8224c15ae72e922fcb1c30662917c07d28151a9f3e0f41ba4e46"} Sep 11 09:14:04.508182 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:04.508134 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-68bcd7f987-4xxgk" podStartSLOduration=2.315645433 podStartE2EDuration="3.508117779s" totalImagesPullingTime="1.192472346s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 09:14:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 09:14:04.507263 +0000 UTC m=+60.847873207" watchObservedRunningTime="2026-09-11 09:14:04.508117779 +0000 UTC m=+60.848727985" Sep 11 09:14:06.334623 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:06.334575 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-rbngs"] Sep 11 09:14:06.337531 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:06.337514 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-rbngs" Sep 11 09:14:06.340997 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:06.340969 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-4xdpg"] Sep 11 09:14:06.342643 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:06.342621 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 11 09:14:06.342643 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:06.342639 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 11 09:14:06.342835 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:06.342806 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-pft5k"] Sep 11 09:14:06.342923 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:06.342908 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-4xdpg" Sep 11 09:14:06.344135 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:06.344089 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 11 09:14:06.344380 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:06.344350 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-bvms2\"" Sep 11 09:14:06.344940 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:06.344921 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-pft5k" Sep 11 09:14:06.345821 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:06.345435 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Sep 11 09:14:06.345821 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:06.345435 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Sep 11 09:14:06.345958 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:06.345939 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-zsdwj\"" Sep 11 09:14:06.349183 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:06.349167 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Sep 11 09:14:06.349355 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:06.349340 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-x8ddl\"" Sep 11 09:14:06.349406 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:06.349377 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Sep 11 09:14:06.349630 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:06.349613 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Sep 11 09:14:06.352894 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:06.352876 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rjnc\" (UniqueName: \"kubernetes.io/projected/fad3f414-87a1-4ae6-a3e4-62346dfa62e4-kube-api-access-8rjnc\") pod \"node-exporter-rbngs\" (UID: \"fad3f414-87a1-4ae6-a3e4-62346dfa62e4\") " pod="openshift-monitoring/node-exporter-rbngs" Sep 11 09:14:06.352975 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:06.352906 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fad3f414-87a1-4ae6-a3e4-62346dfa62e4-sys\") pod \"node-exporter-rbngs\" (UID: \"fad3f414-87a1-4ae6-a3e4-62346dfa62e4\") " pod="openshift-monitoring/node-exporter-rbngs" Sep 11 09:14:06.352975 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:06.352922 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/fad3f414-87a1-4ae6-a3e4-62346dfa62e4-node-exporter-wtmp\") pod \"node-exporter-rbngs\" (UID: \"fad3f414-87a1-4ae6-a3e4-62346dfa62e4\") " pod="openshift-monitoring/node-exporter-rbngs" Sep 11 09:14:06.352975 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:06.352937 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fad3f414-87a1-4ae6-a3e4-62346dfa62e4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rbngs\" (UID: \"fad3f414-87a1-4ae6-a3e4-62346dfa62e4\") " pod="openshift-monitoring/node-exporter-rbngs" Sep 11 09:14:06.353134 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:06.352994 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/fad3f414-87a1-4ae6-a3e4-62346dfa62e4-root\") pod \"node-exporter-rbngs\" (UID: \"fad3f414-87a1-4ae6-a3e4-62346dfa62e4\") " pod="openshift-monitoring/node-exporter-rbngs" Sep 11 09:14:06.353134 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:06.353063 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/fad3f414-87a1-4ae6-a3e4-62346dfa62e4-node-exporter-textfile\") pod \"node-exporter-rbngs\" (UID: \"fad3f414-87a1-4ae6-a3e4-62346dfa62e4\") " pod="openshift-monitoring/node-exporter-rbngs" Sep 11 09:14:06.353134 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:06.353090 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fad3f414-87a1-4ae6-a3e4-62346dfa62e4-node-exporter-tls\") pod \"node-exporter-rbngs\" (UID: \"fad3f414-87a1-4ae6-a3e4-62346dfa62e4\") " pod="openshift-monitoring/node-exporter-rbngs" Sep 11 09:14:06.353273 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:06.353133 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fad3f414-87a1-4ae6-a3e4-62346dfa62e4-metrics-client-ca\") pod \"node-exporter-rbngs\" (UID: \"fad3f414-87a1-4ae6-a3e4-62346dfa62e4\") " pod="openshift-monitoring/node-exporter-rbngs" Sep 11 09:14:06.353273 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:06.353173 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/fad3f414-87a1-4ae6-a3e4-62346dfa62e4-node-exporter-accelerators-collector-config\") pod \"node-exporter-rbngs\" (UID: \"fad3f414-87a1-4ae6-a3e4-62346dfa62e4\") " pod="openshift-monitoring/node-exporter-rbngs" Sep 11 09:14:06.358845 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:06.358824 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-4xdpg"] Sep 11 09:14:06.361544 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:06.361523 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-pft5k"] Sep 11 09:14:06.454271 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:06.454233 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8n5q\" (UniqueName: \"kubernetes.io/projected/2c1e552c-df45-4843-8717-37ee9b7d500f-kube-api-access-g8n5q\") pod \"kube-state-metrics-59fbdd77d-pft5k\" (UID: \"2c1e552c-df45-4843-8717-37ee9b7d500f\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-pft5k" Sep 11 09:14:06.454271 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:06.454272 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/fad3f414-87a1-4ae6-a3e4-62346dfa62e4-node-exporter-wtmp\") pod \"node-exporter-rbngs\" (UID: \"fad3f414-87a1-4ae6-a3e4-62346dfa62e4\") " pod="openshift-monitoring/node-exporter-rbngs" Sep 11 09:14:06.454511 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:06.454300 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/fad3f414-87a1-4ae6-a3e4-62346dfa62e4-root\") pod \"node-exporter-rbngs\" (UID: \"fad3f414-87a1-4ae6-a3e4-62346dfa62e4\") " pod="openshift-monitoring/node-exporter-rbngs" Sep 11 09:14:06.454511 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:06.454317 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rjnc\" (UniqueName: \"kubernetes.io/projected/fad3f414-87a1-4ae6-a3e4-62346dfa62e4-kube-api-access-8rjnc\") pod \"node-exporter-rbngs\" (UID: \"fad3f414-87a1-4ae6-a3e4-62346dfa62e4\") " pod="openshift-monitoring/node-exporter-rbngs" Sep 11 09:14:06.454511 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:06.454370 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/fad3f414-87a1-4ae6-a3e4-62346dfa62e4-root\") pod \"node-exporter-rbngs\" (UID: \"fad3f414-87a1-4ae6-a3e4-62346dfa62e4\") " pod="openshift-monitoring/node-exporter-rbngs" Sep 11 09:14:06.454511 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:06.454416 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/fad3f414-87a1-4ae6-a3e4-62346dfa62e4-node-exporter-wtmp\") pod \"node-exporter-rbngs\" (UID: \"fad3f414-87a1-4ae6-a3e4-62346dfa62e4\") " pod="openshift-monitoring/node-exporter-rbngs" Sep 11 09:14:06.454511 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:06.454422 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/2c1e552c-df45-4843-8717-37ee9b7d500f-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-pft5k\" (UID: \"2c1e552c-df45-4843-8717-37ee9b7d500f\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-pft5k" Sep 11 09:14:06.454511 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:06.454458 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fad3f414-87a1-4ae6-a3e4-62346dfa62e4-metrics-client-ca\") pod \"node-exporter-rbngs\" (UID: \"fad3f414-87a1-4ae6-a3e4-62346dfa62e4\") " pod="openshift-monitoring/node-exporter-rbngs" Sep 11 09:14:06.454511 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:06.454506 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2c1e552c-df45-4843-8717-37ee9b7d500f-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-pft5k\" (UID: \"2c1e552c-df45-4843-8717-37ee9b7d500f\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-pft5k" Sep 11 09:14:06.454737 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:06.454532 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/58d6994e-3003-41c3-857f-734e806d6edd-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-4xdpg\" (UID: \"58d6994e-3003-41c3-857f-734e806d6edd\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-4xdpg" Sep 11 09:14:06.454737 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:06.454592 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fad3f414-87a1-4ae6-a3e4-62346dfa62e4-sys\") pod \"node-exporter-rbngs\" (UID: \"fad3f414-87a1-4ae6-a3e4-62346dfa62e4\") " pod="openshift-monitoring/node-exporter-rbngs" Sep 11 09:14:06.454737 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:06.454617 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fad3f414-87a1-4ae6-a3e4-62346dfa62e4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rbngs\" (UID: \"fad3f414-87a1-4ae6-a3e4-62346dfa62e4\") " pod="openshift-monitoring/node-exporter-rbngs" Sep 11 09:14:06.454737 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:06.454666 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fad3f414-87a1-4ae6-a3e4-62346dfa62e4-sys\") pod \"node-exporter-rbngs\" (UID: \"fad3f414-87a1-4ae6-a3e4-62346dfa62e4\") " pod="openshift-monitoring/node-exporter-rbngs" Sep 11 09:14:06.454737 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:06.454694 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8gwp\" (UniqueName: \"kubernetes.io/projected/58d6994e-3003-41c3-857f-734e806d6edd-kube-api-access-b8gwp\") pod \"openshift-state-metrics-67dcf8c979-4xdpg\" (UID: \"58d6994e-3003-41c3-857f-734e806d6edd\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-4xdpg" Sep 11 09:14:06.454893 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:06.454743 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/58d6994e-3003-41c3-857f-734e806d6edd-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-4xdpg\" (UID: \"58d6994e-3003-41c3-857f-734e806d6edd\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-4xdpg" Sep 11 09:14:06.454893 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:06.454776 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/fad3f414-87a1-4ae6-a3e4-62346dfa62e4-node-exporter-textfile\") pod \"node-exporter-rbngs\" (UID: \"fad3f414-87a1-4ae6-a3e4-62346dfa62e4\") " pod="openshift-monitoring/node-exporter-rbngs" Sep 11 09:14:06.454893 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:06.454793 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2c1e552c-df45-4843-8717-37ee9b7d500f-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-pft5k\" (UID: \"2c1e552c-df45-4843-8717-37ee9b7d500f\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-pft5k" Sep 11 09:14:06.454893 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:06.454821 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/58d6994e-3003-41c3-857f-734e806d6edd-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-4xdpg\" (UID: \"58d6994e-3003-41c3-857f-734e806d6edd\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-4xdpg" Sep 11 09:14:06.454893 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:06.454839 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fad3f414-87a1-4ae6-a3e4-62346dfa62e4-node-exporter-tls\") pod \"node-exporter-rbngs\" (UID: \"fad3f414-87a1-4ae6-a3e4-62346dfa62e4\") " pod="openshift-monitoring/node-exporter-rbngs" Sep 11 09:14:06.454893 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:06.454863 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2c1e552c-df45-4843-8717-37ee9b7d500f-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-pft5k\" (UID: \"2c1e552c-df45-4843-8717-37ee9b7d500f\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-pft5k" Sep 11 09:14:06.455208 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:14:06.454926 2575 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Sep 11 09:14:06.455208 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:14:06.454993 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fad3f414-87a1-4ae6-a3e4-62346dfa62e4-node-exporter-tls podName:fad3f414-87a1-4ae6-a3e4-62346dfa62e4 nodeName:}" failed. No retries permitted until 2026-09-11 09:14:06.954971008 +0000 UTC m=+63.295581197 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/fad3f414-87a1-4ae6-a3e4-62346dfa62e4-node-exporter-tls") pod "node-exporter-rbngs" (UID: "fad3f414-87a1-4ae6-a3e4-62346dfa62e4") : secret "node-exporter-tls" not found Sep 11 09:14:06.455208 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:06.454924 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/fad3f414-87a1-4ae6-a3e4-62346dfa62e4-node-exporter-accelerators-collector-config\") pod \"node-exporter-rbngs\" (UID: \"fad3f414-87a1-4ae6-a3e4-62346dfa62e4\") " pod="openshift-monitoring/node-exporter-rbngs" Sep 11 09:14:06.455208 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:06.455063 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/2c1e552c-df45-4843-8717-37ee9b7d500f-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-pft5k\" (UID: \"2c1e552c-df45-4843-8717-37ee9b7d500f\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-pft5k" Sep 11 09:14:06.455208 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:06.455102 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/fad3f414-87a1-4ae6-a3e4-62346dfa62e4-node-exporter-textfile\") pod \"node-exporter-rbngs\" (UID: \"fad3f414-87a1-4ae6-a3e4-62346dfa62e4\") " pod="openshift-monitoring/node-exporter-rbngs" Sep 11 09:14:06.455208 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:06.455180 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fad3f414-87a1-4ae6-a3e4-62346dfa62e4-metrics-client-ca\") pod \"node-exporter-rbngs\" (UID: \"fad3f414-87a1-4ae6-a3e4-62346dfa62e4\") " pod="openshift-monitoring/node-exporter-rbngs" Sep 11 09:14:06.455407 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:06.455342 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/fad3f414-87a1-4ae6-a3e4-62346dfa62e4-node-exporter-accelerators-collector-config\") pod \"node-exporter-rbngs\" (UID: \"fad3f414-87a1-4ae6-a3e4-62346dfa62e4\") " pod="openshift-monitoring/node-exporter-rbngs" Sep 11 09:14:06.456829 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:06.456801 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fad3f414-87a1-4ae6-a3e4-62346dfa62e4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rbngs\" (UID: \"fad3f414-87a1-4ae6-a3e4-62346dfa62e4\") " pod="openshift-monitoring/node-exporter-rbngs" Sep 11 09:14:06.466126 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:06.466101 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rjnc\" (UniqueName: \"kubernetes.io/projected/fad3f414-87a1-4ae6-a3e4-62346dfa62e4-kube-api-access-8rjnc\") pod \"node-exporter-rbngs\" (UID: \"fad3f414-87a1-4ae6-a3e4-62346dfa62e4\") " pod="openshift-monitoring/node-exporter-rbngs" Sep 11 09:14:06.556194 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:06.556155 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/2c1e552c-df45-4843-8717-37ee9b7d500f-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-pft5k\" (UID: \"2c1e552c-df45-4843-8717-37ee9b7d500f\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-pft5k" Sep 11 09:14:06.556407 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:06.556205 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8n5q\" (UniqueName: \"kubernetes.io/projected/2c1e552c-df45-4843-8717-37ee9b7d500f-kube-api-access-g8n5q\") pod \"kube-state-metrics-59fbdd77d-pft5k\" (UID: \"2c1e552c-df45-4843-8717-37ee9b7d500f\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-pft5k" Sep 11 09:14:06.556407 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:06.556239 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/2c1e552c-df45-4843-8717-37ee9b7d500f-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-pft5k\" (UID: \"2c1e552c-df45-4843-8717-37ee9b7d500f\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-pft5k" Sep 11 09:14:06.556407 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:06.556258 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2c1e552c-df45-4843-8717-37ee9b7d500f-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-pft5k\" (UID: \"2c1e552c-df45-4843-8717-37ee9b7d500f\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-pft5k" Sep 11 09:14:06.556407 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:06.556275 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/58d6994e-3003-41c3-857f-734e806d6edd-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-4xdpg\" (UID: \"58d6994e-3003-41c3-857f-734e806d6edd\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-4xdpg" Sep 11 09:14:06.556407 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:06.556314 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8gwp\" (UniqueName: \"kubernetes.io/projected/58d6994e-3003-41c3-857f-734e806d6edd-kube-api-access-b8gwp\") pod \"openshift-state-metrics-67dcf8c979-4xdpg\" (UID: \"58d6994e-3003-41c3-857f-734e806d6edd\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-4xdpg" Sep 11 09:14:06.556407 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:06.556334 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/58d6994e-3003-41c3-857f-734e806d6edd-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-4xdpg\" (UID: \"58d6994e-3003-41c3-857f-734e806d6edd\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-4xdpg" Sep 11 09:14:06.556407 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:06.556358 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2c1e552c-df45-4843-8717-37ee9b7d500f-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-pft5k\" (UID: \"2c1e552c-df45-4843-8717-37ee9b7d500f\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-pft5k" Sep 11 09:14:06.556407 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:06.556396 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/58d6994e-3003-41c3-857f-734e806d6edd-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-4xdpg\" (UID: \"58d6994e-3003-41c3-857f-734e806d6edd\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-4xdpg" Sep 11 09:14:06.556839 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:06.556434 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2c1e552c-df45-4843-8717-37ee9b7d500f-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-pft5k\" (UID: \"2c1e552c-df45-4843-8717-37ee9b7d500f\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-pft5k" Sep 11 09:14:06.556839 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:06.556761 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/2c1e552c-df45-4843-8717-37ee9b7d500f-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-pft5k\" (UID: \"2c1e552c-df45-4843-8717-37ee9b7d500f\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-pft5k" Sep 11 09:14:06.557145 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:06.557114 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/2c1e552c-df45-4843-8717-37ee9b7d500f-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-pft5k\" (UID: \"2c1e552c-df45-4843-8717-37ee9b7d500f\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-pft5k" Sep 11 09:14:06.557297 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:06.557263 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2c1e552c-df45-4843-8717-37ee9b7d500f-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-pft5k\" (UID: \"2c1e552c-df45-4843-8717-37ee9b7d500f\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-pft5k" Sep 11 09:14:06.557368 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:06.557334 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/58d6994e-3003-41c3-857f-734e806d6edd-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-4xdpg\" (UID: \"58d6994e-3003-41c3-857f-734e806d6edd\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-4xdpg" Sep 11 09:14:06.559172 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:06.559116 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/58d6994e-3003-41c3-857f-734e806d6edd-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-4xdpg\" (UID: \"58d6994e-3003-41c3-857f-734e806d6edd\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-4xdpg" Sep 11 09:14:06.559301 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:06.559278 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2c1e552c-df45-4843-8717-37ee9b7d500f-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-pft5k\" (UID: \"2c1e552c-df45-4843-8717-37ee9b7d500f\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-pft5k" Sep 11 09:14:06.559347 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:06.559301 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2c1e552c-df45-4843-8717-37ee9b7d500f-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-pft5k\" (UID: \"2c1e552c-df45-4843-8717-37ee9b7d500f\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-pft5k" Sep 11 09:14:06.559347 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:06.559310 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/58d6994e-3003-41c3-857f-734e806d6edd-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-4xdpg\" (UID: \"58d6994e-3003-41c3-857f-734e806d6edd\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-4xdpg" Sep 11 09:14:06.574451 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:06.574406 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8gwp\" (UniqueName: \"kubernetes.io/projected/58d6994e-3003-41c3-857f-734e806d6edd-kube-api-access-b8gwp\") pod \"openshift-state-metrics-67dcf8c979-4xdpg\" (UID: \"58d6994e-3003-41c3-857f-734e806d6edd\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-4xdpg" Sep 11 09:14:06.574952 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:06.574926 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8n5q\" (UniqueName: \"kubernetes.io/projected/2c1e552c-df45-4843-8717-37ee9b7d500f-kube-api-access-g8n5q\") pod \"kube-state-metrics-59fbdd77d-pft5k\" (UID: \"2c1e552c-df45-4843-8717-37ee9b7d500f\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-pft5k" Sep 11 09:14:06.655126 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:06.655033 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-4xdpg" Sep 11 09:14:06.660570 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:06.660546 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-pft5k" Sep 11 09:14:06.929042 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:06.928847 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-4xdpg"] Sep 11 09:14:06.931550 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:14:06.931514 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58d6994e_3003_41c3_857f_734e806d6edd.slice/crio-bf8b694df7fa2680d3695b1f0f9789ce54a5937c059f25f3041c3739ff2568e8 WatchSource:0}: Error finding container bf8b694df7fa2680d3695b1f0f9789ce54a5937c059f25f3041c3739ff2568e8: Status 404 returned error can't find the container with id bf8b694df7fa2680d3695b1f0f9789ce54a5937c059f25f3041c3739ff2568e8 Sep 11 09:14:06.932820 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:06.932793 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-pft5k"] Sep 11 09:14:06.935260 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:14:06.935237 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c1e552c_df45_4843_8717_37ee9b7d500f.slice/crio-247032f6394dadb38e236a2316fa620b99f5586183782ea2514f486302a66b58 WatchSource:0}: Error finding container 247032f6394dadb38e236a2316fa620b99f5586183782ea2514f486302a66b58: Status 404 returned error can't find the container with id 247032f6394dadb38e236a2316fa620b99f5586183782ea2514f486302a66b58 Sep 11 09:14:06.959382 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:06.959356 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fad3f414-87a1-4ae6-a3e4-62346dfa62e4-node-exporter-tls\") pod \"node-exporter-rbngs\" (UID: \"fad3f414-87a1-4ae6-a3e4-62346dfa62e4\") " pod="openshift-monitoring/node-exporter-rbngs" Sep 11 09:14:06.961760 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:06.961739 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fad3f414-87a1-4ae6-a3e4-62346dfa62e4-node-exporter-tls\") pod \"node-exporter-rbngs\" (UID: \"fad3f414-87a1-4ae6-a3e4-62346dfa62e4\") " pod="openshift-monitoring/node-exporter-rbngs" Sep 11 09:14:07.248840 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:07.248757 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-rbngs" Sep 11 09:14:07.257371 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:14:07.257346 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfad3f414_87a1_4ae6_a3e4_62346dfa62e4.slice/crio-c67e95034e9eaad9b5eedcca2650e20524ffbdb8f10f8fdba13b2038194a3eea WatchSource:0}: Error finding container c67e95034e9eaad9b5eedcca2650e20524ffbdb8f10f8fdba13b2038194a3eea: Status 404 returned error can't find the container with id c67e95034e9eaad9b5eedcca2650e20524ffbdb8f10f8fdba13b2038194a3eea Sep 11 09:14:07.411756 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:07.411719 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 11 09:14:07.415972 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:07.415359 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 11 09:14:07.417975 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:07.417951 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Sep 11 09:14:07.418180 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:07.418157 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Sep 11 09:14:07.418280 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:07.418264 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Sep 11 09:14:07.418362 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:07.418346 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Sep 11 09:14:07.418418 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:07.418367 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Sep 11 09:14:07.418621 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:07.418604 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Sep 11 09:14:07.418621 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:07.418614 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Sep 11 09:14:07.418760 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:07.418627 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Sep 11 09:14:07.418760 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:07.418707 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-qnr2m\"" Sep 11 09:14:07.418852 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:07.418767 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Sep 11 09:14:07.434240 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:07.434151 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 11 09:14:07.463894 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:07.463861 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bbea5b4b-7d0d-4bbb-be9b-25fe1a714e3f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"bbea5b4b-7d0d-4bbb-be9b-25fe1a714e3f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 09:14:07.464059 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:07.463900 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/bbea5b4b-7d0d-4bbb-be9b-25fe1a714e3f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"bbea5b4b-7d0d-4bbb-be9b-25fe1a714e3f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 09:14:07.464059 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:07.463955 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/bbea5b4b-7d0d-4bbb-be9b-25fe1a714e3f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"bbea5b4b-7d0d-4bbb-be9b-25fe1a714e3f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 09:14:07.464059 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:07.463977 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bbea5b4b-7d0d-4bbb-be9b-25fe1a714e3f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"bbea5b4b-7d0d-4bbb-be9b-25fe1a714e3f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 09:14:07.464059 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:07.464011 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/bbea5b4b-7d0d-4bbb-be9b-25fe1a714e3f-config-volume\") pod \"alertmanager-main-0\" (UID: \"bbea5b4b-7d0d-4bbb-be9b-25fe1a714e3f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 09:14:07.464059 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:07.464050 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jl6k5\" (UniqueName: \"kubernetes.io/projected/bbea5b4b-7d0d-4bbb-be9b-25fe1a714e3f-kube-api-access-jl6k5\") pod \"alertmanager-main-0\" (UID: \"bbea5b4b-7d0d-4bbb-be9b-25fe1a714e3f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 09:14:07.464236 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:07.464099 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bbea5b4b-7d0d-4bbb-be9b-25fe1a714e3f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"bbea5b4b-7d0d-4bbb-be9b-25fe1a714e3f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 09:14:07.464236 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:07.464116 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bbea5b4b-7d0d-4bbb-be9b-25fe1a714e3f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"bbea5b4b-7d0d-4bbb-be9b-25fe1a714e3f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 09:14:07.464236 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:07.464149 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-vb5ck" Sep 11 09:14:07.464236 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:07.464206 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bbea5b4b-7d0d-4bbb-be9b-25fe1a714e3f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"bbea5b4b-7d0d-4bbb-be9b-25fe1a714e3f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 09:14:07.464236 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:07.464234 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bbea5b4b-7d0d-4bbb-be9b-25fe1a714e3f-config-out\") pod \"alertmanager-main-0\" (UID: \"bbea5b4b-7d0d-4bbb-be9b-25fe1a714e3f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 09:14:07.464402 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:07.464257 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/bbea5b4b-7d0d-4bbb-be9b-25fe1a714e3f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"bbea5b4b-7d0d-4bbb-be9b-25fe1a714e3f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 09:14:07.464402 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:07.464285 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/bbea5b4b-7d0d-4bbb-be9b-25fe1a714e3f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"bbea5b4b-7d0d-4bbb-be9b-25fe1a714e3f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 09:14:07.464402 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:07.464314 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bbea5b4b-7d0d-4bbb-be9b-25fe1a714e3f-web-config\") pod \"alertmanager-main-0\" (UID: \"bbea5b4b-7d0d-4bbb-be9b-25fe1a714e3f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 09:14:07.495104 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:07.495056 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-4xdpg" event={"ID":"58d6994e-3003-41c3-857f-734e806d6edd","Type":"ContainerStarted","Data":"c490f283e0cd4a0c0de26ca7be7dc2df36d5297b295f6e742ef488029e5cc7e3"} Sep 11 09:14:07.495300 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:07.495146 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-4xdpg" event={"ID":"58d6994e-3003-41c3-857f-734e806d6edd","Type":"ContainerStarted","Data":"3f0679494184d6f1f8eeabfbc0e9a7a8a889c9dcb688b6a13f095132ab084f86"} Sep 11 09:14:07.495300 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:07.495175 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-4xdpg" event={"ID":"58d6994e-3003-41c3-857f-734e806d6edd","Type":"ContainerStarted","Data":"bf8b694df7fa2680d3695b1f0f9789ce54a5937c059f25f3041c3739ff2568e8"} Sep 11 09:14:07.496313 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:07.496287 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-pft5k" event={"ID":"2c1e552c-df45-4843-8717-37ee9b7d500f","Type":"ContainerStarted","Data":"247032f6394dadb38e236a2316fa620b99f5586183782ea2514f486302a66b58"} Sep 11 09:14:07.497390 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:07.497372 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rbngs" event={"ID":"fad3f414-87a1-4ae6-a3e4-62346dfa62e4","Type":"ContainerStarted","Data":"c67e95034e9eaad9b5eedcca2650e20524ffbdb8f10f8fdba13b2038194a3eea"} Sep 11 09:14:07.564793 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:07.564754 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bbea5b4b-7d0d-4bbb-be9b-25fe1a714e3f-web-config\") pod \"alertmanager-main-0\" (UID: \"bbea5b4b-7d0d-4bbb-be9b-25fe1a714e3f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 09:14:07.564963 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:07.564839 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bbea5b4b-7d0d-4bbb-be9b-25fe1a714e3f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"bbea5b4b-7d0d-4bbb-be9b-25fe1a714e3f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 09:14:07.564963 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:07.564867 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/bbea5b4b-7d0d-4bbb-be9b-25fe1a714e3f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"bbea5b4b-7d0d-4bbb-be9b-25fe1a714e3f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 09:14:07.565081 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:07.565000 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/bbea5b4b-7d0d-4bbb-be9b-25fe1a714e3f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"bbea5b4b-7d0d-4bbb-be9b-25fe1a714e3f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 09:14:07.565081 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:07.565035 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bbea5b4b-7d0d-4bbb-be9b-25fe1a714e3f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"bbea5b4b-7d0d-4bbb-be9b-25fe1a714e3f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 09:14:07.565185 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:07.565086 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/bbea5b4b-7d0d-4bbb-be9b-25fe1a714e3f-config-volume\") pod \"alertmanager-main-0\" (UID: \"bbea5b4b-7d0d-4bbb-be9b-25fe1a714e3f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 09:14:07.565185 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:07.565128 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jl6k5\" (UniqueName: \"kubernetes.io/projected/bbea5b4b-7d0d-4bbb-be9b-25fe1a714e3f-kube-api-access-jl6k5\") pod \"alertmanager-main-0\" (UID: \"bbea5b4b-7d0d-4bbb-be9b-25fe1a714e3f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 09:14:07.565279 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:07.565183 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bbea5b4b-7d0d-4bbb-be9b-25fe1a714e3f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"bbea5b4b-7d0d-4bbb-be9b-25fe1a714e3f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 09:14:07.565279 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:07.565208 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bbea5b4b-7d0d-4bbb-be9b-25fe1a714e3f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"bbea5b4b-7d0d-4bbb-be9b-25fe1a714e3f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 09:14:07.565279 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:07.565273 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bbea5b4b-7d0d-4bbb-be9b-25fe1a714e3f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"bbea5b4b-7d0d-4bbb-be9b-25fe1a714e3f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 09:14:07.565426 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:07.565300 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bbea5b4b-7d0d-4bbb-be9b-25fe1a714e3f-config-out\") pod \"alertmanager-main-0\" (UID: \"bbea5b4b-7d0d-4bbb-be9b-25fe1a714e3f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 09:14:07.565426 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:07.565324 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/bbea5b4b-7d0d-4bbb-be9b-25fe1a714e3f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"bbea5b4b-7d0d-4bbb-be9b-25fe1a714e3f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 09:14:07.565426 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:07.565353 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/bbea5b4b-7d0d-4bbb-be9b-25fe1a714e3f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"bbea5b4b-7d0d-4bbb-be9b-25fe1a714e3f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 09:14:07.565426 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:07.565385 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/bbea5b4b-7d0d-4bbb-be9b-25fe1a714e3f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"bbea5b4b-7d0d-4bbb-be9b-25fe1a714e3f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 09:14:07.565888 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:07.565858 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bbea5b4b-7d0d-4bbb-be9b-25fe1a714e3f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"bbea5b4b-7d0d-4bbb-be9b-25fe1a714e3f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 09:14:07.566673 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:14:07.566362 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bbea5b4b-7d0d-4bbb-be9b-25fe1a714e3f-alertmanager-trusted-ca-bundle podName:bbea5b4b-7d0d-4bbb-be9b-25fe1a714e3f nodeName:}" failed. No retries permitted until 2026-09-11 09:14:08.066319975 +0000 UTC m=+64.406930178 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/bbea5b4b-7d0d-4bbb-be9b-25fe1a714e3f-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "bbea5b4b-7d0d-4bbb-be9b-25fe1a714e3f") : configmap references non-existent config key: ca-bundle.crt Sep 11 09:14:07.566673 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:14:07.566385 2575 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found Sep 11 09:14:07.566673 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:14:07.566506 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbea5b4b-7d0d-4bbb-be9b-25fe1a714e3f-secret-alertmanager-main-tls podName:bbea5b4b-7d0d-4bbb-be9b-25fe1a714e3f nodeName:}" failed. No retries permitted until 2026-09-11 09:14:08.066454399 +0000 UTC m=+64.407064606 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/bbea5b4b-7d0d-4bbb-be9b-25fe1a714e3f-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "bbea5b4b-7d0d-4bbb-be9b-25fe1a714e3f") : secret "alertmanager-main-tls" not found Sep 11 09:14:07.570013 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:07.569970 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/bbea5b4b-7d0d-4bbb-be9b-25fe1a714e3f-config-volume\") pod \"alertmanager-main-0\" (UID: \"bbea5b4b-7d0d-4bbb-be9b-25fe1a714e3f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 09:14:07.570637 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:07.570347 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bbea5b4b-7d0d-4bbb-be9b-25fe1a714e3f-web-config\") pod \"alertmanager-main-0\" (UID: \"bbea5b4b-7d0d-4bbb-be9b-25fe1a714e3f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 09:14:07.570637 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:07.570592 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/bbea5b4b-7d0d-4bbb-be9b-25fe1a714e3f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"bbea5b4b-7d0d-4bbb-be9b-25fe1a714e3f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 09:14:07.570637 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:07.570619 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bbea5b4b-7d0d-4bbb-be9b-25fe1a714e3f-config-out\") pod \"alertmanager-main-0\" (UID: \"bbea5b4b-7d0d-4bbb-be9b-25fe1a714e3f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 09:14:07.570826 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:07.570784 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bbea5b4b-7d0d-4bbb-be9b-25fe1a714e3f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"bbea5b4b-7d0d-4bbb-be9b-25fe1a714e3f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 09:14:07.571440 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:07.571014 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bbea5b4b-7d0d-4bbb-be9b-25fe1a714e3f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"bbea5b4b-7d0d-4bbb-be9b-25fe1a714e3f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 09:14:07.571440 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:07.571423 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bbea5b4b-7d0d-4bbb-be9b-25fe1a714e3f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"bbea5b4b-7d0d-4bbb-be9b-25fe1a714e3f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 09:14:07.572095 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:07.572069 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/bbea5b4b-7d0d-4bbb-be9b-25fe1a714e3f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"bbea5b4b-7d0d-4bbb-be9b-25fe1a714e3f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 09:14:07.576749 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:07.576701 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jl6k5\" (UniqueName: \"kubernetes.io/projected/bbea5b4b-7d0d-4bbb-be9b-25fe1a714e3f-kube-api-access-jl6k5\") pod \"alertmanager-main-0\" (UID: \"bbea5b4b-7d0d-4bbb-be9b-25fe1a714e3f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 09:14:08.071594 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:08.071566 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bbea5b4b-7d0d-4bbb-be9b-25fe1a714e3f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"bbea5b4b-7d0d-4bbb-be9b-25fe1a714e3f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 09:14:08.071756 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:08.071666 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/bbea5b4b-7d0d-4bbb-be9b-25fe1a714e3f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"bbea5b4b-7d0d-4bbb-be9b-25fe1a714e3f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 09:14:08.072621 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:08.072566 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bbea5b4b-7d0d-4bbb-be9b-25fe1a714e3f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"bbea5b4b-7d0d-4bbb-be9b-25fe1a714e3f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 09:14:08.074257 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:08.074229 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/bbea5b4b-7d0d-4bbb-be9b-25fe1a714e3f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"bbea5b4b-7d0d-4bbb-be9b-25fe1a714e3f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 09:14:08.326423 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:08.326382 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 11 09:14:08.425505 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:08.425455 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-6bc45d4b6-qfb5s"] Sep 11 09:14:08.428333 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:08.428295 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6bc45d4b6-qfb5s" Sep 11 09:14:08.431016 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:08.430982 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Sep 11 09:14:08.431295 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:08.431276 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-bg9lb\"" Sep 11 09:14:08.431375 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:08.431335 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Sep 11 09:14:08.431418 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:08.431341 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Sep 11 09:14:08.431489 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:08.431449 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Sep 11 09:14:08.431575 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:08.431556 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Sep 11 09:14:08.431892 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:08.431871 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-a0fli87au5uvv\"" Sep 11 09:14:08.444274 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:08.444245 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6bc45d4b6-qfb5s"] Sep 11 09:14:08.476711 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:08.476679 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/91974a2e-4506-4077-897b-3b0ec57820a1-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6bc45d4b6-qfb5s\" (UID: \"91974a2e-4506-4077-897b-3b0ec57820a1\") " pod="openshift-monitoring/thanos-querier-6bc45d4b6-qfb5s" Sep 11 09:14:08.476875 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:08.476755 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/91974a2e-4506-4077-897b-3b0ec57820a1-secret-thanos-querier-tls\") pod \"thanos-querier-6bc45d4b6-qfb5s\" (UID: \"91974a2e-4506-4077-897b-3b0ec57820a1\") " pod="openshift-monitoring/thanos-querier-6bc45d4b6-qfb5s" Sep 11 09:14:08.476875 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:08.476857 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/91974a2e-4506-4077-897b-3b0ec57820a1-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6bc45d4b6-qfb5s\" (UID: \"91974a2e-4506-4077-897b-3b0ec57820a1\") " pod="openshift-monitoring/thanos-querier-6bc45d4b6-qfb5s" Sep 11 09:14:08.477001 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:08.476908 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/91974a2e-4506-4077-897b-3b0ec57820a1-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6bc45d4b6-qfb5s\" (UID: \"91974a2e-4506-4077-897b-3b0ec57820a1\") " pod="openshift-monitoring/thanos-querier-6bc45d4b6-qfb5s" Sep 11 09:14:08.477001 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:08.476936 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/91974a2e-4506-4077-897b-3b0ec57820a1-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6bc45d4b6-qfb5s\" (UID: \"91974a2e-4506-4077-897b-3b0ec57820a1\") " pod="openshift-monitoring/thanos-querier-6bc45d4b6-qfb5s" Sep 11 09:14:08.477001 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:08.476981 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpfpp\" (UniqueName: \"kubernetes.io/projected/91974a2e-4506-4077-897b-3b0ec57820a1-kube-api-access-zpfpp\") pod \"thanos-querier-6bc45d4b6-qfb5s\" (UID: \"91974a2e-4506-4077-897b-3b0ec57820a1\") " pod="openshift-monitoring/thanos-querier-6bc45d4b6-qfb5s" Sep 11 09:14:08.477184 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:08.477025 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/91974a2e-4506-4077-897b-3b0ec57820a1-metrics-client-ca\") pod \"thanos-querier-6bc45d4b6-qfb5s\" (UID: \"91974a2e-4506-4077-897b-3b0ec57820a1\") " pod="openshift-monitoring/thanos-querier-6bc45d4b6-qfb5s" Sep 11 09:14:08.477184 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:08.477052 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/91974a2e-4506-4077-897b-3b0ec57820a1-secret-grpc-tls\") pod \"thanos-querier-6bc45d4b6-qfb5s\" (UID: \"91974a2e-4506-4077-897b-3b0ec57820a1\") " pod="openshift-monitoring/thanos-querier-6bc45d4b6-qfb5s" Sep 11 09:14:08.504890 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:08.504800 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-4xdpg" event={"ID":"58d6994e-3003-41c3-857f-734e806d6edd","Type":"ContainerStarted","Data":"f4d46c2d09abde105194edbca2b5885dc249fe7914a3c5748e42756069a81e0a"} Sep 11 09:14:08.506705 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:08.506670 2575 generic.go:356] "Generic (PLEG): container finished" podID="fad3f414-87a1-4ae6-a3e4-62346dfa62e4" containerID="11c32def16361c68ae1d1e5819accba318e228af94eb9c18c92a426ed3a69e1e" exitCode=0 Sep 11 09:14:08.506847 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:08.506716 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rbngs" event={"ID":"fad3f414-87a1-4ae6-a3e4-62346dfa62e4","Type":"ContainerDied","Data":"11c32def16361c68ae1d1e5819accba318e228af94eb9c18c92a426ed3a69e1e"} Sep 11 09:14:08.527514 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:08.527435 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-4xdpg" podStartSLOduration=1.555640356 podStartE2EDuration="2.527409038s" totalImagesPullingTime="971.768682ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 09:14:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 09:14:08.527360984 +0000 UTC m=+64.867971192" watchObservedRunningTime="2026-09-11 09:14:08.527409038 +0000 UTC m=+64.868019245" Sep 11 09:14:08.578251 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:08.578216 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpfpp\" (UniqueName: \"kubernetes.io/projected/91974a2e-4506-4077-897b-3b0ec57820a1-kube-api-access-zpfpp\") pod \"thanos-querier-6bc45d4b6-qfb5s\" (UID: \"91974a2e-4506-4077-897b-3b0ec57820a1\") " pod="openshift-monitoring/thanos-querier-6bc45d4b6-qfb5s" Sep 11 09:14:08.578405 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:08.578379 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/91974a2e-4506-4077-897b-3b0ec57820a1-metrics-client-ca\") pod \"thanos-querier-6bc45d4b6-qfb5s\" (UID: \"91974a2e-4506-4077-897b-3b0ec57820a1\") " pod="openshift-monitoring/thanos-querier-6bc45d4b6-qfb5s" Sep 11 09:14:08.578485 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:08.578418 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/91974a2e-4506-4077-897b-3b0ec57820a1-secret-grpc-tls\") pod \"thanos-querier-6bc45d4b6-qfb5s\" (UID: \"91974a2e-4506-4077-897b-3b0ec57820a1\") " pod="openshift-monitoring/thanos-querier-6bc45d4b6-qfb5s" Sep 11 09:14:08.578641 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:08.578616 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/91974a2e-4506-4077-897b-3b0ec57820a1-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6bc45d4b6-qfb5s\" (UID: \"91974a2e-4506-4077-897b-3b0ec57820a1\") " pod="openshift-monitoring/thanos-querier-6bc45d4b6-qfb5s" Sep 11 09:14:08.578746 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:08.578727 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/91974a2e-4506-4077-897b-3b0ec57820a1-secret-thanos-querier-tls\") pod \"thanos-querier-6bc45d4b6-qfb5s\" (UID: \"91974a2e-4506-4077-897b-3b0ec57820a1\") " pod="openshift-monitoring/thanos-querier-6bc45d4b6-qfb5s" Sep 11 09:14:08.579928 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:08.578856 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/91974a2e-4506-4077-897b-3b0ec57820a1-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6bc45d4b6-qfb5s\" (UID: \"91974a2e-4506-4077-897b-3b0ec57820a1\") " pod="openshift-monitoring/thanos-querier-6bc45d4b6-qfb5s" Sep 11 09:14:08.579928 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:08.578945 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/91974a2e-4506-4077-897b-3b0ec57820a1-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6bc45d4b6-qfb5s\" (UID: \"91974a2e-4506-4077-897b-3b0ec57820a1\") " pod="openshift-monitoring/thanos-querier-6bc45d4b6-qfb5s" Sep 11 09:14:08.579928 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:08.578972 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/91974a2e-4506-4077-897b-3b0ec57820a1-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6bc45d4b6-qfb5s\" (UID: \"91974a2e-4506-4077-897b-3b0ec57820a1\") " pod="openshift-monitoring/thanos-querier-6bc45d4b6-qfb5s" Sep 11 09:14:08.579928 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:08.579339 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/91974a2e-4506-4077-897b-3b0ec57820a1-metrics-client-ca\") pod \"thanos-querier-6bc45d4b6-qfb5s\" (UID: \"91974a2e-4506-4077-897b-3b0ec57820a1\") " pod="openshift-monitoring/thanos-querier-6bc45d4b6-qfb5s" Sep 11 09:14:08.583707 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:08.583681 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/91974a2e-4506-4077-897b-3b0ec57820a1-secret-thanos-querier-tls\") pod \"thanos-querier-6bc45d4b6-qfb5s\" (UID: \"91974a2e-4506-4077-897b-3b0ec57820a1\") " pod="openshift-monitoring/thanos-querier-6bc45d4b6-qfb5s" Sep 11 09:14:08.583819 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:08.583768 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/91974a2e-4506-4077-897b-3b0ec57820a1-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6bc45d4b6-qfb5s\" (UID: \"91974a2e-4506-4077-897b-3b0ec57820a1\") " pod="openshift-monitoring/thanos-querier-6bc45d4b6-qfb5s" Sep 11 09:14:08.584227 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:08.584209 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/91974a2e-4506-4077-897b-3b0ec57820a1-secret-grpc-tls\") pod \"thanos-querier-6bc45d4b6-qfb5s\" (UID: \"91974a2e-4506-4077-897b-3b0ec57820a1\") " pod="openshift-monitoring/thanos-querier-6bc45d4b6-qfb5s" Sep 11 09:14:08.584321 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:08.584228 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/91974a2e-4506-4077-897b-3b0ec57820a1-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6bc45d4b6-qfb5s\" (UID: \"91974a2e-4506-4077-897b-3b0ec57820a1\") " pod="openshift-monitoring/thanos-querier-6bc45d4b6-qfb5s" Sep 11 09:14:08.584388 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:08.584324 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/91974a2e-4506-4077-897b-3b0ec57820a1-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6bc45d4b6-qfb5s\" (UID: \"91974a2e-4506-4077-897b-3b0ec57820a1\") " pod="openshift-monitoring/thanos-querier-6bc45d4b6-qfb5s" Sep 11 09:14:08.584730 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:08.584697 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/91974a2e-4506-4077-897b-3b0ec57820a1-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6bc45d4b6-qfb5s\" (UID: \"91974a2e-4506-4077-897b-3b0ec57820a1\") " pod="openshift-monitoring/thanos-querier-6bc45d4b6-qfb5s" Sep 11 09:14:08.591278 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:08.591251 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpfpp\" (UniqueName: \"kubernetes.io/projected/91974a2e-4506-4077-897b-3b0ec57820a1-kube-api-access-zpfpp\") pod \"thanos-querier-6bc45d4b6-qfb5s\" (UID: \"91974a2e-4506-4077-897b-3b0ec57820a1\") " pod="openshift-monitoring/thanos-querier-6bc45d4b6-qfb5s" Sep 11 09:14:08.619067 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:08.618933 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 11 09:14:08.739916 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:08.739875 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6bc45d4b6-qfb5s" Sep 11 09:14:08.845902 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:14:08.845859 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbea5b4b_7d0d_4bbb_be9b_25fe1a714e3f.slice/crio-5f8aa79fca321f433469d58ebf57f41fe6504114154c0c31ec331f77df4c641f WatchSource:0}: Error finding container 5f8aa79fca321f433469d58ebf57f41fe6504114154c0c31ec331f77df4c641f: Status 404 returned error can't find the container with id 5f8aa79fca321f433469d58ebf57f41fe6504114154c0c31ec331f77df4c641f Sep 11 09:14:09.125330 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:09.125220 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6bc45d4b6-qfb5s"] Sep 11 09:14:09.130417 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:14:09.130389 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91974a2e_4506_4077_897b_3b0ec57820a1.slice/crio-c8ea8e62442d90e6e3d45ce5d03ccc04e91a3ced2ffb0367a998c9b2f096afb3 WatchSource:0}: Error finding container c8ea8e62442d90e6e3d45ce5d03ccc04e91a3ced2ffb0367a998c9b2f096afb3: Status 404 returned error can't find the container with id c8ea8e62442d90e6e3d45ce5d03ccc04e91a3ced2ffb0367a998c9b2f096afb3 Sep 11 09:14:09.511902 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:09.511864 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rbngs" event={"ID":"fad3f414-87a1-4ae6-a3e4-62346dfa62e4","Type":"ContainerStarted","Data":"fd4e88c8867e652a83824f132f341fe97d7fb141cb4997ebe899aa67e03e4bc1"} Sep 11 09:14:09.512310 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:09.511932 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rbngs" event={"ID":"fad3f414-87a1-4ae6-a3e4-62346dfa62e4","Type":"ContainerStarted","Data":"78319f5f6a52679187751c67a2ad800757a6838928be0383a9edcde9cd71b0c2"} Sep 11 09:14:09.513205 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:09.513176 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6bc45d4b6-qfb5s" event={"ID":"91974a2e-4506-4077-897b-3b0ec57820a1","Type":"ContainerStarted","Data":"c8ea8e62442d90e6e3d45ce5d03ccc04e91a3ced2ffb0367a998c9b2f096afb3"} Sep 11 09:14:09.515348 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:09.515319 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-pft5k" event={"ID":"2c1e552c-df45-4843-8717-37ee9b7d500f","Type":"ContainerStarted","Data":"ee6503c845d4a241a1ed4aee7b307bd382af1ee5e2037dffc45709cd2613dcb3"} Sep 11 09:14:09.515493 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:09.515371 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-pft5k" event={"ID":"2c1e552c-df45-4843-8717-37ee9b7d500f","Type":"ContainerStarted","Data":"f307839cc59cb382dabd32fae3cf2556e9187ea35770ef5b6f2801ed7ab6c32d"} Sep 11 09:14:09.515493 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:09.515400 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-pft5k" event={"ID":"2c1e552c-df45-4843-8717-37ee9b7d500f","Type":"ContainerStarted","Data":"90861aaced34bd6adf042d7c7c2e82118cb222809ba38784eda934c567384bca"} Sep 11 09:14:09.516543 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:09.516522 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bbea5b4b-7d0d-4bbb-be9b-25fe1a714e3f","Type":"ContainerStarted","Data":"5f8aa79fca321f433469d58ebf57f41fe6504114154c0c31ec331f77df4c641f"} Sep 11 09:14:09.531756 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:09.531709 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-rbngs" podStartSLOduration=2.422821754 podStartE2EDuration="3.531691728s" totalImagesPullingTime="1.023625584s" totalInitContainerRuntime="85.24439ms" isStatefulPod=true podCreationTimestamp="2026-09-11 09:14:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 09:14:09.530873855 +0000 UTC m=+65.871484061" watchObservedRunningTime="2026-09-11 09:14:09.531691728 +0000 UTC m=+65.872301934" Sep 11 09:14:09.552750 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:09.552696 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-pft5k" podStartSLOduration=1.615843822 podStartE2EDuration="3.552677308s" totalImagesPullingTime="1.936833486s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 09:14:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 09:14:09.552197184 +0000 UTC m=+65.892807390" watchObservedRunningTime="2026-09-11 09:14:09.552677308 +0000 UTC m=+65.893287515" Sep 11 09:14:09.790724 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:09.790634 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abeb037b-2e12-4485-974d-35280f982f4a-metrics-certs\") pod \"network-metrics-daemon-rlftv\" (UID: \"abeb037b-2e12-4485-974d-35280f982f4a\") " pod="openshift-multus/network-metrics-daemon-rlftv" Sep 11 09:14:09.793396 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:09.793376 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 11 09:14:09.803818 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:09.803792 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abeb037b-2e12-4485-974d-35280f982f4a-metrics-certs\") pod \"network-metrics-daemon-rlftv\" (UID: \"abeb037b-2e12-4485-974d-35280f982f4a\") " pod="openshift-multus/network-metrics-daemon-rlftv" Sep 11 09:14:09.824670 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:09.824638 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-ggqqp\"" Sep 11 09:14:09.832725 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:09.832674 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rlftv" Sep 11 09:14:09.892235 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:09.892046 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7eddd8ad-0ac3-421d-9649-95dc4fadfd7b-original-pull-secret\") pod \"global-pull-secret-syncer-8ng6h\" (UID: \"7eddd8ad-0ac3-421d-9649-95dc4fadfd7b\") " pod="kube-system/global-pull-secret-syncer-8ng6h" Sep 11 09:14:09.895784 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:09.895268 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 11 09:14:09.905453 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:09.905395 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7eddd8ad-0ac3-421d-9649-95dc4fadfd7b-original-pull-secret\") pod \"global-pull-secret-syncer-8ng6h\" (UID: \"7eddd8ad-0ac3-421d-9649-95dc4fadfd7b\") " pod="kube-system/global-pull-secret-syncer-8ng6h" Sep 11 09:14:09.993033 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:09.992991 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slfls\" (UniqueName: \"kubernetes.io/projected/92eb1c40-b8d0-428c-955a-c005e197f70a-kube-api-access-slfls\") pod \"network-check-target-bb6x9\" (UID: \"92eb1c40-b8d0-428c-955a-c005e197f70a\") " pod="openshift-network-diagnostics/network-check-target-bb6x9" Sep 11 09:14:09.996189 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:09.996160 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 11 09:14:10.006509 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:10.006460 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 11 09:14:10.016833 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:10.016808 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-slfls\" (UniqueName: \"kubernetes.io/projected/92eb1c40-b8d0-428c-955a-c005e197f70a-kube-api-access-slfls\") pod \"network-check-target-bb6x9\" (UID: \"92eb1c40-b8d0-428c-955a-c005e197f70a\") " pod="openshift-network-diagnostics/network-check-target-bb6x9" Sep 11 09:14:10.130193 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:10.130112 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8ng6h" Sep 11 09:14:10.138798 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:10.138767 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-rlftv"] Sep 11 09:14:10.139950 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:10.139924 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-qfqbz\"" Sep 11 09:14:10.143535 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:14:10.143505 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabeb037b_2e12_4485_974d_35280f982f4a.slice/crio-87dff03bcb0e44537b72f28c4247533900d2a24298880deb327bfa2f93dd680f WatchSource:0}: Error finding container 87dff03bcb0e44537b72f28c4247533900d2a24298880deb327bfa2f93dd680f: Status 404 returned error can't find the container with id 87dff03bcb0e44537b72f28c4247533900d2a24298880deb327bfa2f93dd680f Sep 11 09:14:10.147705 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:10.147673 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bb6x9" Sep 11 09:14:10.451711 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:10.451614 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-8ng6h"] Sep 11 09:14:10.488673 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:10.488640 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-bb6x9"] Sep 11 09:14:10.521927 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:10.521894 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rlftv" event={"ID":"abeb037b-2e12-4485-974d-35280f982f4a","Type":"ContainerStarted","Data":"87dff03bcb0e44537b72f28c4247533900d2a24298880deb327bfa2f93dd680f"} Sep 11 09:14:10.636126 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:14:10.636087 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7eddd8ad_0ac3_421d_9649_95dc4fadfd7b.slice/crio-7479d5be89b246e343235c46bdf31b4b0db567e91b683d1f661ab545ca380624 WatchSource:0}: Error finding container 7479d5be89b246e343235c46bdf31b4b0db567e91b683d1f661ab545ca380624: Status 404 returned error can't find the container with id 7479d5be89b246e343235c46bdf31b4b0db567e91b683d1f661ab545ca380624 Sep 11 09:14:10.636828 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:14:10.636803 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92eb1c40_b8d0_428c_955a_c005e197f70a.slice/crio-e54a224376fca49f05e6459bb635e07c1d6766a0c7c0df069ac92a97263f1a8c WatchSource:0}: Error finding container e54a224376fca49f05e6459bb635e07c1d6766a0c7c0df069ac92a97263f1a8c: Status 404 returned error can't find the container with id e54a224376fca49f05e6459bb635e07c1d6766a0c7c0df069ac92a97263f1a8c Sep 11 09:14:10.661100 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:10.661066 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-7888d6449c-zdchh"] Sep 11 09:14:10.677169 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:10.677141 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7888d6449c-zdchh" Sep 11 09:14:10.678005 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:10.677945 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7888d6449c-zdchh"] Sep 11 09:14:10.680183 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:10.680156 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Sep 11 09:14:10.680289 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:10.680196 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-9lkecdgiuegpf\"" Sep 11 09:14:10.681008 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:10.680992 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Sep 11 09:14:10.681366 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:10.681332 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-hmwmk\"" Sep 11 09:14:10.681366 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:10.681345 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Sep 11 09:14:10.681537 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:10.681369 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Sep 11 09:14:10.799133 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:10.799099 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a18b6af-13e2-4090-8750-5a0a06ad2719-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7888d6449c-zdchh\" (UID: \"1a18b6af-13e2-4090-8750-5a0a06ad2719\") " pod="openshift-monitoring/metrics-server-7888d6449c-zdchh" Sep 11 09:14:10.799133 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:10.799141 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/1a18b6af-13e2-4090-8750-5a0a06ad2719-secret-metrics-server-client-certs\") pod \"metrics-server-7888d6449c-zdchh\" (UID: \"1a18b6af-13e2-4090-8750-5a0a06ad2719\") " pod="openshift-monitoring/metrics-server-7888d6449c-zdchh" Sep 11 09:14:10.799332 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:10.799164 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/1a18b6af-13e2-4090-8750-5a0a06ad2719-audit-log\") pod \"metrics-server-7888d6449c-zdchh\" (UID: \"1a18b6af-13e2-4090-8750-5a0a06ad2719\") " pod="openshift-monitoring/metrics-server-7888d6449c-zdchh" Sep 11 09:14:10.799332 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:10.799194 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/1a18b6af-13e2-4090-8750-5a0a06ad2719-metrics-server-audit-profiles\") pod \"metrics-server-7888d6449c-zdchh\" (UID: \"1a18b6af-13e2-4090-8750-5a0a06ad2719\") " pod="openshift-monitoring/metrics-server-7888d6449c-zdchh" Sep 11 09:14:10.799332 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:10.799263 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbn47\" (UniqueName: \"kubernetes.io/projected/1a18b6af-13e2-4090-8750-5a0a06ad2719-kube-api-access-wbn47\") pod \"metrics-server-7888d6449c-zdchh\" (UID: \"1a18b6af-13e2-4090-8750-5a0a06ad2719\") " pod="openshift-monitoring/metrics-server-7888d6449c-zdchh" Sep 11 09:14:10.799332 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:10.799288 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/1a18b6af-13e2-4090-8750-5a0a06ad2719-secret-metrics-server-tls\") pod \"metrics-server-7888d6449c-zdchh\" (UID: \"1a18b6af-13e2-4090-8750-5a0a06ad2719\") " pod="openshift-monitoring/metrics-server-7888d6449c-zdchh" Sep 11 09:14:10.799332 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:10.799303 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/1a18b6af-13e2-4090-8750-5a0a06ad2719-client-ca-bundle\") pod \"metrics-server-7888d6449c-zdchh\" (UID: \"1a18b6af-13e2-4090-8750-5a0a06ad2719\") " pod="openshift-monitoring/metrics-server-7888d6449c-zdchh" Sep 11 09:14:10.900031 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:10.899989 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/1a18b6af-13e2-4090-8750-5a0a06ad2719-metrics-server-audit-profiles\") pod \"metrics-server-7888d6449c-zdchh\" (UID: \"1a18b6af-13e2-4090-8750-5a0a06ad2719\") " pod="openshift-monitoring/metrics-server-7888d6449c-zdchh" Sep 11 09:14:10.900225 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:10.900056 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbn47\" (UniqueName: \"kubernetes.io/projected/1a18b6af-13e2-4090-8750-5a0a06ad2719-kube-api-access-wbn47\") pod \"metrics-server-7888d6449c-zdchh\" (UID: \"1a18b6af-13e2-4090-8750-5a0a06ad2719\") " pod="openshift-monitoring/metrics-server-7888d6449c-zdchh" Sep 11 09:14:10.900225 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:10.900086 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/1a18b6af-13e2-4090-8750-5a0a06ad2719-secret-metrics-server-tls\") pod \"metrics-server-7888d6449c-zdchh\" (UID: \"1a18b6af-13e2-4090-8750-5a0a06ad2719\") " pod="openshift-monitoring/metrics-server-7888d6449c-zdchh" Sep 11 09:14:10.900225 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:10.900109 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/1a18b6af-13e2-4090-8750-5a0a06ad2719-client-ca-bundle\") pod \"metrics-server-7888d6449c-zdchh\" (UID: \"1a18b6af-13e2-4090-8750-5a0a06ad2719\") " pod="openshift-monitoring/metrics-server-7888d6449c-zdchh" Sep 11 09:14:10.900225 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:10.900161 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a18b6af-13e2-4090-8750-5a0a06ad2719-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7888d6449c-zdchh\" (UID: \"1a18b6af-13e2-4090-8750-5a0a06ad2719\") " pod="openshift-monitoring/metrics-server-7888d6449c-zdchh" Sep 11 09:14:10.900225 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:10.900195 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/1a18b6af-13e2-4090-8750-5a0a06ad2719-secret-metrics-server-client-certs\") pod \"metrics-server-7888d6449c-zdchh\" (UID: \"1a18b6af-13e2-4090-8750-5a0a06ad2719\") " pod="openshift-monitoring/metrics-server-7888d6449c-zdchh" Sep 11 09:14:10.900225 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:10.900221 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/1a18b6af-13e2-4090-8750-5a0a06ad2719-audit-log\") pod \"metrics-server-7888d6449c-zdchh\" (UID: \"1a18b6af-13e2-4090-8750-5a0a06ad2719\") " pod="openshift-monitoring/metrics-server-7888d6449c-zdchh" Sep 11 09:14:10.900683 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:10.900658 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/1a18b6af-13e2-4090-8750-5a0a06ad2719-audit-log\") pod \"metrics-server-7888d6449c-zdchh\" (UID: \"1a18b6af-13e2-4090-8750-5a0a06ad2719\") " pod="openshift-monitoring/metrics-server-7888d6449c-zdchh" Sep 11 09:14:10.900925 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:10.900899 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a18b6af-13e2-4090-8750-5a0a06ad2719-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7888d6449c-zdchh\" (UID: \"1a18b6af-13e2-4090-8750-5a0a06ad2719\") " pod="openshift-monitoring/metrics-server-7888d6449c-zdchh" Sep 11 09:14:10.901120 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:10.901098 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/1a18b6af-13e2-4090-8750-5a0a06ad2719-metrics-server-audit-profiles\") pod \"metrics-server-7888d6449c-zdchh\" (UID: \"1a18b6af-13e2-4090-8750-5a0a06ad2719\") " pod="openshift-monitoring/metrics-server-7888d6449c-zdchh" Sep 11 09:14:10.902549 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:10.902528 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/1a18b6af-13e2-4090-8750-5a0a06ad2719-client-ca-bundle\") pod \"metrics-server-7888d6449c-zdchh\" (UID: \"1a18b6af-13e2-4090-8750-5a0a06ad2719\") " pod="openshift-monitoring/metrics-server-7888d6449c-zdchh" Sep 11 09:14:10.902630 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:10.902619 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/1a18b6af-13e2-4090-8750-5a0a06ad2719-secret-metrics-server-client-certs\") pod \"metrics-server-7888d6449c-zdchh\" (UID: \"1a18b6af-13e2-4090-8750-5a0a06ad2719\") " pod="openshift-monitoring/metrics-server-7888d6449c-zdchh" Sep 11 09:14:10.902666 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:10.902653 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/1a18b6af-13e2-4090-8750-5a0a06ad2719-secret-metrics-server-tls\") pod \"metrics-server-7888d6449c-zdchh\" (UID: \"1a18b6af-13e2-4090-8750-5a0a06ad2719\") " pod="openshift-monitoring/metrics-server-7888d6449c-zdchh" Sep 11 09:14:10.909068 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:10.909044 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbn47\" (UniqueName: \"kubernetes.io/projected/1a18b6af-13e2-4090-8750-5a0a06ad2719-kube-api-access-wbn47\") pod \"metrics-server-7888d6449c-zdchh\" (UID: \"1a18b6af-13e2-4090-8750-5a0a06ad2719\") " pod="openshift-monitoring/metrics-server-7888d6449c-zdchh" Sep 11 09:14:10.988323 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:10.988285 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7888d6449c-zdchh" Sep 11 09:14:11.175231 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:11.175003 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-wrc6v"] Sep 11 09:14:11.178127 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:11.178102 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-wrc6v" Sep 11 09:14:11.181392 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:11.181371 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-v9vv9\"" Sep 11 09:14:11.184888 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:11.184860 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Sep 11 09:14:11.195302 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:11.195268 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-wrc6v"] Sep 11 09:14:11.303666 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:11.303629 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9b068dbf-c4bd-4527-9025-7ce2b277ac0a-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-wrc6v\" (UID: \"9b068dbf-c4bd-4527-9025-7ce2b277ac0a\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-wrc6v" Sep 11 09:14:11.303841 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:11.303762 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4k6v\" (UniqueName: \"kubernetes.io/projected/9b068dbf-c4bd-4527-9025-7ce2b277ac0a-kube-api-access-h4k6v\") pod \"monitoring-plugin-5d85778b9d-wrc6v\" (UID: \"9b068dbf-c4bd-4527-9025-7ce2b277ac0a\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-wrc6v" Sep 11 09:14:11.404787 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:11.404750 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9b068dbf-c4bd-4527-9025-7ce2b277ac0a-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-wrc6v\" (UID: \"9b068dbf-c4bd-4527-9025-7ce2b277ac0a\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-wrc6v" Sep 11 09:14:11.404960 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:11.404870 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4k6v\" (UniqueName: \"kubernetes.io/projected/9b068dbf-c4bd-4527-9025-7ce2b277ac0a-kube-api-access-h4k6v\") pod \"monitoring-plugin-5d85778b9d-wrc6v\" (UID: \"9b068dbf-c4bd-4527-9025-7ce2b277ac0a\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-wrc6v" Sep 11 09:14:11.407340 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:11.407314 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9b068dbf-c4bd-4527-9025-7ce2b277ac0a-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-wrc6v\" (UID: \"9b068dbf-c4bd-4527-9025-7ce2b277ac0a\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-wrc6v" Sep 11 09:14:11.415935 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:11.415907 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4k6v\" (UniqueName: \"kubernetes.io/projected/9b068dbf-c4bd-4527-9025-7ce2b277ac0a-kube-api-access-h4k6v\") pod \"monitoring-plugin-5d85778b9d-wrc6v\" (UID: \"9b068dbf-c4bd-4527-9025-7ce2b277ac0a\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-wrc6v" Sep 11 09:14:11.490999 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:11.490908 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-wrc6v" Sep 11 09:14:11.526315 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:11.526271 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-bb6x9" event={"ID":"92eb1c40-b8d0-428c-955a-c005e197f70a","Type":"ContainerStarted","Data":"e54a224376fca49f05e6459bb635e07c1d6766a0c7c0df069ac92a97263f1a8c"} Sep 11 09:14:11.527343 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:11.527320 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-8ng6h" event={"ID":"7eddd8ad-0ac3-421d-9649-95dc4fadfd7b","Type":"ContainerStarted","Data":"7479d5be89b246e343235c46bdf31b4b0db567e91b683d1f661ab545ca380624"} Sep 11 09:14:12.093275 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:12.092646 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7888d6449c-zdchh"] Sep 11 09:14:12.098964 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:14:12.098926 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a18b6af_13e2_4090_8750_5a0a06ad2719.slice/crio-adda0e8a53f5b0f939a987798dafe69049bc474ff3bef759edca4f4a90acce41 WatchSource:0}: Error finding container adda0e8a53f5b0f939a987798dafe69049bc474ff3bef759edca4f4a90acce41: Status 404 returned error can't find the container with id adda0e8a53f5b0f939a987798dafe69049bc474ff3bef759edca4f4a90acce41 Sep 11 09:14:12.119275 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:12.118850 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-wrc6v"] Sep 11 09:14:12.536249 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:12.536205 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-wrc6v" event={"ID":"9b068dbf-c4bd-4527-9025-7ce2b277ac0a","Type":"ContainerStarted","Data":"c4b5ad557a868e2dded4ac481e6b9a2e475709225101967e020118a89bdec272"} Sep 11 09:14:12.538254 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:12.538222 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7888d6449c-zdchh" event={"ID":"1a18b6af-13e2-4090-8750-5a0a06ad2719","Type":"ContainerStarted","Data":"adda0e8a53f5b0f939a987798dafe69049bc474ff3bef759edca4f4a90acce41"} Sep 11 09:14:12.540673 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:12.540632 2575 generic.go:356] "Generic (PLEG): container finished" podID="bbea5b4b-7d0d-4bbb-be9b-25fe1a714e3f" containerID="f0f47011ddb74708827013f1f5e1a6fe569f441aedc49e04bc27a0f215b78cc6" exitCode=0 Sep 11 09:14:12.540762 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:12.540718 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bbea5b4b-7d0d-4bbb-be9b-25fe1a714e3f","Type":"ContainerDied","Data":"f0f47011ddb74708827013f1f5e1a6fe569f441aedc49e04bc27a0f215b78cc6"} Sep 11 09:14:12.543855 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:12.543819 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rlftv" event={"ID":"abeb037b-2e12-4485-974d-35280f982f4a","Type":"ContainerStarted","Data":"5cc25d3abf85ef96498ac5a622f5d2fc5d411349bb06c250d8597f7c7e1584d8"} Sep 11 09:14:12.543966 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:12.543870 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rlftv" event={"ID":"abeb037b-2e12-4485-974d-35280f982f4a","Type":"ContainerStarted","Data":"05fe16f6228480955a3c84f0d99eb248db2d968b2180f5cdfaf09397219c3a85"} Sep 11 09:14:12.546704 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:12.546593 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6bc45d4b6-qfb5s" event={"ID":"91974a2e-4506-4077-897b-3b0ec57820a1","Type":"ContainerStarted","Data":"8c3673ecf70c84de1ac2434c2070387f7b9349f2bdbce94a4d9737dab9f52e27"} Sep 11 09:14:12.546704 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:12.546663 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6bc45d4b6-qfb5s" event={"ID":"91974a2e-4506-4077-897b-3b0ec57820a1","Type":"ContainerStarted","Data":"5bda7b6e829df08ef1e07de43d50f0ffbbb8b0e2cba5f93dbb79ad6564546025"} Sep 11 09:14:12.546704 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:12.546700 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6bc45d4b6-qfb5s" event={"ID":"91974a2e-4506-4077-897b-3b0ec57820a1","Type":"ContainerStarted","Data":"64ac2d5e5f660ad764151a37c0624b8dc4b2b3572a80163b4f53251eabd19925"} Sep 11 09:14:12.657074 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:12.656687 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-rlftv" podStartSLOduration=67.005089311 podStartE2EDuration="1m8.656665704s" totalImagesPullingTime="1.651576393s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 09:13:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 09:14:12.598653065 +0000 UTC m=+68.939263276" watchObservedRunningTime="2026-09-11 09:14:12.656665704 +0000 UTC m=+68.997275912" Sep 11 09:14:12.657389 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:12.657362 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 11 09:14:12.670043 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:12.669902 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 11 09:14:12.675193 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:12.674989 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Sep 11 09:14:12.675358 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:12.675238 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Sep 11 09:14:12.675414 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:12.675362 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Sep 11 09:14:12.675414 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:12.675373 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Sep 11 09:14:12.675508 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:12.675420 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Sep 11 09:14:12.675878 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:12.675857 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Sep 11 09:14:12.675878 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:12.675869 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Sep 11 09:14:12.676103 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:12.676084 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-dh6a3pmfgl4oc\"" Sep 11 09:14:12.676172 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:12.676142 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-wblmj\"" Sep 11 09:14:12.676236 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:12.676092 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Sep 11 09:14:12.676682 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:12.676664 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Sep 11 09:14:12.676777 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:12.676767 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Sep 11 09:14:12.676843 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:12.676773 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Sep 11 09:14:12.676843 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:12.676833 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Sep 11 09:14:12.679024 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:12.678993 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Sep 11 09:14:12.680796 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:12.680657 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Sep 11 09:14:12.690288 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:12.690261 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 11 09:14:12.720453 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:12.720422 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/86b6d5a2-9069-4296-ae71-cc14fe79dac8-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"86b6d5a2-9069-4296-ae71-cc14fe79dac8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 09:14:12.720642 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:12.720462 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5t2ht\" (UniqueName: \"kubernetes.io/projected/86b6d5a2-9069-4296-ae71-cc14fe79dac8-kube-api-access-5t2ht\") pod \"prometheus-k8s-0\" (UID: \"86b6d5a2-9069-4296-ae71-cc14fe79dac8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 09:14:12.720642 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:12.720519 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/86b6d5a2-9069-4296-ae71-cc14fe79dac8-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"86b6d5a2-9069-4296-ae71-cc14fe79dac8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 09:14:12.720642 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:12.720547 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/86b6d5a2-9069-4296-ae71-cc14fe79dac8-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"86b6d5a2-9069-4296-ae71-cc14fe79dac8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 09:14:12.720642 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:12.720570 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/86b6d5a2-9069-4296-ae71-cc14fe79dac8-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"86b6d5a2-9069-4296-ae71-cc14fe79dac8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 09:14:12.720642 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:12.720601 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/86b6d5a2-9069-4296-ae71-cc14fe79dac8-config-out\") pod \"prometheus-k8s-0\" (UID: \"86b6d5a2-9069-4296-ae71-cc14fe79dac8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 09:14:12.720901 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:12.720644 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/86b6d5a2-9069-4296-ae71-cc14fe79dac8-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"86b6d5a2-9069-4296-ae71-cc14fe79dac8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 09:14:12.720901 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:12.720671 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/86b6d5a2-9069-4296-ae71-cc14fe79dac8-web-config\") pod \"prometheus-k8s-0\" (UID: \"86b6d5a2-9069-4296-ae71-cc14fe79dac8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 09:14:12.720901 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:12.720695 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86b6d5a2-9069-4296-ae71-cc14fe79dac8-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"86b6d5a2-9069-4296-ae71-cc14fe79dac8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 09:14:12.720901 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:12.720730 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/86b6d5a2-9069-4296-ae71-cc14fe79dac8-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"86b6d5a2-9069-4296-ae71-cc14fe79dac8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 09:14:12.720901 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:12.720755 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/86b6d5a2-9069-4296-ae71-cc14fe79dac8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"86b6d5a2-9069-4296-ae71-cc14fe79dac8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 09:14:12.720901 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:12.720798 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/86b6d5a2-9069-4296-ae71-cc14fe79dac8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"86b6d5a2-9069-4296-ae71-cc14fe79dac8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 09:14:12.720901 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:12.720826 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86b6d5a2-9069-4296-ae71-cc14fe79dac8-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"86b6d5a2-9069-4296-ae71-cc14fe79dac8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 09:14:12.720901 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:12.720861 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/86b6d5a2-9069-4296-ae71-cc14fe79dac8-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"86b6d5a2-9069-4296-ae71-cc14fe79dac8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 09:14:12.721294 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:12.720911 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/86b6d5a2-9069-4296-ae71-cc14fe79dac8-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"86b6d5a2-9069-4296-ae71-cc14fe79dac8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 09:14:12.721294 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:12.720934 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86b6d5a2-9069-4296-ae71-cc14fe79dac8-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"86b6d5a2-9069-4296-ae71-cc14fe79dac8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 09:14:12.721294 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:12.720965 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/86b6d5a2-9069-4296-ae71-cc14fe79dac8-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"86b6d5a2-9069-4296-ae71-cc14fe79dac8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 09:14:12.721294 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:12.720992 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/86b6d5a2-9069-4296-ae71-cc14fe79dac8-config\") pod \"prometheus-k8s-0\" (UID: \"86b6d5a2-9069-4296-ae71-cc14fe79dac8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 09:14:12.721294 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:12.721020 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/86b6d5a2-9069-4296-ae71-cc14fe79dac8-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"86b6d5a2-9069-4296-ae71-cc14fe79dac8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 09:14:12.721294 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:12.721066 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/86b6d5a2-9069-4296-ae71-cc14fe79dac8-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"86b6d5a2-9069-4296-ae71-cc14fe79dac8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 09:14:12.821928 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:12.821822 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/86b6d5a2-9069-4296-ae71-cc14fe79dac8-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"86b6d5a2-9069-4296-ae71-cc14fe79dac8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 09:14:12.822828 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:12.822777 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/86b6d5a2-9069-4296-ae71-cc14fe79dac8-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"86b6d5a2-9069-4296-ae71-cc14fe79dac8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 09:14:12.824413 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:12.822984 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/86b6d5a2-9069-4296-ae71-cc14fe79dac8-web-config\") pod \"prometheus-k8s-0\" (UID: \"86b6d5a2-9069-4296-ae71-cc14fe79dac8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 09:14:12.824413 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:12.823025 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86b6d5a2-9069-4296-ae71-cc14fe79dac8-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"86b6d5a2-9069-4296-ae71-cc14fe79dac8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 09:14:12.824413 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:12.823063 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/86b6d5a2-9069-4296-ae71-cc14fe79dac8-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"86b6d5a2-9069-4296-ae71-cc14fe79dac8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 09:14:12.824413 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:12.823097 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/86b6d5a2-9069-4296-ae71-cc14fe79dac8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"86b6d5a2-9069-4296-ae71-cc14fe79dac8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 09:14:12.824413 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:12.823137 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/86b6d5a2-9069-4296-ae71-cc14fe79dac8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"86b6d5a2-9069-4296-ae71-cc14fe79dac8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 09:14:12.824413 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:12.823165 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86b6d5a2-9069-4296-ae71-cc14fe79dac8-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"86b6d5a2-9069-4296-ae71-cc14fe79dac8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 09:14:12.824413 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:12.823201 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/86b6d5a2-9069-4296-ae71-cc14fe79dac8-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"86b6d5a2-9069-4296-ae71-cc14fe79dac8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 09:14:12.824413 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:12.823250 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/86b6d5a2-9069-4296-ae71-cc14fe79dac8-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"86b6d5a2-9069-4296-ae71-cc14fe79dac8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 09:14:12.824413 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:12.823274 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86b6d5a2-9069-4296-ae71-cc14fe79dac8-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"86b6d5a2-9069-4296-ae71-cc14fe79dac8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 09:14:12.824413 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:12.823298 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/86b6d5a2-9069-4296-ae71-cc14fe79dac8-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"86b6d5a2-9069-4296-ae71-cc14fe79dac8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 09:14:12.824413 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:12.823327 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/86b6d5a2-9069-4296-ae71-cc14fe79dac8-config\") pod \"prometheus-k8s-0\" (UID: \"86b6d5a2-9069-4296-ae71-cc14fe79dac8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 09:14:12.824413 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:12.823353 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/86b6d5a2-9069-4296-ae71-cc14fe79dac8-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"86b6d5a2-9069-4296-ae71-cc14fe79dac8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 09:14:12.824413 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:12.823397 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/86b6d5a2-9069-4296-ae71-cc14fe79dac8-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"86b6d5a2-9069-4296-ae71-cc14fe79dac8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 09:14:12.824413 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:12.823426 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/86b6d5a2-9069-4296-ae71-cc14fe79dac8-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"86b6d5a2-9069-4296-ae71-cc14fe79dac8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 09:14:12.824413 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:12.823449 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5t2ht\" (UniqueName: \"kubernetes.io/projected/86b6d5a2-9069-4296-ae71-cc14fe79dac8-kube-api-access-5t2ht\") pod \"prometheus-k8s-0\" (UID: \"86b6d5a2-9069-4296-ae71-cc14fe79dac8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 09:14:12.824413 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:12.823501 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/86b6d5a2-9069-4296-ae71-cc14fe79dac8-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"86b6d5a2-9069-4296-ae71-cc14fe79dac8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 09:14:12.824413 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:12.823529 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/86b6d5a2-9069-4296-ae71-cc14fe79dac8-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"86b6d5a2-9069-4296-ae71-cc14fe79dac8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 09:14:12.826158 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:12.823552 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/86b6d5a2-9069-4296-ae71-cc14fe79dac8-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"86b6d5a2-9069-4296-ae71-cc14fe79dac8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 09:14:12.826158 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:12.823585 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/86b6d5a2-9069-4296-ae71-cc14fe79dac8-config-out\") pod \"prometheus-k8s-0\" (UID: \"86b6d5a2-9069-4296-ae71-cc14fe79dac8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 09:14:12.826158 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:12.825048 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86b6d5a2-9069-4296-ae71-cc14fe79dac8-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"86b6d5a2-9069-4296-ae71-cc14fe79dac8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 09:14:12.826340 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:12.826321 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/86b6d5a2-9069-4296-ae71-cc14fe79dac8-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"86b6d5a2-9069-4296-ae71-cc14fe79dac8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 09:14:12.826403 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:12.826377 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86b6d5a2-9069-4296-ae71-cc14fe79dac8-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"86b6d5a2-9069-4296-ae71-cc14fe79dac8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 09:14:12.826860 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:12.826835 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/86b6d5a2-9069-4296-ae71-cc14fe79dac8-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"86b6d5a2-9069-4296-ae71-cc14fe79dac8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 09:14:12.828077 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:12.828054 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86b6d5a2-9069-4296-ae71-cc14fe79dac8-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"86b6d5a2-9069-4296-ae71-cc14fe79dac8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 09:14:12.829512 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:12.829323 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/86b6d5a2-9069-4296-ae71-cc14fe79dac8-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"86b6d5a2-9069-4296-ae71-cc14fe79dac8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 09:14:12.829616 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:12.829531 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/86b6d5a2-9069-4296-ae71-cc14fe79dac8-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"86b6d5a2-9069-4296-ae71-cc14fe79dac8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 09:14:12.831404 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:12.831376 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/86b6d5a2-9069-4296-ae71-cc14fe79dac8-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"86b6d5a2-9069-4296-ae71-cc14fe79dac8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 09:14:12.832788 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:12.832746 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/86b6d5a2-9069-4296-ae71-cc14fe79dac8-web-config\") pod \"prometheus-k8s-0\" (UID: \"86b6d5a2-9069-4296-ae71-cc14fe79dac8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 09:14:12.833178 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:12.833154 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/86b6d5a2-9069-4296-ae71-cc14fe79dac8-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"86b6d5a2-9069-4296-ae71-cc14fe79dac8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 09:14:12.833796 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:12.833776 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/86b6d5a2-9069-4296-ae71-cc14fe79dac8-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"86b6d5a2-9069-4296-ae71-cc14fe79dac8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 09:14:12.834024 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:12.833992 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/86b6d5a2-9069-4296-ae71-cc14fe79dac8-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"86b6d5a2-9069-4296-ae71-cc14fe79dac8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 09:14:12.835092 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:12.834787 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/86b6d5a2-9069-4296-ae71-cc14fe79dac8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"86b6d5a2-9069-4296-ae71-cc14fe79dac8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 09:14:12.835092 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:12.835049 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/86b6d5a2-9069-4296-ae71-cc14fe79dac8-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"86b6d5a2-9069-4296-ae71-cc14fe79dac8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 09:14:12.835216 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:12.835181 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/86b6d5a2-9069-4296-ae71-cc14fe79dac8-config-out\") pod \"prometheus-k8s-0\" (UID: \"86b6d5a2-9069-4296-ae71-cc14fe79dac8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 09:14:12.835341 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:12.835317 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/86b6d5a2-9069-4296-ae71-cc14fe79dac8-config\") pod \"prometheus-k8s-0\" (UID: \"86b6d5a2-9069-4296-ae71-cc14fe79dac8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 09:14:12.837639 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:12.837249 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/86b6d5a2-9069-4296-ae71-cc14fe79dac8-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"86b6d5a2-9069-4296-ae71-cc14fe79dac8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 09:14:12.838131 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:12.837825 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5t2ht\" (UniqueName: \"kubernetes.io/projected/86b6d5a2-9069-4296-ae71-cc14fe79dac8-kube-api-access-5t2ht\") pod \"prometheus-k8s-0\" (UID: \"86b6d5a2-9069-4296-ae71-cc14fe79dac8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 09:14:12.839313 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:12.838781 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/86b6d5a2-9069-4296-ae71-cc14fe79dac8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"86b6d5a2-9069-4296-ae71-cc14fe79dac8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 09:14:12.994729 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:12.992861 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 11 09:14:13.358210 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:13.357887 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 11 09:14:13.479814 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:14:13.479767 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86b6d5a2_9069_4296_ae71_cc14fe79dac8.slice/crio-2f34ce0dbbb36225ef68135fb0cd6b135c83bed4699dc4b705670978ead4d8ed WatchSource:0}: Error finding container 2f34ce0dbbb36225ef68135fb0cd6b135c83bed4699dc4b705670978ead4d8ed: Status 404 returned error can't find the container with id 2f34ce0dbbb36225ef68135fb0cd6b135c83bed4699dc4b705670978ead4d8ed Sep 11 09:14:13.556705 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:13.556517 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"86b6d5a2-9069-4296-ae71-cc14fe79dac8","Type":"ContainerStarted","Data":"2f34ce0dbbb36225ef68135fb0cd6b135c83bed4699dc4b705670978ead4d8ed"} Sep 11 09:14:17.586835 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:17.586097 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-bb6x9" event={"ID":"92eb1c40-b8d0-428c-955a-c005e197f70a","Type":"ContainerStarted","Data":"daa9f500630e9a7bf36a5622f3e790d09d7b42d893df845e48eb0523b0050770"} Sep 11 09:14:17.586835 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:17.586215 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-bb6x9" Sep 11 09:14:17.591828 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:17.591792 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7888d6449c-zdchh" event={"ID":"1a18b6af-13e2-4090-8750-5a0a06ad2719","Type":"ContainerStarted","Data":"462dbafc2191253d2deaa71cb30b8de33d6085944d382cf6036755b671a76f29"} Sep 11 09:14:17.594523 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:17.594494 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bbea5b4b-7d0d-4bbb-be9b-25fe1a714e3f","Type":"ContainerStarted","Data":"9553d99c21e56d8cc96f85c8324915329b26fd00f665c294c4bfbd6c70198e8c"} Sep 11 09:14:17.594733 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:17.594709 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bbea5b4b-7d0d-4bbb-be9b-25fe1a714e3f","Type":"ContainerStarted","Data":"b7a251364458c9d6f98603a9eac68b3f1ac85b3207a005f63de9d7427f41d5af"} Sep 11 09:14:17.594830 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:17.594813 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bbea5b4b-7d0d-4bbb-be9b-25fe1a714e3f","Type":"ContainerStarted","Data":"8bcf52aa25c47ef7a93f92211aa8f96d9743221c17c38d379ae78efd41849067"} Sep 11 09:14:17.598031 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:17.598005 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6bc45d4b6-qfb5s" event={"ID":"91974a2e-4506-4077-897b-3b0ec57820a1","Type":"ContainerStarted","Data":"1f622f4e04fa64d9212caf34683524d0a8e94bf6b916e14ce72a4be567c09fbf"} Sep 11 09:14:17.598108 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:17.598055 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6bc45d4b6-qfb5s" event={"ID":"91974a2e-4506-4077-897b-3b0ec57820a1","Type":"ContainerStarted","Data":"e07a7f13e0da7f88f0b82cef9dd7bed66a50d4ee9d25e6dc84c43febed17a215"} Sep 11 09:14:17.598108 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:17.598088 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6bc45d4b6-qfb5s" event={"ID":"91974a2e-4506-4077-897b-3b0ec57820a1","Type":"ContainerStarted","Data":"6624ede10fcf98bc2a36d6838b2c4a68884fc9071b1a65520f18078783986e95"} Sep 11 09:14:17.599787 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:17.599752 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-8ng6h" event={"ID":"7eddd8ad-0ac3-421d-9649-95dc4fadfd7b","Type":"ContainerStarted","Data":"47f4187bd31d6349f4dd2f628683559d5c9a80eac9e3eab5cd263893b34a382f"} Sep 11 09:14:17.601406 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:17.601385 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-wrc6v" event={"ID":"9b068dbf-c4bd-4527-9025-7ce2b277ac0a","Type":"ContainerStarted","Data":"d6fdb2c3ff1e1d17b629c17f88d3d4ebfbbe285f3dac09dfb7fe5971bf89cc7f"} Sep 11 09:14:17.601842 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:17.601790 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-wrc6v" Sep 11 09:14:17.603363 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:17.603343 2575 generic.go:356] "Generic (PLEG): container finished" podID="86b6d5a2-9069-4296-ae71-cc14fe79dac8" containerID="15e237c2ed442f867e20ade32077fc3ee71cae175cd18b64ff5c1ff6b628f360" exitCode=0 Sep 11 09:14:17.603537 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:17.603396 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"86b6d5a2-9069-4296-ae71-cc14fe79dac8","Type":"ContainerDied","Data":"15e237c2ed442f867e20ade32077fc3ee71cae175cd18b64ff5c1ff6b628f360"} Sep 11 09:14:17.605485 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:17.605423 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-bb6x9" podStartSLOduration=66.99943355 podStartE2EDuration="1m13.605405717s" totalImagesPullingTime="6.605972167s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 09:13:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 09:14:17.605002964 +0000 UTC m=+73.945613174" watchObservedRunningTime="2026-09-11 09:14:17.605405717 +0000 UTC m=+73.946015919" Sep 11 09:14:17.608729 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:17.608708 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-wrc6v" Sep 11 09:14:17.622915 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:17.622873 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-wrc6v" podStartSLOduration=1.52581452 podStartE2EDuration="6.62285859s" totalImagesPullingTime="5.09704407s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 09:14:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 09:14:17.622555341 +0000 UTC m=+73.963165548" watchObservedRunningTime="2026-09-11 09:14:17.62285859 +0000 UTC m=+73.963468796" Sep 11 09:14:17.640495 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:17.640238 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-8ng6h" podStartSLOduration=67.01516786 podStartE2EDuration="1m13.64021828s" totalImagesPullingTime="6.62505042s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 09:13:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 09:14:17.639507595 +0000 UTC m=+73.980117802" watchObservedRunningTime="2026-09-11 09:14:17.64021828 +0000 UTC m=+73.980828485" Sep 11 09:14:17.667936 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:17.667877 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-7888d6449c-zdchh" podStartSLOduration=2.54724238 podStartE2EDuration="7.667857297s" totalImagesPullingTime="5.120614917s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 09:14:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 09:14:17.665783944 +0000 UTC m=+74.006394150" watchObservedRunningTime="2026-09-11 09:14:17.667857297 +0000 UTC m=+74.008467506" Sep 11 09:14:17.691938 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:17.691885 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-6bc45d4b6-qfb5s" podStartSLOduration=1.9391412460000002 podStartE2EDuration="9.691863709s" totalImagesPullingTime="7.752722463s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 09:14:08 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 09:14:17.690137173 +0000 UTC m=+74.030747404" watchObservedRunningTime="2026-09-11 09:14:17.691863709 +0000 UTC m=+74.032473916" Sep 11 09:14:18.612160 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:18.612121 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bbea5b4b-7d0d-4bbb-be9b-25fe1a714e3f","Type":"ContainerStarted","Data":"093d3b0a90c7f462f9cf9883d8f1729a420ed97e8071a7d2dd744ff7081a89cf"} Sep 11 09:14:18.612655 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:18.612208 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bbea5b4b-7d0d-4bbb-be9b-25fe1a714e3f","Type":"ContainerStarted","Data":"4e9413393a2b8157047f07415dcb5d519a7cbe6853aab7bc0dd90d85cae22a22"} Sep 11 09:14:18.612655 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:18.612242 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bbea5b4b-7d0d-4bbb-be9b-25fe1a714e3f","Type":"ContainerStarted","Data":"38918eb841dfc3178381736169d2689794f7e9a34333e06d531ff99967a203b5"} Sep 11 09:14:18.613108 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:18.613075 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-6bc45d4b6-qfb5s" Sep 11 09:14:18.620936 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:18.620911 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-6bc45d4b6-qfb5s" Sep 11 09:14:18.640058 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:18.639918 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.996980307 podStartE2EDuration="11.639896151s" totalImagesPullingTime="7.617092104s" totalInitContainerRuntime="25.82374ms" isStatefulPod=false podCreationTimestamp="2026-09-11 09:14:07 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 09:14:18.638607952 +0000 UTC m=+74.979218159" watchObservedRunningTime="2026-09-11 09:14:18.639896151 +0000 UTC m=+74.980506358" Sep 11 09:14:21.630455 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:21.630422 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"86b6d5a2-9069-4296-ae71-cc14fe79dac8","Type":"ContainerStarted","Data":"9b581166c46b332ede95e70af9b7131c22684be3b6a2a51571e315a8d78526d9"} Sep 11 09:14:21.630829 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:21.630505 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"86b6d5a2-9069-4296-ae71-cc14fe79dac8","Type":"ContainerStarted","Data":"377ff4b2156632ad6af33ca8905ebb6a5567ef7cf2dc963b2e3382be8e58f033"} Sep 11 09:14:21.630829 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:21.630534 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"86b6d5a2-9069-4296-ae71-cc14fe79dac8","Type":"ContainerStarted","Data":"eddfb4e5bdd3b6615373931756d531b1d6ab43e38035db84d082f24f3898d94b"} Sep 11 09:14:21.630829 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:21.630560 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"86b6d5a2-9069-4296-ae71-cc14fe79dac8","Type":"ContainerStarted","Data":"6a4a82a83cd4761f2452b94677d5900a6889e3b3bea709d863d9b61ad80c97d8"} Sep 11 09:14:21.630829 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:21.630582 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"86b6d5a2-9069-4296-ae71-cc14fe79dac8","Type":"ContainerStarted","Data":"f6617a9438f09026a31faef00e848b4e21fe1298658150da928a7d7eaf1a2d26"} Sep 11 09:14:21.630829 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:21.630604 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"86b6d5a2-9069-4296-ae71-cc14fe79dac8","Type":"ContainerStarted","Data":"b96653957d42e3f3b5a09f97109894229a502c5355f88e9afcbd20a068bec86e"} Sep 11 09:14:21.660175 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:21.660119 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=6.62787461 podStartE2EDuration="9.660103732s" totalImagesPullingTime="2.987585098s" totalInitContainerRuntime="44.644024ms" isStatefulPod=false podCreationTimestamp="2026-09-11 09:14:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 09:14:21.658380018 +0000 UTC m=+77.998990223" watchObservedRunningTime="2026-09-11 09:14:21.660103732 +0000 UTC m=+78.000713937" Sep 11 09:14:22.995301 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:22.995248 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 11 09:14:30.989272 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:30.989157 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-7888d6449c-zdchh" Sep 11 09:14:30.989272 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:30.989252 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-7888d6449c-zdchh" Sep 11 09:14:48.614492 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:48.614439 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-bb6x9" Sep 11 09:14:50.994433 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:50.994398 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-7888d6449c-zdchh" Sep 11 09:14:50.998311 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:14:50.998292 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-7888d6449c-zdchh" Sep 11 09:15:12.993921 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:15:12.993876 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Sep 11 09:15:13.013034 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:15:13.013004 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Sep 11 09:15:13.876786 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:15:13.876754 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 11 09:15:30.684657 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:15:30.684612 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-655fd9c684-n6hz2"] Sep 11 09:15:30.687151 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:15:30.687126 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-655fd9c684-n6hz2" Sep 11 09:15:30.690543 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:15:30.690443 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Sep 11 09:15:30.690543 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:15:30.690443 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Sep 11 09:15:30.690543 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:15:30.690442 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-kgzx7\"" Sep 11 09:15:30.690543 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:15:30.690510 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Sep 11 09:15:30.690543 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:15:30.690538 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Sep 11 09:15:30.690893 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:15:30.690641 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Sep 11 09:15:30.694678 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:15:30.694656 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Sep 11 09:15:30.705220 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:15:30.705190 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-655fd9c684-n6hz2"] Sep 11 09:15:30.797691 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:15:30.797654 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/aea7ca13-21b0-4745-9493-cfa07e40e36b-secret-telemeter-client\") pod \"telemeter-client-655fd9c684-n6hz2\" (UID: \"aea7ca13-21b0-4745-9493-cfa07e40e36b\") " pod="openshift-monitoring/telemeter-client-655fd9c684-n6hz2" Sep 11 09:15:30.797691 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:15:30.797693 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/aea7ca13-21b0-4745-9493-cfa07e40e36b-telemeter-client-tls\") pod \"telemeter-client-655fd9c684-n6hz2\" (UID: \"aea7ca13-21b0-4745-9493-cfa07e40e36b\") " pod="openshift-monitoring/telemeter-client-655fd9c684-n6hz2" Sep 11 09:15:30.797952 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:15:30.797712 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2szbm\" (UniqueName: \"kubernetes.io/projected/aea7ca13-21b0-4745-9493-cfa07e40e36b-kube-api-access-2szbm\") pod \"telemeter-client-655fd9c684-n6hz2\" (UID: \"aea7ca13-21b0-4745-9493-cfa07e40e36b\") " pod="openshift-monitoring/telemeter-client-655fd9c684-n6hz2" Sep 11 09:15:30.797952 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:15:30.797804 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aea7ca13-21b0-4745-9493-cfa07e40e36b-telemeter-trusted-ca-bundle\") pod \"telemeter-client-655fd9c684-n6hz2\" (UID: \"aea7ca13-21b0-4745-9493-cfa07e40e36b\") " pod="openshift-monitoring/telemeter-client-655fd9c684-n6hz2" Sep 11 09:15:30.797952 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:15:30.797848 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aea7ca13-21b0-4745-9493-cfa07e40e36b-serving-certs-ca-bundle\") pod \"telemeter-client-655fd9c684-n6hz2\" (UID: \"aea7ca13-21b0-4745-9493-cfa07e40e36b\") " pod="openshift-monitoring/telemeter-client-655fd9c684-n6hz2" Sep 11 09:15:30.797952 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:15:30.797887 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/aea7ca13-21b0-4745-9493-cfa07e40e36b-federate-client-tls\") pod \"telemeter-client-655fd9c684-n6hz2\" (UID: \"aea7ca13-21b0-4745-9493-cfa07e40e36b\") " pod="openshift-monitoring/telemeter-client-655fd9c684-n6hz2" Sep 11 09:15:30.797952 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:15:30.797906 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aea7ca13-21b0-4745-9493-cfa07e40e36b-metrics-client-ca\") pod \"telemeter-client-655fd9c684-n6hz2\" (UID: \"aea7ca13-21b0-4745-9493-cfa07e40e36b\") " pod="openshift-monitoring/telemeter-client-655fd9c684-n6hz2" Sep 11 09:15:30.797952 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:15:30.797933 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/aea7ca13-21b0-4745-9493-cfa07e40e36b-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-655fd9c684-n6hz2\" (UID: \"aea7ca13-21b0-4745-9493-cfa07e40e36b\") " pod="openshift-monitoring/telemeter-client-655fd9c684-n6hz2" Sep 11 09:15:30.898997 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:15:30.898941 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/aea7ca13-21b0-4745-9493-cfa07e40e36b-secret-telemeter-client\") pod \"telemeter-client-655fd9c684-n6hz2\" (UID: \"aea7ca13-21b0-4745-9493-cfa07e40e36b\") " pod="openshift-monitoring/telemeter-client-655fd9c684-n6hz2" Sep 11 09:15:30.898997 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:15:30.898997 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/aea7ca13-21b0-4745-9493-cfa07e40e36b-telemeter-client-tls\") pod \"telemeter-client-655fd9c684-n6hz2\" (UID: \"aea7ca13-21b0-4745-9493-cfa07e40e36b\") " pod="openshift-monitoring/telemeter-client-655fd9c684-n6hz2" Sep 11 09:15:30.899233 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:15:30.899018 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2szbm\" (UniqueName: \"kubernetes.io/projected/aea7ca13-21b0-4745-9493-cfa07e40e36b-kube-api-access-2szbm\") pod \"telemeter-client-655fd9c684-n6hz2\" (UID: \"aea7ca13-21b0-4745-9493-cfa07e40e36b\") " pod="openshift-monitoring/telemeter-client-655fd9c684-n6hz2" Sep 11 09:15:30.899233 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:15:30.899058 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aea7ca13-21b0-4745-9493-cfa07e40e36b-telemeter-trusted-ca-bundle\") pod \"telemeter-client-655fd9c684-n6hz2\" (UID: \"aea7ca13-21b0-4745-9493-cfa07e40e36b\") " pod="openshift-monitoring/telemeter-client-655fd9c684-n6hz2" Sep 11 09:15:30.899233 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:15:30.899096 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aea7ca13-21b0-4745-9493-cfa07e40e36b-serving-certs-ca-bundle\") pod \"telemeter-client-655fd9c684-n6hz2\" (UID: \"aea7ca13-21b0-4745-9493-cfa07e40e36b\") " pod="openshift-monitoring/telemeter-client-655fd9c684-n6hz2" Sep 11 09:15:30.899233 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:15:30.899124 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/aea7ca13-21b0-4745-9493-cfa07e40e36b-federate-client-tls\") pod \"telemeter-client-655fd9c684-n6hz2\" (UID: \"aea7ca13-21b0-4745-9493-cfa07e40e36b\") " pod="openshift-monitoring/telemeter-client-655fd9c684-n6hz2" Sep 11 09:15:30.899233 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:15:30.899157 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aea7ca13-21b0-4745-9493-cfa07e40e36b-metrics-client-ca\") pod \"telemeter-client-655fd9c684-n6hz2\" (UID: \"aea7ca13-21b0-4745-9493-cfa07e40e36b\") " pod="openshift-monitoring/telemeter-client-655fd9c684-n6hz2" Sep 11 09:15:30.899233 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:15:30.899207 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/aea7ca13-21b0-4745-9493-cfa07e40e36b-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-655fd9c684-n6hz2\" (UID: \"aea7ca13-21b0-4745-9493-cfa07e40e36b\") " pod="openshift-monitoring/telemeter-client-655fd9c684-n6hz2" Sep 11 09:15:30.900056 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:15:30.900023 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aea7ca13-21b0-4745-9493-cfa07e40e36b-serving-certs-ca-bundle\") pod \"telemeter-client-655fd9c684-n6hz2\" (UID: \"aea7ca13-21b0-4745-9493-cfa07e40e36b\") " pod="openshift-monitoring/telemeter-client-655fd9c684-n6hz2" Sep 11 09:15:30.900189 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:15:30.900091 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aea7ca13-21b0-4745-9493-cfa07e40e36b-metrics-client-ca\") pod \"telemeter-client-655fd9c684-n6hz2\" (UID: \"aea7ca13-21b0-4745-9493-cfa07e40e36b\") " pod="openshift-monitoring/telemeter-client-655fd9c684-n6hz2" Sep 11 09:15:30.900251 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:15:30.900200 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aea7ca13-21b0-4745-9493-cfa07e40e36b-telemeter-trusted-ca-bundle\") pod \"telemeter-client-655fd9c684-n6hz2\" (UID: \"aea7ca13-21b0-4745-9493-cfa07e40e36b\") " pod="openshift-monitoring/telemeter-client-655fd9c684-n6hz2" Sep 11 09:15:30.901658 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:15:30.901634 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/aea7ca13-21b0-4745-9493-cfa07e40e36b-secret-telemeter-client\") pod \"telemeter-client-655fd9c684-n6hz2\" (UID: \"aea7ca13-21b0-4745-9493-cfa07e40e36b\") " pod="openshift-monitoring/telemeter-client-655fd9c684-n6hz2" Sep 11 09:15:30.901758 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:15:30.901742 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/aea7ca13-21b0-4745-9493-cfa07e40e36b-telemeter-client-tls\") pod \"telemeter-client-655fd9c684-n6hz2\" (UID: \"aea7ca13-21b0-4745-9493-cfa07e40e36b\") " pod="openshift-monitoring/telemeter-client-655fd9c684-n6hz2" Sep 11 09:15:30.901959 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:15:30.901942 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/aea7ca13-21b0-4745-9493-cfa07e40e36b-federate-client-tls\") pod \"telemeter-client-655fd9c684-n6hz2\" (UID: \"aea7ca13-21b0-4745-9493-cfa07e40e36b\") " pod="openshift-monitoring/telemeter-client-655fd9c684-n6hz2" Sep 11 09:15:30.902049 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:15:30.902027 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/aea7ca13-21b0-4745-9493-cfa07e40e36b-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-655fd9c684-n6hz2\" (UID: \"aea7ca13-21b0-4745-9493-cfa07e40e36b\") " pod="openshift-monitoring/telemeter-client-655fd9c684-n6hz2" Sep 11 09:15:30.911194 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:15:30.911166 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2szbm\" (UniqueName: \"kubernetes.io/projected/aea7ca13-21b0-4745-9493-cfa07e40e36b-kube-api-access-2szbm\") pod \"telemeter-client-655fd9c684-n6hz2\" (UID: \"aea7ca13-21b0-4745-9493-cfa07e40e36b\") " pod="openshift-monitoring/telemeter-client-655fd9c684-n6hz2" Sep 11 09:15:30.998498 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:15:30.998413 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-655fd9c684-n6hz2" Sep 11 09:15:31.276189 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:15:31.276152 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-655fd9c684-n6hz2"] Sep 11 09:15:31.279928 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:15:31.279896 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaea7ca13_21b0_4745_9493_cfa07e40e36b.slice/crio-2ea6c76129c7dff6de81c3d8452869d1a9d1b4e5958c1a491fa567b0537d9006 WatchSource:0}: Error finding container 2ea6c76129c7dff6de81c3d8452869d1a9d1b4e5958c1a491fa567b0537d9006: Status 404 returned error can't find the container with id 2ea6c76129c7dff6de81c3d8452869d1a9d1b4e5958c1a491fa567b0537d9006 Sep 11 09:15:31.937198 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:15:31.937159 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-655fd9c684-n6hz2" event={"ID":"aea7ca13-21b0-4745-9493-cfa07e40e36b","Type":"ContainerStarted","Data":"2ea6c76129c7dff6de81c3d8452869d1a9d1b4e5958c1a491fa567b0537d9006"} Sep 11 09:15:33.947642 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:15:33.947601 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-655fd9c684-n6hz2" event={"ID":"aea7ca13-21b0-4745-9493-cfa07e40e36b","Type":"ContainerStarted","Data":"fdac95220eb04e6ca2daf24dec6389f74feb81ceb3e5f40aa38b35bb5edc788e"} Sep 11 09:15:33.948067 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:15:33.947686 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-655fd9c684-n6hz2" event={"ID":"aea7ca13-21b0-4745-9493-cfa07e40e36b","Type":"ContainerStarted","Data":"8240d5803440ff1c3c649fb2da93a2f2349d0bedf27c2ca6c7c200d683bae40a"} Sep 11 09:15:33.948067 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:15:33.947708 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-655fd9c684-n6hz2" event={"ID":"aea7ca13-21b0-4745-9493-cfa07e40e36b","Type":"ContainerStarted","Data":"ef3bc06328ec5aa7769a90b8e878f63de87c55c2d3d389628bd3555cf11c79a8"} Sep 11 09:15:33.971538 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:15:33.971482 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-655fd9c684-n6hz2" podStartSLOduration=2.329177837 podStartE2EDuration="3.971451391s" totalImagesPullingTime="1.642273554s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 09:15:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 09:15:33.970375437 +0000 UTC m=+150.310985669" watchObservedRunningTime="2026-09-11 09:15:33.971451391 +0000 UTC m=+150.312061598" Sep 11 09:18:04.005770 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:18:04.005740 2575 kubelet.go:1648] "Image garbage collection succeeded" Sep 11 09:18:31.811861 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:18:31.811826 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-95b9b66c7-xvkzz"] Sep 11 09:18:31.815474 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:18:31.815449 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-xvkzz" Sep 11 09:18:31.818005 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:18:31.817974 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-ui-tls\"" Sep 11 09:18:31.818168 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:18:31.818143 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-intra-comm\"" Sep 11 09:18:31.818298 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:18:31.818167 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-offline-tls\"" Sep 11 09:18:31.818298 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:18:31.818202 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\"" Sep 11 09:18:31.818298 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:18:31.818245 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-kw4td\"" Sep 11 09:18:31.818298 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:18:31.818175 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\"" Sep 11 09:18:31.818298 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:18:31.818288 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-online-tls\"" Sep 11 09:18:31.818562 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:18:31.818367 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-registry-tls\"" Sep 11 09:18:31.828506 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:18:31.828485 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-95b9b66c7-xvkzz"] Sep 11 09:18:31.866029 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:18:31.866001 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/ef59443e-e32b-460b-9a26-5b959f2c7cd4-online-tls\") pod \"feast-simple-feast-setup-95b9b66c7-xvkzz\" (UID: \"ef59443e-e32b-460b-9a26-5b959f2c7cd4\") " pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-xvkzz" Sep 11 09:18:31.866029 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:18:31.866030 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/ef59443e-e32b-460b-9a26-5b959f2c7cd4-registry-tls\") pod \"feast-simple-feast-setup-95b9b66c7-xvkzz\" (UID: \"ef59443e-e32b-460b-9a26-5b959f2c7cd4\") " pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-xvkzz" Sep 11 09:18:31.866262 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:18:31.866052 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/ef59443e-e32b-460b-9a26-5b959f2c7cd4-offline-tls\") pod \"feast-simple-feast-setup-95b9b66c7-xvkzz\" (UID: \"ef59443e-e32b-460b-9a26-5b959f2c7cd4\") " pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-xvkzz" Sep 11 09:18:31.866262 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:18:31.866087 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/ef59443e-e32b-460b-9a26-5b959f2c7cd4-ui-tls\") pod \"feast-simple-feast-setup-95b9b66c7-xvkzz\" (UID: \"ef59443e-e32b-460b-9a26-5b959f2c7cd4\") " pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-xvkzz" Sep 11 09:18:31.866262 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:18:31.866151 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/ef59443e-e32b-460b-9a26-5b959f2c7cd4-feast-data\") pod \"feast-simple-feast-setup-95b9b66c7-xvkzz\" (UID: \"ef59443e-e32b-460b-9a26-5b959f2c7cd4\") " pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-xvkzz" Sep 11 09:18:31.866262 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:18:31.866226 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s52l2\" (UniqueName: \"kubernetes.io/projected/ef59443e-e32b-460b-9a26-5b959f2c7cd4-kube-api-access-s52l2\") pod \"feast-simple-feast-setup-95b9b66c7-xvkzz\" (UID: \"ef59443e-e32b-460b-9a26-5b959f2c7cd4\") " pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-xvkzz" Sep 11 09:18:31.966671 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:18:31.966636 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s52l2\" (UniqueName: \"kubernetes.io/projected/ef59443e-e32b-460b-9a26-5b959f2c7cd4-kube-api-access-s52l2\") pod \"feast-simple-feast-setup-95b9b66c7-xvkzz\" (UID: \"ef59443e-e32b-460b-9a26-5b959f2c7cd4\") " pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-xvkzz" Sep 11 09:18:31.966872 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:18:31.966690 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/ef59443e-e32b-460b-9a26-5b959f2c7cd4-online-tls\") pod \"feast-simple-feast-setup-95b9b66c7-xvkzz\" (UID: \"ef59443e-e32b-460b-9a26-5b959f2c7cd4\") " pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-xvkzz" Sep 11 09:18:31.966872 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:18:31.966709 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/ef59443e-e32b-460b-9a26-5b959f2c7cd4-registry-tls\") pod \"feast-simple-feast-setup-95b9b66c7-xvkzz\" (UID: \"ef59443e-e32b-460b-9a26-5b959f2c7cd4\") " pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-xvkzz" Sep 11 09:18:31.966872 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:18:31.966729 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/ef59443e-e32b-460b-9a26-5b959f2c7cd4-offline-tls\") pod \"feast-simple-feast-setup-95b9b66c7-xvkzz\" (UID: \"ef59443e-e32b-460b-9a26-5b959f2c7cd4\") " pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-xvkzz" Sep 11 09:18:31.966872 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:18:31.966770 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/ef59443e-e32b-460b-9a26-5b959f2c7cd4-ui-tls\") pod \"feast-simple-feast-setup-95b9b66c7-xvkzz\" (UID: \"ef59443e-e32b-460b-9a26-5b959f2c7cd4\") " pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-xvkzz" Sep 11 09:18:31.966872 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:18:31.966795 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/ef59443e-e32b-460b-9a26-5b959f2c7cd4-feast-data\") pod \"feast-simple-feast-setup-95b9b66c7-xvkzz\" (UID: \"ef59443e-e32b-460b-9a26-5b959f2c7cd4\") " pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-xvkzz" Sep 11 09:18:31.966872 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:18:31.966810 2575 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-online-tls: secret "feast-simple-feast-setup-online-tls" not found Sep 11 09:18:31.967169 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:18:31.966886 2575 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-ui-tls: secret "feast-simple-feast-setup-ui-tls" not found Sep 11 09:18:31.967169 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:18:31.966894 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef59443e-e32b-460b-9a26-5b959f2c7cd4-online-tls podName:ef59443e-e32b-460b-9a26-5b959f2c7cd4 nodeName:}" failed. No retries permitted until 2026-09-11 09:18:32.466872119 +0000 UTC m=+328.807482304 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/ef59443e-e32b-460b-9a26-5b959f2c7cd4-online-tls") pod "feast-simple-feast-setup-95b9b66c7-xvkzz" (UID: "ef59443e-e32b-460b-9a26-5b959f2c7cd4") : secret "feast-simple-feast-setup-online-tls" not found Sep 11 09:18:31.967169 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:18:31.966956 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef59443e-e32b-460b-9a26-5b959f2c7cd4-ui-tls podName:ef59443e-e32b-460b-9a26-5b959f2c7cd4 nodeName:}" failed. No retries permitted until 2026-09-11 09:18:32.466941495 +0000 UTC m=+328.807551687 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "ui-tls" (UniqueName: "kubernetes.io/secret/ef59443e-e32b-460b-9a26-5b959f2c7cd4-ui-tls") pod "feast-simple-feast-setup-95b9b66c7-xvkzz" (UID: "ef59443e-e32b-460b-9a26-5b959f2c7cd4") : secret "feast-simple-feast-setup-ui-tls" not found Sep 11 09:18:31.967297 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:18:31.967233 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/ef59443e-e32b-460b-9a26-5b959f2c7cd4-feast-data\") pod \"feast-simple-feast-setup-95b9b66c7-xvkzz\" (UID: \"ef59443e-e32b-460b-9a26-5b959f2c7cd4\") " pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-xvkzz" Sep 11 09:18:31.969283 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:18:31.969261 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/ef59443e-e32b-460b-9a26-5b959f2c7cd4-registry-tls\") pod \"feast-simple-feast-setup-95b9b66c7-xvkzz\" (UID: \"ef59443e-e32b-460b-9a26-5b959f2c7cd4\") " pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-xvkzz" Sep 11 09:18:31.969376 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:18:31.969318 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/ef59443e-e32b-460b-9a26-5b959f2c7cd4-offline-tls\") pod \"feast-simple-feast-setup-95b9b66c7-xvkzz\" (UID: \"ef59443e-e32b-460b-9a26-5b959f2c7cd4\") " pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-xvkzz" Sep 11 09:18:31.975578 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:18:31.975556 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s52l2\" (UniqueName: \"kubernetes.io/projected/ef59443e-e32b-460b-9a26-5b959f2c7cd4-kube-api-access-s52l2\") pod \"feast-simple-feast-setup-95b9b66c7-xvkzz\" (UID: \"ef59443e-e32b-460b-9a26-5b959f2c7cd4\") " pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-xvkzz" Sep 11 09:18:32.472458 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:18:32.472413 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/ef59443e-e32b-460b-9a26-5b959f2c7cd4-ui-tls\") pod \"feast-simple-feast-setup-95b9b66c7-xvkzz\" (UID: \"ef59443e-e32b-460b-9a26-5b959f2c7cd4\") " pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-xvkzz" Sep 11 09:18:32.472690 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:18:32.472541 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/ef59443e-e32b-460b-9a26-5b959f2c7cd4-online-tls\") pod \"feast-simple-feast-setup-95b9b66c7-xvkzz\" (UID: \"ef59443e-e32b-460b-9a26-5b959f2c7cd4\") " pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-xvkzz" Sep 11 09:18:32.474922 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:18:32.474892 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/ef59443e-e32b-460b-9a26-5b959f2c7cd4-ui-tls\") pod \"feast-simple-feast-setup-95b9b66c7-xvkzz\" (UID: \"ef59443e-e32b-460b-9a26-5b959f2c7cd4\") " pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-xvkzz" Sep 11 09:18:32.475037 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:18:32.474939 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/ef59443e-e32b-460b-9a26-5b959f2c7cd4-online-tls\") pod \"feast-simple-feast-setup-95b9b66c7-xvkzz\" (UID: \"ef59443e-e32b-460b-9a26-5b959f2c7cd4\") " pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-xvkzz" Sep 11 09:18:32.732944 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:18:32.732841 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-xvkzz" Sep 11 09:18:33.011028 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:18:33.010999 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-95b9b66c7-xvkzz"] Sep 11 09:18:33.013068 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:18:33.013041 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podef59443e_e32b_460b_9a26_5b959f2c7cd4.slice/crio-1b135463b90cee33226c030a329cd889d4f888987a480ac728730998a4717400 WatchSource:0}: Error finding container 1b135463b90cee33226c030a329cd889d4f888987a480ac728730998a4717400: Status 404 returned error can't find the container with id 1b135463b90cee33226c030a329cd889d4f888987a480ac728730998a4717400 Sep 11 09:18:33.014986 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:18:33.014957 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 09:18:33.645870 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:18:33.645836 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-xvkzz" event={"ID":"ef59443e-e32b-460b-9a26-5b959f2c7cd4","Type":"ContainerStarted","Data":"1b135463b90cee33226c030a329cd889d4f888987a480ac728730998a4717400"} Sep 11 09:18:59.762951 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:18:59.762915 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-xvkzz" event={"ID":"ef59443e-e32b-460b-9a26-5b959f2c7cd4","Type":"ContainerStarted","Data":"805a389f16c5609ceac6daca3c272757db340a2e721b246aff3245bc7629d0bd"} Sep 11 09:19:04.789276 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:19:04.789190 2575 generic.go:356] "Generic (PLEG): container finished" podID="ef59443e-e32b-460b-9a26-5b959f2c7cd4" containerID="805a389f16c5609ceac6daca3c272757db340a2e721b246aff3245bc7629d0bd" exitCode=0 Sep 11 09:19:04.789276 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:19:04.789242 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-xvkzz" event={"ID":"ef59443e-e32b-460b-9a26-5b959f2c7cd4","Type":"ContainerDied","Data":"805a389f16c5609ceac6daca3c272757db340a2e721b246aff3245bc7629d0bd"} Sep 11 09:19:05.795741 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:19:05.795706 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-xvkzz" event={"ID":"ef59443e-e32b-460b-9a26-5b959f2c7cd4","Type":"ContainerStarted","Data":"f106c247ef0fb59ad3f7370e83215553820e1b0acec713f5afc76328cb177b12"} Sep 11 09:19:10.820054 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:19:10.820014 2575 generic.go:356] "Generic (PLEG): container finished" podID="ef59443e-e32b-460b-9a26-5b959f2c7cd4" containerID="f106c247ef0fb59ad3f7370e83215553820e1b0acec713f5afc76328cb177b12" exitCode=0 Sep 11 09:19:10.820434 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:19:10.820074 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-xvkzz" event={"ID":"ef59443e-e32b-460b-9a26-5b959f2c7cd4","Type":"ContainerDied","Data":"f106c247ef0fb59ad3f7370e83215553820e1b0acec713f5afc76328cb177b12"} Sep 11 09:19:10.829318 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:19:10.829296 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-intra-comm\"" Sep 11 09:19:11.835210 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:19:11.835165 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-xvkzz" event={"ID":"ef59443e-e32b-460b-9a26-5b959f2c7cd4","Type":"ContainerStarted","Data":"50f748ac813f1098e036771a2eb2473b30ad8a53fe964dd30d2670e3e185be8e"} Sep 11 09:19:11.835700 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:19:11.835234 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-xvkzz" event={"ID":"ef59443e-e32b-460b-9a26-5b959f2c7cd4","Type":"ContainerStarted","Data":"3a28736c33bbbbc1135ee28dd0dd17241910713a7b52bd1229c6b2a61f093a92"} Sep 11 09:19:11.835700 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:19:11.835257 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-xvkzz" event={"ID":"ef59443e-e32b-460b-9a26-5b959f2c7cd4","Type":"ContainerStarted","Data":"b211004e74892763d08b108581ccff2fb78b9d58b89ee2aca353ca7fa43559c9"} Sep 11 09:19:11.835700 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:19:11.835276 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-xvkzz" event={"ID":"ef59443e-e32b-460b-9a26-5b959f2c7cd4","Type":"ContainerStarted","Data":"66f7e1c2f582f9dc8a6abc3bd6acb844f30e61fae84e63ad736b0a63adc14eb1"} Sep 11 09:19:11.872108 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:19:11.872032 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-xvkzz" podStartSLOduration=14.320191851 podStartE2EDuration="40.872007677s" totalImagesPullingTime="26.551815826s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 09:18:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 09:19:11.86852631 +0000 UTC m=+368.209136517" watchObservedRunningTime="2026-09-11 09:19:11.872007677 +0000 UTC m=+368.212617885" Sep 11 09:19:12.733325 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:19:12.733285 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-xvkzz" Sep 11 09:19:12.733559 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:19:12.733360 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-xvkzz" Sep 11 09:19:12.733559 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:19:12.733387 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-xvkzz" Sep 11 09:19:12.733559 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:19:12.733406 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-xvkzz" Sep 11 09:19:14.733336 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:19:14.733294 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-xvkzz" Sep 11 09:19:14.733960 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:19:14.733934 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-xvkzz" Sep 11 09:19:14.733960 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:19:14.733968 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-xvkzz" Sep 11 09:19:14.734155 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:19:14.733989 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-xvkzz" Sep 11 09:19:14.735502 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:19:14.735435 2575 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-xvkzz" podUID="ef59443e-e32b-460b-9a26-5b959f2c7cd4" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.18:8443: connect: connection refused" Sep 11 09:19:14.735502 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:19:14.735460 2575 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-xvkzz" podUID="ef59443e-e32b-460b-9a26-5b959f2c7cd4" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.18:6571: connect: connection refused" Sep 11 09:19:14.735662 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:19:14.735491 2575 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-xvkzz" podUID="ef59443e-e32b-460b-9a26-5b959f2c7cd4" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.18:8816: connect: connection refused" Sep 11 09:19:14.735662 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:19:14.735567 2575 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-xvkzz" podUID="ef59443e-e32b-460b-9a26-5b959f2c7cd4" containerName="online" probeResult="failure" output="Get \"https://10.134.0.18:6567/health\": dial tcp 10.134.0.18:6567: connect: connection refused" Sep 11 09:19:17.734212 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:19:17.734157 2575 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-xvkzz" podUID="ef59443e-e32b-460b-9a26-5b959f2c7cd4" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.18:8443: connect: connection refused" Sep 11 09:19:17.734716 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:19:17.734158 2575 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-xvkzz" podUID="ef59443e-e32b-460b-9a26-5b959f2c7cd4" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.18:8816: connect: connection refused" Sep 11 09:19:17.734716 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:19:17.734180 2575 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-xvkzz" podUID="ef59443e-e32b-460b-9a26-5b959f2c7cd4" containerName="online" probeResult="failure" output="Get \"https://10.134.0.18:6567/health\": dial tcp 10.134.0.18:6567: connect: connection refused" Sep 11 09:19:17.734953 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:19:17.734917 2575 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-xvkzz" podUID="ef59443e-e32b-460b-9a26-5b959f2c7cd4" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.18:6571: connect: connection refused" Sep 11 09:19:20.734520 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:19:20.734484 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-xvkzz" Sep 11 09:19:20.734999 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:19:20.734804 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-xvkzz" Sep 11 09:19:20.735086 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:19:20.735044 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-xvkzz" Sep 11 09:19:20.735337 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:19:20.735314 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-xvkzz" Sep 11 09:19:20.735505 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:19:20.735488 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-xvkzz" Sep 11 09:19:20.740173 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:19:20.740154 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-xvkzz" Sep 11 09:19:20.884094 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:19:20.884063 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-xvkzz" Sep 11 09:19:20.888576 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:19:20.888551 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-xvkzz" Sep 11 09:20:22.161197 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:20:22.161162 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-95b9b66c7-xvkzz"] Sep 11 09:20:22.161789 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:20:22.161408 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-xvkzz" podUID="ef59443e-e32b-460b-9a26-5b959f2c7cd4" containerName="registry" containerID="cri-o://66f7e1c2f582f9dc8a6abc3bd6acb844f30e61fae84e63ad736b0a63adc14eb1" gracePeriod=30 Sep 11 09:20:22.161789 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:20:22.161506 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-xvkzz" podUID="ef59443e-e32b-460b-9a26-5b959f2c7cd4" containerName="ui" containerID="cri-o://50f748ac813f1098e036771a2eb2473b30ad8a53fe964dd30d2670e3e185be8e" gracePeriod=30 Sep 11 09:20:22.162369 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:20:22.162147 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-xvkzz" podUID="ef59443e-e32b-460b-9a26-5b959f2c7cd4" containerName="offline" containerID="cri-o://3a28736c33bbbbc1135ee28dd0dd17241910713a7b52bd1229c6b2a61f093a92" gracePeriod=30 Sep 11 09:20:22.163524 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:20:22.163331 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-xvkzz" podUID="ef59443e-e32b-460b-9a26-5b959f2c7cd4" containerName="online" containerID="cri-o://b211004e74892763d08b108581ccff2fb78b9d58b89ee2aca353ca7fa43559c9" gracePeriod=30 Sep 11 09:20:22.667297 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:20:22.667261 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-95b9b66c7-9rd5b"] Sep 11 09:20:22.670522 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:20:22.670503 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-9rd5b" Sep 11 09:20:22.673288 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:20:22.673266 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-jq6tq\"" Sep 11 09:20:22.685913 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:20:22.685886 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-95b9b66c7-9rd5b"] Sep 11 09:20:22.760896 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:20:22.760856 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gf9dk\" (UniqueName: \"kubernetes.io/projected/cfdada3e-9f75-4329-ad8f-0305a6d2e4d4-kube-api-access-gf9dk\") pod \"feast-simple-feast-setup-95b9b66c7-9rd5b\" (UID: \"cfdada3e-9f75-4329-ad8f-0305a6d2e4d4\") " pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-9rd5b" Sep 11 09:20:22.761092 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:20:22.760933 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/cfdada3e-9f75-4329-ad8f-0305a6d2e4d4-ui-tls\") pod \"feast-simple-feast-setup-95b9b66c7-9rd5b\" (UID: \"cfdada3e-9f75-4329-ad8f-0305a6d2e4d4\") " pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-9rd5b" Sep 11 09:20:22.761092 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:20:22.760980 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/cfdada3e-9f75-4329-ad8f-0305a6d2e4d4-registry-tls\") pod \"feast-simple-feast-setup-95b9b66c7-9rd5b\" (UID: \"cfdada3e-9f75-4329-ad8f-0305a6d2e4d4\") " pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-9rd5b" Sep 11 09:20:22.761092 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:20:22.761048 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/cfdada3e-9f75-4329-ad8f-0305a6d2e4d4-feast-data\") pod \"feast-simple-feast-setup-95b9b66c7-9rd5b\" (UID: \"cfdada3e-9f75-4329-ad8f-0305a6d2e4d4\") " pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-9rd5b" Sep 11 09:20:22.761234 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:20:22.761105 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/cfdada3e-9f75-4329-ad8f-0305a6d2e4d4-online-tls\") pod \"feast-simple-feast-setup-95b9b66c7-9rd5b\" (UID: \"cfdada3e-9f75-4329-ad8f-0305a6d2e4d4\") " pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-9rd5b" Sep 11 09:20:22.761234 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:20:22.761140 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/cfdada3e-9f75-4329-ad8f-0305a6d2e4d4-offline-tls\") pod \"feast-simple-feast-setup-95b9b66c7-9rd5b\" (UID: \"cfdada3e-9f75-4329-ad8f-0305a6d2e4d4\") " pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-9rd5b" Sep 11 09:20:22.862178 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:20:22.862141 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/cfdada3e-9f75-4329-ad8f-0305a6d2e4d4-online-tls\") pod \"feast-simple-feast-setup-95b9b66c7-9rd5b\" (UID: \"cfdada3e-9f75-4329-ad8f-0305a6d2e4d4\") " pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-9rd5b" Sep 11 09:20:22.862178 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:20:22.862187 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/cfdada3e-9f75-4329-ad8f-0305a6d2e4d4-offline-tls\") pod \"feast-simple-feast-setup-95b9b66c7-9rd5b\" (UID: \"cfdada3e-9f75-4329-ad8f-0305a6d2e4d4\") " pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-9rd5b" Sep 11 09:20:22.862448 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:20:22.862229 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gf9dk\" (UniqueName: \"kubernetes.io/projected/cfdada3e-9f75-4329-ad8f-0305a6d2e4d4-kube-api-access-gf9dk\") pod \"feast-simple-feast-setup-95b9b66c7-9rd5b\" (UID: \"cfdada3e-9f75-4329-ad8f-0305a6d2e4d4\") " pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-9rd5b" Sep 11 09:20:22.862448 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:20:22.862296 2575 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-online-tls: secret "feast-simple-feast-setup-online-tls" not found Sep 11 09:20:22.862448 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:20:22.862343 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/cfdada3e-9f75-4329-ad8f-0305a6d2e4d4-ui-tls\") pod \"feast-simple-feast-setup-95b9b66c7-9rd5b\" (UID: \"cfdada3e-9f75-4329-ad8f-0305a6d2e4d4\") " pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-9rd5b" Sep 11 09:20:22.862448 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:20:22.862374 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cfdada3e-9f75-4329-ad8f-0305a6d2e4d4-online-tls podName:cfdada3e-9f75-4329-ad8f-0305a6d2e4d4 nodeName:}" failed. No retries permitted until 2026-09-11 09:20:23.3623575 +0000 UTC m=+439.702967684 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/cfdada3e-9f75-4329-ad8f-0305a6d2e4d4-online-tls") pod "feast-simple-feast-setup-95b9b66c7-9rd5b" (UID: "cfdada3e-9f75-4329-ad8f-0305a6d2e4d4") : secret "feast-simple-feast-setup-online-tls" not found Sep 11 09:20:22.862448 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:20:22.862428 2575 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-ui-tls: secret "feast-simple-feast-setup-ui-tls" not found Sep 11 09:20:22.862448 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:20:22.862431 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/cfdada3e-9f75-4329-ad8f-0305a6d2e4d4-registry-tls\") pod \"feast-simple-feast-setup-95b9b66c7-9rd5b\" (UID: \"cfdada3e-9f75-4329-ad8f-0305a6d2e4d4\") " pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-9rd5b" Sep 11 09:20:22.862810 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:20:22.862519 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cfdada3e-9f75-4329-ad8f-0305a6d2e4d4-ui-tls podName:cfdada3e-9f75-4329-ad8f-0305a6d2e4d4 nodeName:}" failed. No retries permitted until 2026-09-11 09:20:23.362502639 +0000 UTC m=+439.703112822 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "ui-tls" (UniqueName: "kubernetes.io/secret/cfdada3e-9f75-4329-ad8f-0305a6d2e4d4-ui-tls") pod "feast-simple-feast-setup-95b9b66c7-9rd5b" (UID: "cfdada3e-9f75-4329-ad8f-0305a6d2e4d4") : secret "feast-simple-feast-setup-ui-tls" not found Sep 11 09:20:22.862810 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:20:22.862541 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/cfdada3e-9f75-4329-ad8f-0305a6d2e4d4-feast-data\") pod \"feast-simple-feast-setup-95b9b66c7-9rd5b\" (UID: \"cfdada3e-9f75-4329-ad8f-0305a6d2e4d4\") " pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-9rd5b" Sep 11 09:20:22.863004 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:20:22.862985 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/cfdada3e-9f75-4329-ad8f-0305a6d2e4d4-feast-data\") pod \"feast-simple-feast-setup-95b9b66c7-9rd5b\" (UID: \"cfdada3e-9f75-4329-ad8f-0305a6d2e4d4\") " pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-9rd5b" Sep 11 09:20:22.864785 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:20:22.864756 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/cfdada3e-9f75-4329-ad8f-0305a6d2e4d4-registry-tls\") pod \"feast-simple-feast-setup-95b9b66c7-9rd5b\" (UID: \"cfdada3e-9f75-4329-ad8f-0305a6d2e4d4\") " pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-9rd5b" Sep 11 09:20:22.864901 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:20:22.864823 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/cfdada3e-9f75-4329-ad8f-0305a6d2e4d4-offline-tls\") pod \"feast-simple-feast-setup-95b9b66c7-9rd5b\" (UID: \"cfdada3e-9f75-4329-ad8f-0305a6d2e4d4\") " pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-9rd5b" Sep 11 09:20:22.870853 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:20:22.870829 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gf9dk\" (UniqueName: \"kubernetes.io/projected/cfdada3e-9f75-4329-ad8f-0305a6d2e4d4-kube-api-access-gf9dk\") pod \"feast-simple-feast-setup-95b9b66c7-9rd5b\" (UID: \"cfdada3e-9f75-4329-ad8f-0305a6d2e4d4\") " pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-9rd5b" Sep 11 09:20:23.145481 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:20:23.145435 2575 generic.go:356] "Generic (PLEG): container finished" podID="ef59443e-e32b-460b-9a26-5b959f2c7cd4" containerID="50f748ac813f1098e036771a2eb2473b30ad8a53fe964dd30d2670e3e185be8e" exitCode=0 Sep 11 09:20:23.145481 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:20:23.145460 2575 generic.go:356] "Generic (PLEG): container finished" podID="ef59443e-e32b-460b-9a26-5b959f2c7cd4" containerID="b211004e74892763d08b108581ccff2fb78b9d58b89ee2aca353ca7fa43559c9" exitCode=0 Sep 11 09:20:23.145693 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:20:23.145508 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-xvkzz" event={"ID":"ef59443e-e32b-460b-9a26-5b959f2c7cd4","Type":"ContainerDied","Data":"50f748ac813f1098e036771a2eb2473b30ad8a53fe964dd30d2670e3e185be8e"} Sep 11 09:20:23.145693 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:20:23.145553 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-xvkzz" event={"ID":"ef59443e-e32b-460b-9a26-5b959f2c7cd4","Type":"ContainerDied","Data":"b211004e74892763d08b108581ccff2fb78b9d58b89ee2aca353ca7fa43559c9"} Sep 11 09:20:23.366138 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:20:23.366096 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/cfdada3e-9f75-4329-ad8f-0305a6d2e4d4-ui-tls\") pod \"feast-simple-feast-setup-95b9b66c7-9rd5b\" (UID: \"cfdada3e-9f75-4329-ad8f-0305a6d2e4d4\") " pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-9rd5b" Sep 11 09:20:23.366534 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:20:23.366185 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/cfdada3e-9f75-4329-ad8f-0305a6d2e4d4-online-tls\") pod \"feast-simple-feast-setup-95b9b66c7-9rd5b\" (UID: \"cfdada3e-9f75-4329-ad8f-0305a6d2e4d4\") " pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-9rd5b" Sep 11 09:20:23.368692 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:20:23.368664 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/cfdada3e-9f75-4329-ad8f-0305a6d2e4d4-ui-tls\") pod \"feast-simple-feast-setup-95b9b66c7-9rd5b\" (UID: \"cfdada3e-9f75-4329-ad8f-0305a6d2e4d4\") " pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-9rd5b" Sep 11 09:20:23.368808 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:20:23.368664 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/cfdada3e-9f75-4329-ad8f-0305a6d2e4d4-online-tls\") pod \"feast-simple-feast-setup-95b9b66c7-9rd5b\" (UID: \"cfdada3e-9f75-4329-ad8f-0305a6d2e4d4\") " pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-9rd5b" Sep 11 09:20:23.581403 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:20:23.581363 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-9rd5b" Sep 11 09:20:23.859028 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:20:23.858968 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-95b9b66c7-9rd5b"] Sep 11 09:20:23.861250 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:20:23.861226 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcfdada3e_9f75_4329_ad8f_0305a6d2e4d4.slice/crio-944fd836887977e96df8830b9aa07d0b55da8c9ac6893c57c51c20ed6de23a18 WatchSource:0}: Error finding container 944fd836887977e96df8830b9aa07d0b55da8c9ac6893c57c51c20ed6de23a18: Status 404 returned error can't find the container with id 944fd836887977e96df8830b9aa07d0b55da8c9ac6893c57c51c20ed6de23a18 Sep 11 09:20:24.154763 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:20:24.154663 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-9rd5b" event={"ID":"cfdada3e-9f75-4329-ad8f-0305a6d2e4d4","Type":"ContainerStarted","Data":"85e3cb1b856aa3be504822bfe22f58a15df5dc0e2d75b4d19288b260fd593e08"} Sep 11 09:20:24.154763 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:20:24.154726 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-9rd5b" event={"ID":"cfdada3e-9f75-4329-ad8f-0305a6d2e4d4","Type":"ContainerStarted","Data":"944fd836887977e96df8830b9aa07d0b55da8c9ac6893c57c51c20ed6de23a18"} Sep 11 09:20:29.176961 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:20:29.176928 2575 generic.go:356] "Generic (PLEG): container finished" podID="cfdada3e-9f75-4329-ad8f-0305a6d2e4d4" containerID="85e3cb1b856aa3be504822bfe22f58a15df5dc0e2d75b4d19288b260fd593e08" exitCode=0 Sep 11 09:20:29.177317 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:20:29.177002 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-9rd5b" event={"ID":"cfdada3e-9f75-4329-ad8f-0305a6d2e4d4","Type":"ContainerDied","Data":"85e3cb1b856aa3be504822bfe22f58a15df5dc0e2d75b4d19288b260fd593e08"} Sep 11 09:20:30.183655 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:20:30.183618 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-9rd5b" event={"ID":"cfdada3e-9f75-4329-ad8f-0305a6d2e4d4","Type":"ContainerStarted","Data":"4d7333e770079a9f2945f9a13dd14af095a0818f1e6db59cc24cfec9ffec6487"} Sep 11 09:20:30.735384 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:20:30.735333 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-xvkzz" podUID="ef59443e-e32b-460b-9a26-5b959f2c7cd4" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.18:8443: connect: connection refused" Sep 11 09:20:30.884791 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:20:30.884741 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-xvkzz" podUID="ef59443e-e32b-460b-9a26-5b959f2c7cd4" containerName="online" probeResult="failure" output="Get \"https://10.134.0.18:6567/health\": dial tcp 10.134.0.18:6567: connect: connection refused" Sep 11 09:20:35.208342 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:20:35.208308 2575 generic.go:356] "Generic (PLEG): container finished" podID="cfdada3e-9f75-4329-ad8f-0305a6d2e4d4" containerID="4d7333e770079a9f2945f9a13dd14af095a0818f1e6db59cc24cfec9ffec6487" exitCode=0 Sep 11 09:20:35.208830 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:20:35.208380 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-9rd5b" event={"ID":"cfdada3e-9f75-4329-ad8f-0305a6d2e4d4","Type":"ContainerDied","Data":"4d7333e770079a9f2945f9a13dd14af095a0818f1e6db59cc24cfec9ffec6487"} Sep 11 09:20:36.219309 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:20:36.219251 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-9rd5b" event={"ID":"cfdada3e-9f75-4329-ad8f-0305a6d2e4d4","Type":"ContainerStarted","Data":"959fa38a0f10c849bb857ec25d1395cb59c588beee59f3f1121660a99b57fe06"} Sep 11 09:20:36.219792 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:20:36.219331 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-9rd5b" event={"ID":"cfdada3e-9f75-4329-ad8f-0305a6d2e4d4","Type":"ContainerStarted","Data":"854357ae71636866704e60cb51cb7f7499655d7d7cd626e97742fde096a6ea52"} Sep 11 09:20:36.219792 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:20:36.219762 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-9rd5b" event={"ID":"cfdada3e-9f75-4329-ad8f-0305a6d2e4d4","Type":"ContainerStarted","Data":"178c5c755beb76e04665f96da6ba5e071ca0f0e6d031ea0e598dd5c45d8aca66"} Sep 11 09:20:36.219945 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:20:36.219791 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-9rd5b" event={"ID":"cfdada3e-9f75-4329-ad8f-0305a6d2e4d4","Type":"ContainerStarted","Data":"3e623297693ca54d3b8413a804e360d70d28b4a7bd9ebf48fd604075fe119e07"} Sep 11 09:20:36.248884 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:20:36.248805 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-9rd5b" podStartSLOduration=14.248781722 podStartE2EDuration="14.248781722s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 09:20:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 09:20:36.246239273 +0000 UTC m=+452.586849480" watchObservedRunningTime="2026-09-11 09:20:36.248781722 +0000 UTC m=+452.589391930" Sep 11 09:20:38.582334 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:20:38.582282 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-9rd5b" Sep 11 09:20:38.582833 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:20:38.582442 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-9rd5b" Sep 11 09:20:38.582833 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:20:38.582480 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-9rd5b" Sep 11 09:20:38.582833 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:20:38.582498 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-9rd5b" Sep 11 09:20:38.584671 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:20:38.584624 2575 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-9rd5b" podUID="cfdada3e-9f75-4329-ad8f-0305a6d2e4d4" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.19:8816: connect: connection refused" Sep 11 09:20:38.584671 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:20:38.584647 2575 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-9rd5b" podUID="cfdada3e-9f75-4329-ad8f-0305a6d2e4d4" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.19:8443: connect: connection refused" Sep 11 09:20:38.584865 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:20:38.584694 2575 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-9rd5b" podUID="cfdada3e-9f75-4329-ad8f-0305a6d2e4d4" containerName="online" probeResult="failure" output="Get \"https://10.134.0.19:6567/health\": dial tcp 10.134.0.19:6567: connect: connection refused" Sep 11 09:20:38.584865 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:20:38.584796 2575 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-9rd5b" podUID="cfdada3e-9f75-4329-ad8f-0305a6d2e4d4" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.19:6571: connect: connection refused" Sep 11 09:20:40.735666 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:20:40.735553 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-xvkzz" podUID="ef59443e-e32b-460b-9a26-5b959f2c7cd4" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.18:8443: connect: connection refused" Sep 11 09:20:40.884177 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:20:40.884127 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-xvkzz" podUID="ef59443e-e32b-460b-9a26-5b959f2c7cd4" containerName="online" probeResult="failure" output="Get \"https://10.134.0.18:6567/health\": dial tcp 10.134.0.18:6567: connect: connection refused" Sep 11 09:20:41.582673 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:20:41.582618 2575 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-9rd5b" podUID="cfdada3e-9f75-4329-ad8f-0305a6d2e4d4" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.19:8816: connect: connection refused" Sep 11 09:20:41.582875 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:20:41.582619 2575 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-9rd5b" podUID="cfdada3e-9f75-4329-ad8f-0305a6d2e4d4" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.19:6571: connect: connection refused" Sep 11 09:20:41.582875 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:20:41.582738 2575 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-9rd5b" podUID="cfdada3e-9f75-4329-ad8f-0305a6d2e4d4" containerName="online" probeResult="failure" output="Get \"https://10.134.0.19:6567/health\": dial tcp 10.134.0.19:6567: connect: connection refused" Sep 11 09:20:41.582875 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:20:41.582619 2575 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-9rd5b" podUID="cfdada3e-9f75-4329-ad8f-0305a6d2e4d4" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.19:8443: connect: connection refused" Sep 11 09:20:43.582390 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:20:43.582346 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-9rd5b" Sep 11 09:20:43.582813 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:20:43.582421 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-9rd5b" Sep 11 09:20:43.582813 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:20:43.582447 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-9rd5b" Sep 11 09:20:43.582813 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:20:43.582494 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-9rd5b" Sep 11 09:20:44.582785 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:20:44.582746 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-9rd5b" Sep 11 09:20:44.583202 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:20:44.582911 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-9rd5b" Sep 11 09:20:44.583283 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:20:44.583213 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-9rd5b" Sep 11 09:20:44.587916 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:20:44.587896 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-9rd5b" Sep 11 09:20:45.269695 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:20:45.269666 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-9rd5b" Sep 11 09:20:45.269947 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:20:45.269926 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-9rd5b" Sep 11 09:20:45.270231 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:20:45.270210 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-9rd5b" Sep 11 09:20:45.272512 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:20:45.272494 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-9rd5b" Sep 11 09:20:50.735283 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:20:50.735233 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-xvkzz" podUID="ef59443e-e32b-460b-9a26-5b959f2c7cd4" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.18:8443: connect: connection refused" Sep 11 09:20:50.735688 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:20:50.735372 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-xvkzz" Sep 11 09:20:50.884532 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:20:50.884458 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-xvkzz" podUID="ef59443e-e32b-460b-9a26-5b959f2c7cd4" containerName="online" probeResult="failure" output="Get \"https://10.134.0.18:6567/health\": dial tcp 10.134.0.18:6567: connect: connection refused" Sep 11 09:20:50.884732 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:20:50.884640 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-xvkzz" Sep 11 09:20:52.301511 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:20:52.301456 2575 generic.go:356] "Generic (PLEG): container finished" podID="ef59443e-e32b-460b-9a26-5b959f2c7cd4" containerID="3a28736c33bbbbc1135ee28dd0dd17241910713a7b52bd1229c6b2a61f093a92" exitCode=137 Sep 11 09:20:52.301511 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:20:52.301501 2575 generic.go:356] "Generic (PLEG): container finished" podID="ef59443e-e32b-460b-9a26-5b959f2c7cd4" containerID="66f7e1c2f582f9dc8a6abc3bd6acb844f30e61fae84e63ad736b0a63adc14eb1" exitCode=137 Sep 11 09:20:52.301511 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:20:52.301497 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-xvkzz" event={"ID":"ef59443e-e32b-460b-9a26-5b959f2c7cd4","Type":"ContainerDied","Data":"3a28736c33bbbbc1135ee28dd0dd17241910713a7b52bd1229c6b2a61f093a92"} Sep 11 09:20:52.301931 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:20:52.301542 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-xvkzz" event={"ID":"ef59443e-e32b-460b-9a26-5b959f2c7cd4","Type":"ContainerDied","Data":"66f7e1c2f582f9dc8a6abc3bd6acb844f30e61fae84e63ad736b0a63adc14eb1"} Sep 11 09:20:52.820353 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:20:52.820329 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-xvkzz" Sep 11 09:20:52.947236 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:20:52.947141 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/ef59443e-e32b-460b-9a26-5b959f2c7cd4-registry-tls\") pod \"ef59443e-e32b-460b-9a26-5b959f2c7cd4\" (UID: \"ef59443e-e32b-460b-9a26-5b959f2c7cd4\") " Sep 11 09:20:52.947236 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:20:52.947206 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/ef59443e-e32b-460b-9a26-5b959f2c7cd4-online-tls\") pod \"ef59443e-e32b-460b-9a26-5b959f2c7cd4\" (UID: \"ef59443e-e32b-460b-9a26-5b959f2c7cd4\") " Sep 11 09:20:52.947236 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:20:52.947223 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/ef59443e-e32b-460b-9a26-5b959f2c7cd4-offline-tls\") pod \"ef59443e-e32b-460b-9a26-5b959f2c7cd4\" (UID: \"ef59443e-e32b-460b-9a26-5b959f2c7cd4\") " Sep 11 09:20:52.947563 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:20:52.947343 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/ef59443e-e32b-460b-9a26-5b959f2c7cd4-ui-tls\") pod \"ef59443e-e32b-460b-9a26-5b959f2c7cd4\" (UID: \"ef59443e-e32b-460b-9a26-5b959f2c7cd4\") " Sep 11 09:20:52.947563 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:20:52.947384 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/ef59443e-e32b-460b-9a26-5b959f2c7cd4-feast-data\") pod \"ef59443e-e32b-460b-9a26-5b959f2c7cd4\" (UID: \"ef59443e-e32b-460b-9a26-5b959f2c7cd4\") " Sep 11 09:20:52.947563 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:20:52.947416 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s52l2\" (UniqueName: \"kubernetes.io/projected/ef59443e-e32b-460b-9a26-5b959f2c7cd4-kube-api-access-s52l2\") pod \"ef59443e-e32b-460b-9a26-5b959f2c7cd4\" (UID: \"ef59443e-e32b-460b-9a26-5b959f2c7cd4\") " Sep 11 09:20:52.948009 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:20:52.947951 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef59443e-e32b-460b-9a26-5b959f2c7cd4-feast-data" (OuterVolumeSpecName: "feast-data") pod "ef59443e-e32b-460b-9a26-5b959f2c7cd4" (UID: "ef59443e-e32b-460b-9a26-5b959f2c7cd4"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 09:20:52.949528 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:20:52.949500 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef59443e-e32b-460b-9a26-5b959f2c7cd4-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "ef59443e-e32b-460b-9a26-5b959f2c7cd4" (UID: "ef59443e-e32b-460b-9a26-5b959f2c7cd4"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 09:20:52.949620 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:20:52.949574 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef59443e-e32b-460b-9a26-5b959f2c7cd4-kube-api-access-s52l2" (OuterVolumeSpecName: "kube-api-access-s52l2") pod "ef59443e-e32b-460b-9a26-5b959f2c7cd4" (UID: "ef59443e-e32b-460b-9a26-5b959f2c7cd4"). InnerVolumeSpecName "kube-api-access-s52l2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 09:20:52.949620 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:20:52.949596 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef59443e-e32b-460b-9a26-5b959f2c7cd4-online-tls" (OuterVolumeSpecName: "online-tls") pod "ef59443e-e32b-460b-9a26-5b959f2c7cd4" (UID: "ef59443e-e32b-460b-9a26-5b959f2c7cd4"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 09:20:52.949767 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:20:52.949748 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef59443e-e32b-460b-9a26-5b959f2c7cd4-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "ef59443e-e32b-460b-9a26-5b959f2c7cd4" (UID: "ef59443e-e32b-460b-9a26-5b959f2c7cd4"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 09:20:52.949825 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:20:52.949803 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef59443e-e32b-460b-9a26-5b959f2c7cd4-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "ef59443e-e32b-460b-9a26-5b959f2c7cd4" (UID: "ef59443e-e32b-460b-9a26-5b959f2c7cd4"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 09:20:53.048857 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:20:53.048821 2575 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/ef59443e-e32b-460b-9a26-5b959f2c7cd4-registry-tls\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\"" Sep 11 09:20:53.048857 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:20:53.048851 2575 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/ef59443e-e32b-460b-9a26-5b959f2c7cd4-online-tls\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\"" Sep 11 09:20:53.048857 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:20:53.048860 2575 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/ef59443e-e32b-460b-9a26-5b959f2c7cd4-offline-tls\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\"" Sep 11 09:20:53.048857 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:20:53.048867 2575 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/ef59443e-e32b-460b-9a26-5b959f2c7cd4-ui-tls\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\"" Sep 11 09:20:53.049166 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:20:53.048875 2575 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/ef59443e-e32b-460b-9a26-5b959f2c7cd4-feast-data\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\"" Sep 11 09:20:53.049166 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:20:53.048884 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s52l2\" (UniqueName: \"kubernetes.io/projected/ef59443e-e32b-460b-9a26-5b959f2c7cd4-kube-api-access-s52l2\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\"" Sep 11 09:20:53.308795 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:20:53.308762 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-xvkzz" event={"ID":"ef59443e-e32b-460b-9a26-5b959f2c7cd4","Type":"ContainerDied","Data":"1b135463b90cee33226c030a329cd889d4f888987a480ac728730998a4717400"} Sep 11 09:20:53.308795 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:20:53.308782 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-xvkzz" Sep 11 09:20:53.309313 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:20:53.308822 2575 scope.go:117] "RemoveContainer" containerID="50f748ac813f1098e036771a2eb2473b30ad8a53fe964dd30d2670e3e185be8e" Sep 11 09:20:53.320638 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:20:53.319585 2575 scope.go:117] "RemoveContainer" containerID="3a28736c33bbbbc1135ee28dd0dd17241910713a7b52bd1229c6b2a61f093a92" Sep 11 09:20:53.328088 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:20:53.328062 2575 scope.go:117] "RemoveContainer" containerID="b211004e74892763d08b108581ccff2fb78b9d58b89ee2aca353ca7fa43559c9" Sep 11 09:20:53.335803 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:20:53.335785 2575 scope.go:117] "RemoveContainer" containerID="66f7e1c2f582f9dc8a6abc3bd6acb844f30e61fae84e63ad736b0a63adc14eb1" Sep 11 09:20:53.338067 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:20:53.338045 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-95b9b66c7-xvkzz"] Sep 11 09:20:53.341247 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:20:53.341227 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-95b9b66c7-xvkzz"] Sep 11 09:20:53.345668 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:20:53.345442 2575 scope.go:117] "RemoveContainer" containerID="f106c247ef0fb59ad3f7370e83215553820e1b0acec713f5afc76328cb177b12" Sep 11 09:20:53.353805 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:20:53.353782 2575 scope.go:117] "RemoveContainer" containerID="805a389f16c5609ceac6daca3c272757db340a2e721b246aff3245bc7629d0bd" Sep 11 09:20:54.114456 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:20:54.114421 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef59443e-e32b-460b-9a26-5b959f2c7cd4" path="/var/lib/kubelet/pods/ef59443e-e32b-460b-9a26-5b959f2c7cd4/volumes" Sep 11 09:21:47.576757 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:21:47.576717 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-cf947b468-g775t"] Sep 11 09:21:47.577216 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:21:47.577054 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef59443e-e32b-460b-9a26-5b959f2c7cd4" containerName="feast-apply" Sep 11 09:21:47.577216 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:21:47.577064 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef59443e-e32b-460b-9a26-5b959f2c7cd4" containerName="feast-apply" Sep 11 09:21:47.577216 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:21:47.577082 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef59443e-e32b-460b-9a26-5b959f2c7cd4" containerName="online" Sep 11 09:21:47.577216 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:21:47.577087 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef59443e-e32b-460b-9a26-5b959f2c7cd4" containerName="online" Sep 11 09:21:47.577216 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:21:47.577095 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef59443e-e32b-460b-9a26-5b959f2c7cd4" containerName="ui" Sep 11 09:21:47.577216 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:21:47.577100 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef59443e-e32b-460b-9a26-5b959f2c7cd4" containerName="ui" Sep 11 09:21:47.577216 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:21:47.577110 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef59443e-e32b-460b-9a26-5b959f2c7cd4" containerName="feast-init" Sep 11 09:21:47.577216 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:21:47.577116 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef59443e-e32b-460b-9a26-5b959f2c7cd4" containerName="feast-init" Sep 11 09:21:47.577216 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:21:47.577122 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef59443e-e32b-460b-9a26-5b959f2c7cd4" containerName="registry" Sep 11 09:21:47.577216 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:21:47.577127 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef59443e-e32b-460b-9a26-5b959f2c7cd4" containerName="registry" Sep 11 09:21:47.577216 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:21:47.577132 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef59443e-e32b-460b-9a26-5b959f2c7cd4" containerName="offline" Sep 11 09:21:47.577216 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:21:47.577137 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef59443e-e32b-460b-9a26-5b959f2c7cd4" containerName="offline" Sep 11 09:21:47.577216 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:21:47.577190 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef59443e-e32b-460b-9a26-5b959f2c7cd4" containerName="offline" Sep 11 09:21:47.577216 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:21:47.577198 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef59443e-e32b-460b-9a26-5b959f2c7cd4" containerName="ui" Sep 11 09:21:47.577216 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:21:47.577207 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef59443e-e32b-460b-9a26-5b959f2c7cd4" containerName="registry" Sep 11 09:21:47.577216 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:21:47.577213 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef59443e-e32b-460b-9a26-5b959f2c7cd4" containerName="online" Sep 11 09:21:47.580652 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:21:47.580635 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-cf947b468-g775t" Sep 11 09:21:47.583154 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:21:47.583130 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-intra-comm\"" Sep 11 09:21:47.583154 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:21:47.583134 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-offline-tls\"" Sep 11 09:21:47.583425 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:21:47.583408 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-client-ca\"" Sep 11 09:21:47.583729 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:21:47.583711 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"openshift-service-ca.crt\"" Sep 11 09:21:47.583805 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:21:47.583746 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-online-tls\"" Sep 11 09:21:47.583805 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:21:47.583771 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"kube-root-ca.crt\"" Sep 11 09:21:47.583805 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:21:47.583785 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-ui-tls\"" Sep 11 09:21:47.583941 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:21:47.583801 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-dockercfg-4cjsc\"" Sep 11 09:21:47.592450 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:21:47.592428 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-cf947b468-g775t"] Sep 11 09:21:47.619349 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:21:47.619316 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/1db82794-0c4f-4114-9b07-669edbf49db9-registry-tls\") pod \"feast-simple-feast-remote-setup-cf947b468-g775t\" (UID: \"1db82794-0c4f-4114-9b07-669edbf49db9\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-cf947b468-g775t" Sep 11 09:21:47.619512 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:21:47.619358 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-984hq\" (UniqueName: \"kubernetes.io/projected/1db82794-0c4f-4114-9b07-669edbf49db9-kube-api-access-984hq\") pod \"feast-simple-feast-remote-setup-cf947b468-g775t\" (UID: \"1db82794-0c4f-4114-9b07-669edbf49db9\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-cf947b468-g775t" Sep 11 09:21:47.619512 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:21:47.619413 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/1db82794-0c4f-4114-9b07-669edbf49db9-ui-tls\") pod \"feast-simple-feast-remote-setup-cf947b468-g775t\" (UID: \"1db82794-0c4f-4114-9b07-669edbf49db9\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-cf947b468-g775t" Sep 11 09:21:47.619512 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:21:47.619450 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/1db82794-0c4f-4114-9b07-669edbf49db9-online-tls\") pod \"feast-simple-feast-remote-setup-cf947b468-g775t\" (UID: \"1db82794-0c4f-4114-9b07-669edbf49db9\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-cf947b468-g775t" Sep 11 09:21:47.619618 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:21:47.619510 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/1db82794-0c4f-4114-9b07-669edbf49db9-offline-tls\") pod \"feast-simple-feast-remote-setup-cf947b468-g775t\" (UID: \"1db82794-0c4f-4114-9b07-669edbf49db9\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-cf947b468-g775t" Sep 11 09:21:47.619618 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:21:47.619534 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/1db82794-0c4f-4114-9b07-669edbf49db9-feast-data\") pod \"feast-simple-feast-remote-setup-cf947b468-g775t\" (UID: \"1db82794-0c4f-4114-9b07-669edbf49db9\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-cf947b468-g775t" Sep 11 09:21:47.720213 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:21:47.720172 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/1db82794-0c4f-4114-9b07-669edbf49db9-online-tls\") pod \"feast-simple-feast-remote-setup-cf947b468-g775t\" (UID: \"1db82794-0c4f-4114-9b07-669edbf49db9\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-cf947b468-g775t" Sep 11 09:21:47.720213 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:21:47.720214 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/1db82794-0c4f-4114-9b07-669edbf49db9-offline-tls\") pod \"feast-simple-feast-remote-setup-cf947b468-g775t\" (UID: \"1db82794-0c4f-4114-9b07-669edbf49db9\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-cf947b468-g775t" Sep 11 09:21:47.720501 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:21:47.720235 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/1db82794-0c4f-4114-9b07-669edbf49db9-feast-data\") pod \"feast-simple-feast-remote-setup-cf947b468-g775t\" (UID: \"1db82794-0c4f-4114-9b07-669edbf49db9\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-cf947b468-g775t" Sep 11 09:21:47.720501 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:21:47.720313 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/1db82794-0c4f-4114-9b07-669edbf49db9-registry-tls\") pod \"feast-simple-feast-remote-setup-cf947b468-g775t\" (UID: \"1db82794-0c4f-4114-9b07-669edbf49db9\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-cf947b468-g775t" Sep 11 09:21:47.720501 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:21:47.720339 2575 secret.go:189] Couldn't get secret test-ns-remote-registry/feast-simple-feast-remote-setup-offline-tls: secret "feast-simple-feast-remote-setup-offline-tls" not found Sep 11 09:21:47.720501 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:21:47.720362 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-984hq\" (UniqueName: \"kubernetes.io/projected/1db82794-0c4f-4114-9b07-669edbf49db9-kube-api-access-984hq\") pod \"feast-simple-feast-remote-setup-cf947b468-g775t\" (UID: \"1db82794-0c4f-4114-9b07-669edbf49db9\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-cf947b468-g775t" Sep 11 09:21:47.720501 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:21:47.720406 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/1db82794-0c4f-4114-9b07-669edbf49db9-ui-tls\") pod \"feast-simple-feast-remote-setup-cf947b468-g775t\" (UID: \"1db82794-0c4f-4114-9b07-669edbf49db9\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-cf947b468-g775t" Sep 11 09:21:47.720501 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:21:47.720339 2575 secret.go:189] Couldn't get secret test-ns-remote-registry/feast-simple-feast-remote-setup-online-tls: secret "feast-simple-feast-remote-setup-online-tls" not found Sep 11 09:21:47.720501 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:21:47.720439 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1db82794-0c4f-4114-9b07-669edbf49db9-offline-tls podName:1db82794-0c4f-4114-9b07-669edbf49db9 nodeName:}" failed. No retries permitted until 2026-09-11 09:21:48.220416352 +0000 UTC m=+524.561026539 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/1db82794-0c4f-4114-9b07-669edbf49db9-offline-tls") pod "feast-simple-feast-remote-setup-cf947b468-g775t" (UID: "1db82794-0c4f-4114-9b07-669edbf49db9") : secret "feast-simple-feast-remote-setup-offline-tls" not found Sep 11 09:21:47.720810 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:21:47.720536 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1db82794-0c4f-4114-9b07-669edbf49db9-online-tls podName:1db82794-0c4f-4114-9b07-669edbf49db9 nodeName:}" failed. No retries permitted until 2026-09-11 09:21:48.220513727 +0000 UTC m=+524.561123913 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/1db82794-0c4f-4114-9b07-669edbf49db9-online-tls") pod "feast-simple-feast-remote-setup-cf947b468-g775t" (UID: "1db82794-0c4f-4114-9b07-669edbf49db9") : secret "feast-simple-feast-remote-setup-online-tls" not found Sep 11 09:21:47.720810 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:21:47.720536 2575 secret.go:189] Couldn't get secret test-ns-remote-registry/feast-simple-feast-remote-setup-ui-tls: secret "feast-simple-feast-remote-setup-ui-tls" not found Sep 11 09:21:47.720810 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:21:47.720578 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1db82794-0c4f-4114-9b07-669edbf49db9-ui-tls podName:1db82794-0c4f-4114-9b07-669edbf49db9 nodeName:}" failed. No retries permitted until 2026-09-11 09:21:48.220568951 +0000 UTC m=+524.561179134 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "ui-tls" (UniqueName: "kubernetes.io/secret/1db82794-0c4f-4114-9b07-669edbf49db9-ui-tls") pod "feast-simple-feast-remote-setup-cf947b468-g775t" (UID: "1db82794-0c4f-4114-9b07-669edbf49db9") : secret "feast-simple-feast-remote-setup-ui-tls" not found Sep 11 09:21:47.720810 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:21:47.720715 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/1db82794-0c4f-4114-9b07-669edbf49db9-feast-data\") pod \"feast-simple-feast-remote-setup-cf947b468-g775t\" (UID: \"1db82794-0c4f-4114-9b07-669edbf49db9\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-cf947b468-g775t" Sep 11 09:21:47.721011 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:21:47.720994 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/1db82794-0c4f-4114-9b07-669edbf49db9-registry-tls\") pod \"feast-simple-feast-remote-setup-cf947b468-g775t\" (UID: \"1db82794-0c4f-4114-9b07-669edbf49db9\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-cf947b468-g775t" Sep 11 09:21:47.732323 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:21:47.732295 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-984hq\" (UniqueName: \"kubernetes.io/projected/1db82794-0c4f-4114-9b07-669edbf49db9-kube-api-access-984hq\") pod \"feast-simple-feast-remote-setup-cf947b468-g775t\" (UID: \"1db82794-0c4f-4114-9b07-669edbf49db9\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-cf947b468-g775t" Sep 11 09:21:48.224817 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:21:48.224775 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/1db82794-0c4f-4114-9b07-669edbf49db9-ui-tls\") pod \"feast-simple-feast-remote-setup-cf947b468-g775t\" (UID: \"1db82794-0c4f-4114-9b07-669edbf49db9\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-cf947b468-g775t" Sep 11 09:21:48.224817 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:21:48.224821 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/1db82794-0c4f-4114-9b07-669edbf49db9-online-tls\") pod \"feast-simple-feast-remote-setup-cf947b468-g775t\" (UID: \"1db82794-0c4f-4114-9b07-669edbf49db9\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-cf947b468-g775t" Sep 11 09:21:48.225091 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:21:48.224841 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/1db82794-0c4f-4114-9b07-669edbf49db9-offline-tls\") pod \"feast-simple-feast-remote-setup-cf947b468-g775t\" (UID: \"1db82794-0c4f-4114-9b07-669edbf49db9\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-cf947b468-g775t" Sep 11 09:21:48.227285 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:21:48.227254 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/1db82794-0c4f-4114-9b07-669edbf49db9-offline-tls\") pod \"feast-simple-feast-remote-setup-cf947b468-g775t\" (UID: \"1db82794-0c4f-4114-9b07-669edbf49db9\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-cf947b468-g775t" Sep 11 09:21:48.227285 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:21:48.227282 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/1db82794-0c4f-4114-9b07-669edbf49db9-online-tls\") pod \"feast-simple-feast-remote-setup-cf947b468-g775t\" (UID: \"1db82794-0c4f-4114-9b07-669edbf49db9\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-cf947b468-g775t" Sep 11 09:21:48.227443 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:21:48.227311 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/1db82794-0c4f-4114-9b07-669edbf49db9-ui-tls\") pod \"feast-simple-feast-remote-setup-cf947b468-g775t\" (UID: \"1db82794-0c4f-4114-9b07-669edbf49db9\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-cf947b468-g775t" Sep 11 09:21:48.490649 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:21:48.490540 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-cf947b468-g775t" Sep 11 09:21:48.770074 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:21:48.770050 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-cf947b468-g775t"] Sep 11 09:21:48.771320 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:21:48.771294 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1db82794_0c4f_4114_9b07_669edbf49db9.slice/crio-920dde175ce1494fc2fd29b7c33859626402da0925590658596fc8fc9d245e0a WatchSource:0}: Error finding container 920dde175ce1494fc2fd29b7c33859626402da0925590658596fc8fc9d245e0a: Status 404 returned error can't find the container with id 920dde175ce1494fc2fd29b7c33859626402da0925590658596fc8fc9d245e0a Sep 11 09:21:49.541426 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:21:49.541386 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-cf947b468-g775t" event={"ID":"1db82794-0c4f-4114-9b07-669edbf49db9","Type":"ContainerStarted","Data":"723a51c2014407ee093f4f296f27c660de4df83bd06eb8ad53afdc9c8f12cb39"} Sep 11 09:21:49.541426 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:21:49.541433 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-cf947b468-g775t" event={"ID":"1db82794-0c4f-4114-9b07-669edbf49db9","Type":"ContainerStarted","Data":"920dde175ce1494fc2fd29b7c33859626402da0925590658596fc8fc9d245e0a"} Sep 11 09:21:54.564344 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:21:54.564306 2575 generic.go:356] "Generic (PLEG): container finished" podID="1db82794-0c4f-4114-9b07-669edbf49db9" containerID="723a51c2014407ee093f4f296f27c660de4df83bd06eb8ad53afdc9c8f12cb39" exitCode=0 Sep 11 09:21:54.564849 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:21:54.564364 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-cf947b468-g775t" event={"ID":"1db82794-0c4f-4114-9b07-669edbf49db9","Type":"ContainerDied","Data":"723a51c2014407ee093f4f296f27c660de4df83bd06eb8ad53afdc9c8f12cb39"} Sep 11 09:21:55.571850 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:21:55.571810 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-cf947b468-g775t" event={"ID":"1db82794-0c4f-4114-9b07-669edbf49db9","Type":"ContainerStarted","Data":"26360a3a8092f3affa77e61a2f76c134204102ae34c7b3acaa8d29c55ee48746"} Sep 11 09:22:16.662802 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:22:16.662769 2575 generic.go:356] "Generic (PLEG): container finished" podID="1db82794-0c4f-4114-9b07-669edbf49db9" containerID="26360a3a8092f3affa77e61a2f76c134204102ae34c7b3acaa8d29c55ee48746" exitCode=0 Sep 11 09:22:16.663181 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:22:16.662837 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-cf947b468-g775t" event={"ID":"1db82794-0c4f-4114-9b07-669edbf49db9","Type":"ContainerDied","Data":"26360a3a8092f3affa77e61a2f76c134204102ae34c7b3acaa8d29c55ee48746"} Sep 11 09:22:16.666495 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:22:16.666459 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-intra-comm\"" Sep 11 09:22:17.672913 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:22:17.672868 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-cf947b468-g775t" event={"ID":"1db82794-0c4f-4114-9b07-669edbf49db9","Type":"ContainerStarted","Data":"ce2f485cb28f95e4a5588db4a4fd07ab52208bf3993ce2baa651f395e9da097a"} Sep 11 09:22:17.673409 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:22:17.672932 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-cf947b468-g775t" event={"ID":"1db82794-0c4f-4114-9b07-669edbf49db9","Type":"ContainerStarted","Data":"fc950f6e6a3a47b1c54b3c146ce954615418d0f0223850ada7129efb4735c982"} Sep 11 09:22:17.673409 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:22:17.672953 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-cf947b468-g775t" event={"ID":"1db82794-0c4f-4114-9b07-669edbf49db9","Type":"ContainerStarted","Data":"7f80665657744e131d783901a6db3abb1695f4fbbe448a51a6468f31064b528d"} Sep 11 09:22:17.699722 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:22:17.699651 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-cf947b468-g775t" podStartSLOduration=30.699627842 podStartE2EDuration="30.699627842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 09:21:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 09:22:17.697446075 +0000 UTC m=+554.038056281" watchObservedRunningTime="2026-09-11 09:22:17.699627842 +0000 UTC m=+554.040238049" Sep 11 09:22:18.491572 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:22:18.491463 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-cf947b468-g775t" Sep 11 09:22:18.491572 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:22:18.491556 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-cf947b468-g775t" Sep 11 09:22:18.491572 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:22:18.491576 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-cf947b468-g775t" Sep 11 09:22:18.491891 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:22:18.491590 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-cf947b468-g775t" Sep 11 09:22:18.491891 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:22:18.491605 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-cf947b468-g775t" Sep 11 09:22:18.491891 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:22:18.491620 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-cf947b468-g775t" Sep 11 09:22:18.493897 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:22:18.493852 2575 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-cf947b468-g775t" podUID="1db82794-0c4f-4114-9b07-669edbf49db9" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.20:8816: connect: connection refused" Sep 11 09:22:18.494037 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:22:18.493915 2575 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-cf947b468-g775t" podUID="1db82794-0c4f-4114-9b07-669edbf49db9" containerName="online" probeResult="failure" output="Get \"https://10.134.0.20:6567/health\": dial tcp 10.134.0.20:6567: connect: connection refused" Sep 11 09:22:18.494037 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:22:18.493852 2575 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-cf947b468-g775t" podUID="1db82794-0c4f-4114-9b07-669edbf49db9" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.20:8443: connect: connection refused" Sep 11 09:22:21.491567 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:22:21.491520 2575 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-cf947b468-g775t" podUID="1db82794-0c4f-4114-9b07-669edbf49db9" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.20:8816: connect: connection refused" Sep 11 09:22:21.492259 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:22:21.491523 2575 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-cf947b468-g775t" podUID="1db82794-0c4f-4114-9b07-669edbf49db9" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.20:8443: connect: connection refused" Sep 11 09:22:21.492397 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:22:21.491534 2575 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-cf947b468-g775t" podUID="1db82794-0c4f-4114-9b07-669edbf49db9" containerName="online" probeResult="failure" output="Get \"https://10.134.0.20:6567/health\": dial tcp 10.134.0.20:6567: connect: connection refused" Sep 11 09:22:24.491547 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:22:24.491493 2575 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-cf947b468-g775t" podUID="1db82794-0c4f-4114-9b07-669edbf49db9" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.20:8443: connect: connection refused" Sep 11 09:22:24.492113 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:22:24.492083 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-cf947b468-g775t" Sep 11 09:22:24.492956 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:22:24.492935 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-cf947b468-g775t" Sep 11 09:22:24.623670 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:22:24.623637 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-cf947b468-g775t" Sep 11 09:22:24.731151 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:22:24.731117 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-cf947b468-g775t" Sep 11 09:22:27.492199 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:22:27.492166 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-cf947b468-g775t" Sep 11 09:22:27.725525 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:22:27.725491 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-cf947b468-g775t" Sep 11 09:23:29.911319 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:23:29.911233 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-cf947b468-g775t"] Sep 11 09:23:29.911894 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:23:29.911460 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-cf947b468-g775t" podUID="1db82794-0c4f-4114-9b07-669edbf49db9" containerName="online" containerID="cri-o://7f80665657744e131d783901a6db3abb1695f4fbbe448a51a6468f31064b528d" gracePeriod=30 Sep 11 09:23:29.911894 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:23:29.911534 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-cf947b468-g775t" podUID="1db82794-0c4f-4114-9b07-669edbf49db9" containerName="ui" containerID="cri-o://ce2f485cb28f95e4a5588db4a4fd07ab52208bf3993ce2baa651f395e9da097a" gracePeriod=30 Sep 11 09:23:29.911894 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:23:29.911584 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-cf947b468-g775t" podUID="1db82794-0c4f-4114-9b07-669edbf49db9" containerName="offline" containerID="cri-o://fc950f6e6a3a47b1c54b3c146ce954615418d0f0223850ada7129efb4735c982" gracePeriod=30 Sep 11 09:23:30.227651 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:23:30.227558 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-95b9b66c7-9rd5b"] Sep 11 09:23:30.227819 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:23:30.227779 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-9rd5b" podUID="cfdada3e-9f75-4329-ad8f-0305a6d2e4d4" containerName="registry" containerID="cri-o://3e623297693ca54d3b8413a804e360d70d28b4a7bd9ebf48fd604075fe119e07" gracePeriod=30 Sep 11 09:23:30.227887 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:23:30.227832 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-9rd5b" podUID="cfdada3e-9f75-4329-ad8f-0305a6d2e4d4" containerName="ui" containerID="cri-o://959fa38a0f10c849bb857ec25d1395cb59c588beee59f3f1121660a99b57fe06" gracePeriod=30 Sep 11 09:23:30.227887 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:23:30.227857 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-9rd5b" podUID="cfdada3e-9f75-4329-ad8f-0305a6d2e4d4" containerName="online" containerID="cri-o://178c5c755beb76e04665f96da6ba5e071ca0f0e6d031ea0e598dd5c45d8aca66" gracePeriod=30 Sep 11 09:23:30.227991 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:23:30.227858 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-9rd5b" podUID="cfdada3e-9f75-4329-ad8f-0305a6d2e4d4" containerName="offline" containerID="cri-o://854357ae71636866704e60cb51cb7f7499655d7d7cd626e97742fde096a6ea52" gracePeriod=30 Sep 11 09:23:30.990395 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:23:30.990362 2575 generic.go:356] "Generic (PLEG): container finished" podID="1db82794-0c4f-4114-9b07-669edbf49db9" containerID="ce2f485cb28f95e4a5588db4a4fd07ab52208bf3993ce2baa651f395e9da097a" exitCode=0 Sep 11 09:23:30.990395 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:23:30.990389 2575 generic.go:356] "Generic (PLEG): container finished" podID="1db82794-0c4f-4114-9b07-669edbf49db9" containerID="7f80665657744e131d783901a6db3abb1695f4fbbe448a51a6468f31064b528d" exitCode=0 Sep 11 09:23:30.990908 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:23:30.990444 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-cf947b468-g775t" event={"ID":"1db82794-0c4f-4114-9b07-669edbf49db9","Type":"ContainerDied","Data":"ce2f485cb28f95e4a5588db4a4fd07ab52208bf3993ce2baa651f395e9da097a"} Sep 11 09:23:30.990908 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:23:30.990499 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-cf947b468-g775t" event={"ID":"1db82794-0c4f-4114-9b07-669edbf49db9","Type":"ContainerDied","Data":"7f80665657744e131d783901a6db3abb1695f4fbbe448a51a6468f31064b528d"} Sep 11 09:23:30.994205 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:23:30.994180 2575 generic.go:356] "Generic (PLEG): container finished" podID="cfdada3e-9f75-4329-ad8f-0305a6d2e4d4" containerID="959fa38a0f10c849bb857ec25d1395cb59c588beee59f3f1121660a99b57fe06" exitCode=0 Sep 11 09:23:30.994346 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:23:30.994214 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-9rd5b" event={"ID":"cfdada3e-9f75-4329-ad8f-0305a6d2e4d4","Type":"ContainerDied","Data":"959fa38a0f10c849bb857ec25d1395cb59c588beee59f3f1121660a99b57fe06"} Sep 11 09:23:32.004145 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:23:32.004117 2575 generic.go:356] "Generic (PLEG): container finished" podID="cfdada3e-9f75-4329-ad8f-0305a6d2e4d4" containerID="178c5c755beb76e04665f96da6ba5e071ca0f0e6d031ea0e598dd5c45d8aca66" exitCode=0 Sep 11 09:23:32.011090 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:23:32.004197 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-9rd5b" event={"ID":"cfdada3e-9f75-4329-ad8f-0305a6d2e4d4","Type":"ContainerDied","Data":"178c5c755beb76e04665f96da6ba5e071ca0f0e6d031ea0e598dd5c45d8aca66"} Sep 11 09:23:34.712586 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:23:34.712543 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-cf947b468-g775t" podUID="1db82794-0c4f-4114-9b07-669edbf49db9" containerName="online" probeResult="failure" output="Get \"https://10.134.0.20:6567/health\": dial tcp 10.134.0.20:6567: connect: connection refused" Sep 11 09:23:35.269554 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:23:35.269508 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-9rd5b" podUID="cfdada3e-9f75-4329-ad8f-0305a6d2e4d4" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.19:8443: connect: connection refused" Sep 11 09:23:35.269753 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:23:35.269569 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-9rd5b" podUID="cfdada3e-9f75-4329-ad8f-0305a6d2e4d4" containerName="online" probeResult="failure" output="Get \"https://10.134.0.19:6567/health\": dial tcp 10.134.0.19:6567: connect: connection refused" Sep 11 09:23:37.725771 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:23:37.725727 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-cf947b468-g775t" podUID="1db82794-0c4f-4114-9b07-669edbf49db9" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.20:8443: connect: connection refused" Sep 11 09:23:44.712346 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:23:44.712300 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-cf947b468-g775t" podUID="1db82794-0c4f-4114-9b07-669edbf49db9" containerName="online" probeResult="failure" output="Get \"https://10.134.0.20:6567/health\": dial tcp 10.134.0.20:6567: connect: connection refused" Sep 11 09:23:45.269607 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:23:45.269562 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-9rd5b" podUID="cfdada3e-9f75-4329-ad8f-0305a6d2e4d4" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.19:8443: connect: connection refused" Sep 11 09:23:45.269819 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:23:45.269631 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-9rd5b" podUID="cfdada3e-9f75-4329-ad8f-0305a6d2e4d4" containerName="online" probeResult="failure" output="Get \"https://10.134.0.19:6567/health\": dial tcp 10.134.0.19:6567: connect: connection refused" Sep 11 09:23:47.725861 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:23:47.725810 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-cf947b468-g775t" podUID="1db82794-0c4f-4114-9b07-669edbf49db9" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.20:8443: connect: connection refused" Sep 11 09:23:54.713094 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:23:54.713052 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-cf947b468-g775t" podUID="1db82794-0c4f-4114-9b07-669edbf49db9" containerName="online" probeResult="failure" output="Get \"https://10.134.0.20:6567/health\": dial tcp 10.134.0.20:6567: connect: connection refused" Sep 11 09:23:54.713524 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:23:54.713171 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-cf947b468-g775t" Sep 11 09:23:55.269799 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:23:55.269747 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-9rd5b" podUID="cfdada3e-9f75-4329-ad8f-0305a6d2e4d4" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.19:8443: connect: connection refused" Sep 11 09:23:55.269799 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:23:55.269779 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-9rd5b" podUID="cfdada3e-9f75-4329-ad8f-0305a6d2e4d4" containerName="online" probeResult="failure" output="Get \"https://10.134.0.19:6567/health\": dial tcp 10.134.0.19:6567: connect: connection refused" Sep 11 09:23:55.270080 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:23:55.269947 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-9rd5b" Sep 11 09:23:55.270080 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:23:55.270020 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-9rd5b" Sep 11 09:23:57.725692 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:23:57.725650 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-cf947b468-g775t" podUID="1db82794-0c4f-4114-9b07-669edbf49db9" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.20:8443: connect: connection refused" Sep 11 09:23:57.726095 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:23:57.725771 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-cf947b468-g775t" Sep 11 09:24:00.125229 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:00.125198 2575 generic.go:356] "Generic (PLEG): container finished" podID="1db82794-0c4f-4114-9b07-669edbf49db9" containerID="fc950f6e6a3a47b1c54b3c146ce954615418d0f0223850ada7129efb4735c982" exitCode=137 Sep 11 09:24:00.125637 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:00.125275 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-cf947b468-g775t" event={"ID":"1db82794-0c4f-4114-9b07-669edbf49db9","Type":"ContainerDied","Data":"fc950f6e6a3a47b1c54b3c146ce954615418d0f0223850ada7129efb4735c982"} Sep 11 09:24:00.562996 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:00.562973 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-cf947b468-g775t" Sep 11 09:24:00.640411 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:00.640381 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-984hq\" (UniqueName: \"kubernetes.io/projected/1db82794-0c4f-4114-9b07-669edbf49db9-kube-api-access-984hq\") pod \"1db82794-0c4f-4114-9b07-669edbf49db9\" (UID: \"1db82794-0c4f-4114-9b07-669edbf49db9\") " Sep 11 09:24:00.640411 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:00.640414 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/1db82794-0c4f-4114-9b07-669edbf49db9-ui-tls\") pod \"1db82794-0c4f-4114-9b07-669edbf49db9\" (UID: \"1db82794-0c4f-4114-9b07-669edbf49db9\") " Sep 11 09:24:00.640674 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:00.640448 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/1db82794-0c4f-4114-9b07-669edbf49db9-feast-data\") pod \"1db82794-0c4f-4114-9b07-669edbf49db9\" (UID: \"1db82794-0c4f-4114-9b07-669edbf49db9\") " Sep 11 09:24:00.640674 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:00.640495 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/1db82794-0c4f-4114-9b07-669edbf49db9-registry-tls\") pod \"1db82794-0c4f-4114-9b07-669edbf49db9\" (UID: \"1db82794-0c4f-4114-9b07-669edbf49db9\") " Sep 11 09:24:00.640782 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:00.640695 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/1db82794-0c4f-4114-9b07-669edbf49db9-offline-tls\") pod \"1db82794-0c4f-4114-9b07-669edbf49db9\" (UID: \"1db82794-0c4f-4114-9b07-669edbf49db9\") " Sep 11 09:24:00.640844 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:00.640805 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/1db82794-0c4f-4114-9b07-669edbf49db9-online-tls\") pod \"1db82794-0c4f-4114-9b07-669edbf49db9\" (UID: \"1db82794-0c4f-4114-9b07-669edbf49db9\") " Sep 11 09:24:00.640894 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:00.640838 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1db82794-0c4f-4114-9b07-669edbf49db9-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "1db82794-0c4f-4114-9b07-669edbf49db9" (UID: "1db82794-0c4f-4114-9b07-669edbf49db9"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 09:24:00.641035 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:00.641007 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1db82794-0c4f-4114-9b07-669edbf49db9-feast-data" (OuterVolumeSpecName: "feast-data") pod "1db82794-0c4f-4114-9b07-669edbf49db9" (UID: "1db82794-0c4f-4114-9b07-669edbf49db9"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 09:24:00.641164 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:00.641106 2575 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/1db82794-0c4f-4114-9b07-669edbf49db9-feast-data\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\"" Sep 11 09:24:00.641164 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:00.641127 2575 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/1db82794-0c4f-4114-9b07-669edbf49db9-registry-tls\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\"" Sep 11 09:24:00.642711 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:00.642689 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1db82794-0c4f-4114-9b07-669edbf49db9-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "1db82794-0c4f-4114-9b07-669edbf49db9" (UID: "1db82794-0c4f-4114-9b07-669edbf49db9"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 09:24:00.642798 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:00.642785 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1db82794-0c4f-4114-9b07-669edbf49db9-online-tls" (OuterVolumeSpecName: "online-tls") pod "1db82794-0c4f-4114-9b07-669edbf49db9" (UID: "1db82794-0c4f-4114-9b07-669edbf49db9"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 09:24:00.643017 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:00.642986 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1db82794-0c4f-4114-9b07-669edbf49db9-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "1db82794-0c4f-4114-9b07-669edbf49db9" (UID: "1db82794-0c4f-4114-9b07-669edbf49db9"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 09:24:00.643096 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:00.643016 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1db82794-0c4f-4114-9b07-669edbf49db9-kube-api-access-984hq" (OuterVolumeSpecName: "kube-api-access-984hq") pod "1db82794-0c4f-4114-9b07-669edbf49db9" (UID: "1db82794-0c4f-4114-9b07-669edbf49db9"). InnerVolumeSpecName "kube-api-access-984hq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 09:24:00.741719 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:00.741679 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-984hq\" (UniqueName: \"kubernetes.io/projected/1db82794-0c4f-4114-9b07-669edbf49db9-kube-api-access-984hq\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\"" Sep 11 09:24:00.741719 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:00.741710 2575 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/1db82794-0c4f-4114-9b07-669edbf49db9-ui-tls\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\"" Sep 11 09:24:00.741719 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:00.741720 2575 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/1db82794-0c4f-4114-9b07-669edbf49db9-offline-tls\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\"" Sep 11 09:24:00.741719 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:00.741729 2575 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/1db82794-0c4f-4114-9b07-669edbf49db9-online-tls\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\"" Sep 11 09:24:00.876362 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:00.876337 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-9rd5b" Sep 11 09:24:00.944119 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:00.944085 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/cfdada3e-9f75-4329-ad8f-0305a6d2e4d4-offline-tls\") pod \"cfdada3e-9f75-4329-ad8f-0305a6d2e4d4\" (UID: \"cfdada3e-9f75-4329-ad8f-0305a6d2e4d4\") " Sep 11 09:24:00.944302 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:00.944134 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gf9dk\" (UniqueName: \"kubernetes.io/projected/cfdada3e-9f75-4329-ad8f-0305a6d2e4d4-kube-api-access-gf9dk\") pod \"cfdada3e-9f75-4329-ad8f-0305a6d2e4d4\" (UID: \"cfdada3e-9f75-4329-ad8f-0305a6d2e4d4\") " Sep 11 09:24:00.944302 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:00.944151 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/cfdada3e-9f75-4329-ad8f-0305a6d2e4d4-ui-tls\") pod \"cfdada3e-9f75-4329-ad8f-0305a6d2e4d4\" (UID: \"cfdada3e-9f75-4329-ad8f-0305a6d2e4d4\") " Sep 11 09:24:00.944302 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:00.944203 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/cfdada3e-9f75-4329-ad8f-0305a6d2e4d4-registry-tls\") pod \"cfdada3e-9f75-4329-ad8f-0305a6d2e4d4\" (UID: \"cfdada3e-9f75-4329-ad8f-0305a6d2e4d4\") " Sep 11 09:24:00.944302 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:00.944251 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/cfdada3e-9f75-4329-ad8f-0305a6d2e4d4-online-tls\") pod \"cfdada3e-9f75-4329-ad8f-0305a6d2e4d4\" (UID: \"cfdada3e-9f75-4329-ad8f-0305a6d2e4d4\") " Sep 11 09:24:00.944302 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:00.944273 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/cfdada3e-9f75-4329-ad8f-0305a6d2e4d4-feast-data\") pod \"cfdada3e-9f75-4329-ad8f-0305a6d2e4d4\" (UID: \"cfdada3e-9f75-4329-ad8f-0305a6d2e4d4\") " Sep 11 09:24:00.944903 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:00.944857 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfdada3e-9f75-4329-ad8f-0305a6d2e4d4-feast-data" (OuterVolumeSpecName: "feast-data") pod "cfdada3e-9f75-4329-ad8f-0305a6d2e4d4" (UID: "cfdada3e-9f75-4329-ad8f-0305a6d2e4d4"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 09:24:00.946262 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:00.946228 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfdada3e-9f75-4329-ad8f-0305a6d2e4d4-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "cfdada3e-9f75-4329-ad8f-0305a6d2e4d4" (UID: "cfdada3e-9f75-4329-ad8f-0305a6d2e4d4"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 09:24:00.946819 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:00.946792 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfdada3e-9f75-4329-ad8f-0305a6d2e4d4-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "cfdada3e-9f75-4329-ad8f-0305a6d2e4d4" (UID: "cfdada3e-9f75-4329-ad8f-0305a6d2e4d4"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 09:24:00.946902 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:00.946820 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfdada3e-9f75-4329-ad8f-0305a6d2e4d4-kube-api-access-gf9dk" (OuterVolumeSpecName: "kube-api-access-gf9dk") pod "cfdada3e-9f75-4329-ad8f-0305a6d2e4d4" (UID: "cfdada3e-9f75-4329-ad8f-0305a6d2e4d4"). InnerVolumeSpecName "kube-api-access-gf9dk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 09:24:00.946902 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:00.946860 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfdada3e-9f75-4329-ad8f-0305a6d2e4d4-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "cfdada3e-9f75-4329-ad8f-0305a6d2e4d4" (UID: "cfdada3e-9f75-4329-ad8f-0305a6d2e4d4"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 09:24:00.946971 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:00.946903 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfdada3e-9f75-4329-ad8f-0305a6d2e4d4-online-tls" (OuterVolumeSpecName: "online-tls") pod "cfdada3e-9f75-4329-ad8f-0305a6d2e4d4" (UID: "cfdada3e-9f75-4329-ad8f-0305a6d2e4d4"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 09:24:01.045266 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:01.045175 2575 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/cfdada3e-9f75-4329-ad8f-0305a6d2e4d4-registry-tls\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\"" Sep 11 09:24:01.045266 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:01.045207 2575 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/cfdada3e-9f75-4329-ad8f-0305a6d2e4d4-online-tls\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\"" Sep 11 09:24:01.045266 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:01.045216 2575 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/cfdada3e-9f75-4329-ad8f-0305a6d2e4d4-feast-data\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\"" Sep 11 09:24:01.045266 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:01.045223 2575 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/cfdada3e-9f75-4329-ad8f-0305a6d2e4d4-offline-tls\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\"" Sep 11 09:24:01.045266 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:01.045233 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gf9dk\" (UniqueName: \"kubernetes.io/projected/cfdada3e-9f75-4329-ad8f-0305a6d2e4d4-kube-api-access-gf9dk\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\"" Sep 11 09:24:01.045266 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:01.045243 2575 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/cfdada3e-9f75-4329-ad8f-0305a6d2e4d4-ui-tls\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\"" Sep 11 09:24:01.131842 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:01.131803 2575 generic.go:356] "Generic (PLEG): container finished" podID="cfdada3e-9f75-4329-ad8f-0305a6d2e4d4" containerID="854357ae71636866704e60cb51cb7f7499655d7d7cd626e97742fde096a6ea52" exitCode=137 Sep 11 09:24:01.131842 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:01.131833 2575 generic.go:356] "Generic (PLEG): container finished" podID="cfdada3e-9f75-4329-ad8f-0305a6d2e4d4" containerID="3e623297693ca54d3b8413a804e360d70d28b4a7bd9ebf48fd604075fe119e07" exitCode=137 Sep 11 09:24:01.132346 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:01.131896 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-9rd5b" Sep 11 09:24:01.132346 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:01.131897 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-9rd5b" event={"ID":"cfdada3e-9f75-4329-ad8f-0305a6d2e4d4","Type":"ContainerDied","Data":"854357ae71636866704e60cb51cb7f7499655d7d7cd626e97742fde096a6ea52"} Sep 11 09:24:01.132346 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:01.131957 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-9rd5b" event={"ID":"cfdada3e-9f75-4329-ad8f-0305a6d2e4d4","Type":"ContainerDied","Data":"3e623297693ca54d3b8413a804e360d70d28b4a7bd9ebf48fd604075fe119e07"} Sep 11 09:24:01.132346 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:01.131980 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-95b9b66c7-9rd5b" event={"ID":"cfdada3e-9f75-4329-ad8f-0305a6d2e4d4","Type":"ContainerDied","Data":"944fd836887977e96df8830b9aa07d0b55da8c9ac6893c57c51c20ed6de23a18"} Sep 11 09:24:01.132346 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:01.132009 2575 scope.go:117] "RemoveContainer" containerID="959fa38a0f10c849bb857ec25d1395cb59c588beee59f3f1121660a99b57fe06" Sep 11 09:24:01.134813 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:01.134789 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-cf947b468-g775t" event={"ID":"1db82794-0c4f-4114-9b07-669edbf49db9","Type":"ContainerDied","Data":"920dde175ce1494fc2fd29b7c33859626402da0925590658596fc8fc9d245e0a"} Sep 11 09:24:01.134907 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:01.134835 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-cf947b468-g775t" Sep 11 09:24:01.141354 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:01.141178 2575 scope.go:117] "RemoveContainer" containerID="854357ae71636866704e60cb51cb7f7499655d7d7cd626e97742fde096a6ea52" Sep 11 09:24:01.149360 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:01.149340 2575 scope.go:117] "RemoveContainer" containerID="178c5c755beb76e04665f96da6ba5e071ca0f0e6d031ea0e598dd5c45d8aca66" Sep 11 09:24:01.157000 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:01.156979 2575 scope.go:117] "RemoveContainer" containerID="3e623297693ca54d3b8413a804e360d70d28b4a7bd9ebf48fd604075fe119e07" Sep 11 09:24:01.163693 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:01.163671 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-95b9b66c7-9rd5b"] Sep 11 09:24:01.164797 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:01.164781 2575 scope.go:117] "RemoveContainer" containerID="4d7333e770079a9f2945f9a13dd14af095a0818f1e6db59cc24cfec9ffec6487" Sep 11 09:24:01.171814 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:01.171737 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-95b9b66c7-9rd5b"] Sep 11 09:24:01.175831 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:01.175812 2575 scope.go:117] "RemoveContainer" containerID="85e3cb1b856aa3be504822bfe22f58a15df5dc0e2d75b4d19288b260fd593e08" Sep 11 09:24:01.183765 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:01.183746 2575 scope.go:117] "RemoveContainer" containerID="959fa38a0f10c849bb857ec25d1395cb59c588beee59f3f1121660a99b57fe06" Sep 11 09:24:01.184103 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:24:01.184081 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"959fa38a0f10c849bb857ec25d1395cb59c588beee59f3f1121660a99b57fe06\": container with ID starting with 959fa38a0f10c849bb857ec25d1395cb59c588beee59f3f1121660a99b57fe06 not found: ID does not exist" containerID="959fa38a0f10c849bb857ec25d1395cb59c588beee59f3f1121660a99b57fe06" Sep 11 09:24:01.184150 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:01.184112 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"959fa38a0f10c849bb857ec25d1395cb59c588beee59f3f1121660a99b57fe06"} err="failed to get container status \"959fa38a0f10c849bb857ec25d1395cb59c588beee59f3f1121660a99b57fe06\": rpc error: code = NotFound desc = could not find container \"959fa38a0f10c849bb857ec25d1395cb59c588beee59f3f1121660a99b57fe06\": container with ID starting with 959fa38a0f10c849bb857ec25d1395cb59c588beee59f3f1121660a99b57fe06 not found: ID does not exist" Sep 11 09:24:01.184150 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:01.184147 2575 scope.go:117] "RemoveContainer" containerID="854357ae71636866704e60cb51cb7f7499655d7d7cd626e97742fde096a6ea52" Sep 11 09:24:01.184419 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:24:01.184394 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"854357ae71636866704e60cb51cb7f7499655d7d7cd626e97742fde096a6ea52\": container with ID starting with 854357ae71636866704e60cb51cb7f7499655d7d7cd626e97742fde096a6ea52 not found: ID does not exist" containerID="854357ae71636866704e60cb51cb7f7499655d7d7cd626e97742fde096a6ea52" Sep 11 09:24:01.184458 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:01.184429 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"854357ae71636866704e60cb51cb7f7499655d7d7cd626e97742fde096a6ea52"} err="failed to get container status \"854357ae71636866704e60cb51cb7f7499655d7d7cd626e97742fde096a6ea52\": rpc error: code = NotFound desc = could not find container \"854357ae71636866704e60cb51cb7f7499655d7d7cd626e97742fde096a6ea52\": container with ID starting with 854357ae71636866704e60cb51cb7f7499655d7d7cd626e97742fde096a6ea52 not found: ID does not exist" Sep 11 09:24:01.184458 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:01.184450 2575 scope.go:117] "RemoveContainer" containerID="178c5c755beb76e04665f96da6ba5e071ca0f0e6d031ea0e598dd5c45d8aca66" Sep 11 09:24:01.184636 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:01.184617 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-cf947b468-g775t"] Sep 11 09:24:01.184819 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:24:01.184804 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"178c5c755beb76e04665f96da6ba5e071ca0f0e6d031ea0e598dd5c45d8aca66\": container with ID starting with 178c5c755beb76e04665f96da6ba5e071ca0f0e6d031ea0e598dd5c45d8aca66 not found: ID does not exist" containerID="178c5c755beb76e04665f96da6ba5e071ca0f0e6d031ea0e598dd5c45d8aca66" Sep 11 09:24:01.184859 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:01.184823 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"178c5c755beb76e04665f96da6ba5e071ca0f0e6d031ea0e598dd5c45d8aca66"} err="failed to get container status \"178c5c755beb76e04665f96da6ba5e071ca0f0e6d031ea0e598dd5c45d8aca66\": rpc error: code = NotFound desc = could not find container \"178c5c755beb76e04665f96da6ba5e071ca0f0e6d031ea0e598dd5c45d8aca66\": container with ID starting with 178c5c755beb76e04665f96da6ba5e071ca0f0e6d031ea0e598dd5c45d8aca66 not found: ID does not exist" Sep 11 09:24:01.184859 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:01.184837 2575 scope.go:117] "RemoveContainer" containerID="3e623297693ca54d3b8413a804e360d70d28b4a7bd9ebf48fd604075fe119e07" Sep 11 09:24:01.185077 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:24:01.185059 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e623297693ca54d3b8413a804e360d70d28b4a7bd9ebf48fd604075fe119e07\": container with ID starting with 3e623297693ca54d3b8413a804e360d70d28b4a7bd9ebf48fd604075fe119e07 not found: ID does not exist" containerID="3e623297693ca54d3b8413a804e360d70d28b4a7bd9ebf48fd604075fe119e07" Sep 11 09:24:01.185122 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:01.185084 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e623297693ca54d3b8413a804e360d70d28b4a7bd9ebf48fd604075fe119e07"} err="failed to get container status \"3e623297693ca54d3b8413a804e360d70d28b4a7bd9ebf48fd604075fe119e07\": rpc error: code = NotFound desc = could not find container \"3e623297693ca54d3b8413a804e360d70d28b4a7bd9ebf48fd604075fe119e07\": container with ID starting with 3e623297693ca54d3b8413a804e360d70d28b4a7bd9ebf48fd604075fe119e07 not found: ID does not exist" Sep 11 09:24:01.185122 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:01.185100 2575 scope.go:117] "RemoveContainer" containerID="4d7333e770079a9f2945f9a13dd14af095a0818f1e6db59cc24cfec9ffec6487" Sep 11 09:24:01.185312 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:24:01.185298 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d7333e770079a9f2945f9a13dd14af095a0818f1e6db59cc24cfec9ffec6487\": container with ID starting with 4d7333e770079a9f2945f9a13dd14af095a0818f1e6db59cc24cfec9ffec6487 not found: ID does not exist" containerID="4d7333e770079a9f2945f9a13dd14af095a0818f1e6db59cc24cfec9ffec6487" Sep 11 09:24:01.185365 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:01.185314 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d7333e770079a9f2945f9a13dd14af095a0818f1e6db59cc24cfec9ffec6487"} err="failed to get container status \"4d7333e770079a9f2945f9a13dd14af095a0818f1e6db59cc24cfec9ffec6487\": rpc error: code = NotFound desc = could not find container \"4d7333e770079a9f2945f9a13dd14af095a0818f1e6db59cc24cfec9ffec6487\": container with ID starting with 4d7333e770079a9f2945f9a13dd14af095a0818f1e6db59cc24cfec9ffec6487 not found: ID does not exist" Sep 11 09:24:01.185365 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:01.185326 2575 scope.go:117] "RemoveContainer" containerID="85e3cb1b856aa3be504822bfe22f58a15df5dc0e2d75b4d19288b260fd593e08" Sep 11 09:24:01.185588 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:24:01.185572 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85e3cb1b856aa3be504822bfe22f58a15df5dc0e2d75b4d19288b260fd593e08\": container with ID starting with 85e3cb1b856aa3be504822bfe22f58a15df5dc0e2d75b4d19288b260fd593e08 not found: ID does not exist" containerID="85e3cb1b856aa3be504822bfe22f58a15df5dc0e2d75b4d19288b260fd593e08" Sep 11 09:24:01.185638 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:01.185593 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85e3cb1b856aa3be504822bfe22f58a15df5dc0e2d75b4d19288b260fd593e08"} err="failed to get container status \"85e3cb1b856aa3be504822bfe22f58a15df5dc0e2d75b4d19288b260fd593e08\": rpc error: code = NotFound desc = could not find container \"85e3cb1b856aa3be504822bfe22f58a15df5dc0e2d75b4d19288b260fd593e08\": container with ID starting with 85e3cb1b856aa3be504822bfe22f58a15df5dc0e2d75b4d19288b260fd593e08 not found: ID does not exist" Sep 11 09:24:01.185638 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:01.185606 2575 scope.go:117] "RemoveContainer" containerID="959fa38a0f10c849bb857ec25d1395cb59c588beee59f3f1121660a99b57fe06" Sep 11 09:24:01.185840 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:01.185822 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"959fa38a0f10c849bb857ec25d1395cb59c588beee59f3f1121660a99b57fe06"} err="failed to get container status \"959fa38a0f10c849bb857ec25d1395cb59c588beee59f3f1121660a99b57fe06\": rpc error: code = NotFound desc = could not find container \"959fa38a0f10c849bb857ec25d1395cb59c588beee59f3f1121660a99b57fe06\": container with ID starting with 959fa38a0f10c849bb857ec25d1395cb59c588beee59f3f1121660a99b57fe06 not found: ID does not exist" Sep 11 09:24:01.185903 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:01.185843 2575 scope.go:117] "RemoveContainer" containerID="854357ae71636866704e60cb51cb7f7499655d7d7cd626e97742fde096a6ea52" Sep 11 09:24:01.186219 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:01.186192 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"854357ae71636866704e60cb51cb7f7499655d7d7cd626e97742fde096a6ea52"} err="failed to get container status \"854357ae71636866704e60cb51cb7f7499655d7d7cd626e97742fde096a6ea52\": rpc error: code = NotFound desc = could not find container \"854357ae71636866704e60cb51cb7f7499655d7d7cd626e97742fde096a6ea52\": container with ID starting with 854357ae71636866704e60cb51cb7f7499655d7d7cd626e97742fde096a6ea52 not found: ID does not exist" Sep 11 09:24:01.186363 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:01.186341 2575 scope.go:117] "RemoveContainer" containerID="178c5c755beb76e04665f96da6ba5e071ca0f0e6d031ea0e598dd5c45d8aca66" Sep 11 09:24:01.186768 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:01.186734 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"178c5c755beb76e04665f96da6ba5e071ca0f0e6d031ea0e598dd5c45d8aca66"} err="failed to get container status \"178c5c755beb76e04665f96da6ba5e071ca0f0e6d031ea0e598dd5c45d8aca66\": rpc error: code = NotFound desc = could not find container \"178c5c755beb76e04665f96da6ba5e071ca0f0e6d031ea0e598dd5c45d8aca66\": container with ID starting with 178c5c755beb76e04665f96da6ba5e071ca0f0e6d031ea0e598dd5c45d8aca66 not found: ID does not exist" Sep 11 09:24:01.186768 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:01.186759 2575 scope.go:117] "RemoveContainer" containerID="3e623297693ca54d3b8413a804e360d70d28b4a7bd9ebf48fd604075fe119e07" Sep 11 09:24:01.187180 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:01.187079 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e623297693ca54d3b8413a804e360d70d28b4a7bd9ebf48fd604075fe119e07"} err="failed to get container status \"3e623297693ca54d3b8413a804e360d70d28b4a7bd9ebf48fd604075fe119e07\": rpc error: code = NotFound desc = could not find container \"3e623297693ca54d3b8413a804e360d70d28b4a7bd9ebf48fd604075fe119e07\": container with ID starting with 3e623297693ca54d3b8413a804e360d70d28b4a7bd9ebf48fd604075fe119e07 not found: ID does not exist" Sep 11 09:24:01.187180 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:01.187102 2575 scope.go:117] "RemoveContainer" containerID="4d7333e770079a9f2945f9a13dd14af095a0818f1e6db59cc24cfec9ffec6487" Sep 11 09:24:01.187393 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:01.187372 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d7333e770079a9f2945f9a13dd14af095a0818f1e6db59cc24cfec9ffec6487"} err="failed to get container status \"4d7333e770079a9f2945f9a13dd14af095a0818f1e6db59cc24cfec9ffec6487\": rpc error: code = NotFound desc = could not find container \"4d7333e770079a9f2945f9a13dd14af095a0818f1e6db59cc24cfec9ffec6487\": container with ID starting with 4d7333e770079a9f2945f9a13dd14af095a0818f1e6db59cc24cfec9ffec6487 not found: ID does not exist" Sep 11 09:24:01.187486 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:01.187395 2575 scope.go:117] "RemoveContainer" containerID="85e3cb1b856aa3be504822bfe22f58a15df5dc0e2d75b4d19288b260fd593e08" Sep 11 09:24:01.187673 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:01.187654 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85e3cb1b856aa3be504822bfe22f58a15df5dc0e2d75b4d19288b260fd593e08"} err="failed to get container status \"85e3cb1b856aa3be504822bfe22f58a15df5dc0e2d75b4d19288b260fd593e08\": rpc error: code = NotFound desc = could not find container \"85e3cb1b856aa3be504822bfe22f58a15df5dc0e2d75b4d19288b260fd593e08\": container with ID starting with 85e3cb1b856aa3be504822bfe22f58a15df5dc0e2d75b4d19288b260fd593e08 not found: ID does not exist" Sep 11 09:24:01.187727 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:01.187674 2575 scope.go:117] "RemoveContainer" containerID="ce2f485cb28f95e4a5588db4a4fd07ab52208bf3993ce2baa651f395e9da097a" Sep 11 09:24:01.188335 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:01.188319 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-cf947b468-g775t"] Sep 11 09:24:01.194976 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:01.194959 2575 scope.go:117] "RemoveContainer" containerID="fc950f6e6a3a47b1c54b3c146ce954615418d0f0223850ada7129efb4735c982" Sep 11 09:24:01.202691 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:01.202675 2575 scope.go:117] "RemoveContainer" containerID="7f80665657744e131d783901a6db3abb1695f4fbbe448a51a6468f31064b528d" Sep 11 09:24:01.210284 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:01.210266 2575 scope.go:117] "RemoveContainer" containerID="26360a3a8092f3affa77e61a2f76c134204102ae34c7b3acaa8d29c55ee48746" Sep 11 09:24:01.217457 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:01.217441 2575 scope.go:117] "RemoveContainer" containerID="723a51c2014407ee093f4f296f27c660de4df83bd06eb8ad53afdc9c8f12cb39" Sep 11 09:24:02.114254 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:02.114212 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1db82794-0c4f-4114-9b07-669edbf49db9" path="/var/lib/kubelet/pods/1db82794-0c4f-4114-9b07-669edbf49db9/volumes" Sep 11 09:24:02.114881 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:02.114860 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cfdada3e-9f75-4329-ad8f-0305a6d2e4d4" path="/var/lib/kubelet/pods/cfdada3e-9f75-4329-ad8f-0305a6d2e4d4/volumes" Sep 11 09:24:09.463790 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:09.463760 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/postgres-864846d4f5-72t7s"] Sep 11 09:24:09.464170 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:09.464071 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1db82794-0c4f-4114-9b07-669edbf49db9" containerName="feast-init" Sep 11 09:24:09.464170 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:09.464082 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="1db82794-0c4f-4114-9b07-669edbf49db9" containerName="feast-init" Sep 11 09:24:09.464170 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:09.464092 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfdada3e-9f75-4329-ad8f-0305a6d2e4d4" containerName="ui" Sep 11 09:24:09.464170 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:09.464098 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfdada3e-9f75-4329-ad8f-0305a6d2e4d4" containerName="ui" Sep 11 09:24:09.464170 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:09.464108 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfdada3e-9f75-4329-ad8f-0305a6d2e4d4" containerName="feast-init" Sep 11 09:24:09.464170 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:09.464114 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfdada3e-9f75-4329-ad8f-0305a6d2e4d4" containerName="feast-init" Sep 11 09:24:09.464170 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:09.464121 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfdada3e-9f75-4329-ad8f-0305a6d2e4d4" containerName="registry" Sep 11 09:24:09.464170 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:09.464126 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfdada3e-9f75-4329-ad8f-0305a6d2e4d4" containerName="registry" Sep 11 09:24:09.464170 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:09.464134 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1db82794-0c4f-4114-9b07-669edbf49db9" containerName="feast-apply" Sep 11 09:24:09.464170 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:09.464139 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="1db82794-0c4f-4114-9b07-669edbf49db9" containerName="feast-apply" Sep 11 09:24:09.464170 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:09.464145 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfdada3e-9f75-4329-ad8f-0305a6d2e4d4" containerName="online" Sep 11 09:24:09.464170 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:09.464150 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfdada3e-9f75-4329-ad8f-0305a6d2e4d4" containerName="online" Sep 11 09:24:09.464170 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:09.464156 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1db82794-0c4f-4114-9b07-669edbf49db9" containerName="online" Sep 11 09:24:09.464170 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:09.464161 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="1db82794-0c4f-4114-9b07-669edbf49db9" containerName="online" Sep 11 09:24:09.464170 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:09.464167 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1db82794-0c4f-4114-9b07-669edbf49db9" containerName="ui" Sep 11 09:24:09.464170 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:09.464172 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="1db82794-0c4f-4114-9b07-669edbf49db9" containerName="ui" Sep 11 09:24:09.464714 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:09.464183 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfdada3e-9f75-4329-ad8f-0305a6d2e4d4" containerName="offline" Sep 11 09:24:09.464714 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:09.464188 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfdada3e-9f75-4329-ad8f-0305a6d2e4d4" containerName="offline" Sep 11 09:24:09.464714 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:09.464199 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1db82794-0c4f-4114-9b07-669edbf49db9" containerName="offline" Sep 11 09:24:09.464714 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:09.464204 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="1db82794-0c4f-4114-9b07-669edbf49db9" containerName="offline" Sep 11 09:24:09.464714 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:09.464209 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfdada3e-9f75-4329-ad8f-0305a6d2e4d4" containerName="feast-apply" Sep 11 09:24:09.464714 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:09.464214 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfdada3e-9f75-4329-ad8f-0305a6d2e4d4" containerName="feast-apply" Sep 11 09:24:09.464714 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:09.464264 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="1db82794-0c4f-4114-9b07-669edbf49db9" containerName="offline" Sep 11 09:24:09.464714 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:09.464271 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="cfdada3e-9f75-4329-ad8f-0305a6d2e4d4" containerName="registry" Sep 11 09:24:09.464714 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:09.464277 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="1db82794-0c4f-4114-9b07-669edbf49db9" containerName="ui" Sep 11 09:24:09.464714 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:09.464282 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="cfdada3e-9f75-4329-ad8f-0305a6d2e4d4" containerName="offline" Sep 11 09:24:09.464714 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:09.464286 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="cfdada3e-9f75-4329-ad8f-0305a6d2e4d4" containerName="ui" Sep 11 09:24:09.464714 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:09.464293 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="cfdada3e-9f75-4329-ad8f-0305a6d2e4d4" containerName="online" Sep 11 09:24:09.464714 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:09.464298 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="1db82794-0c4f-4114-9b07-669edbf49db9" containerName="online" Sep 11 09:24:09.469622 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:09.469599 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-864846d4f5-72t7s" Sep 11 09:24:09.472096 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:09.472072 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"postgres-secret\"" Sep 11 09:24:09.472229 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:09.472186 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\"" Sep 11 09:24:09.473147 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:09.473126 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\"" Sep 11 09:24:09.473147 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:09.473138 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"default-dockercfg-sdcjx\"" Sep 11 09:24:09.475662 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:09.475641 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/postgres-864846d4f5-72t7s"] Sep 11 09:24:09.499115 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:09.499087 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/redis-69bbd798b6-rcs47"] Sep 11 09:24:09.502929 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:09.502910 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/redis-69bbd798b6-rcs47" Sep 11 09:24:09.509768 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:09.509745 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/redis-69bbd798b6-rcs47"] Sep 11 09:24:09.520976 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:09.520814 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55sn8\" (UniqueName: \"kubernetes.io/projected/db2ecbb0-ce0c-436a-99dc-e5c0ee4832b7-kube-api-access-55sn8\") pod \"postgres-864846d4f5-72t7s\" (UID: \"db2ecbb0-ce0c-436a-99dc-e5c0ee4832b7\") " pod="test-ns-feast/postgres-864846d4f5-72t7s" Sep 11 09:24:09.520976 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:09.520944 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/db2ecbb0-ce0c-436a-99dc-e5c0ee4832b7-postgresdata\") pod \"postgres-864846d4f5-72t7s\" (UID: \"db2ecbb0-ce0c-436a-99dc-e5c0ee4832b7\") " pod="test-ns-feast/postgres-864846d4f5-72t7s" Sep 11 09:24:09.621702 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:09.621658 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55sn8\" (UniqueName: \"kubernetes.io/projected/db2ecbb0-ce0c-436a-99dc-e5c0ee4832b7-kube-api-access-55sn8\") pod \"postgres-864846d4f5-72t7s\" (UID: \"db2ecbb0-ce0c-436a-99dc-e5c0ee4832b7\") " pod="test-ns-feast/postgres-864846d4f5-72t7s" Sep 11 09:24:09.621879 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:09.621714 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7skxl\" (UniqueName: \"kubernetes.io/projected/a62e8ddb-4603-4797-9e16-63c7b28c2e1a-kube-api-access-7skxl\") pod \"redis-69bbd798b6-rcs47\" (UID: \"a62e8ddb-4603-4797-9e16-63c7b28c2e1a\") " pod="test-ns-feast/redis-69bbd798b6-rcs47" Sep 11 09:24:09.621879 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:09.621765 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/db2ecbb0-ce0c-436a-99dc-e5c0ee4832b7-postgresdata\") pod \"postgres-864846d4f5-72t7s\" (UID: \"db2ecbb0-ce0c-436a-99dc-e5c0ee4832b7\") " pod="test-ns-feast/postgres-864846d4f5-72t7s" Sep 11 09:24:09.622098 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:09.622083 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/db2ecbb0-ce0c-436a-99dc-e5c0ee4832b7-postgresdata\") pod \"postgres-864846d4f5-72t7s\" (UID: \"db2ecbb0-ce0c-436a-99dc-e5c0ee4832b7\") " pod="test-ns-feast/postgres-864846d4f5-72t7s" Sep 11 09:24:09.630174 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:09.630148 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-55sn8\" (UniqueName: \"kubernetes.io/projected/db2ecbb0-ce0c-436a-99dc-e5c0ee4832b7-kube-api-access-55sn8\") pod \"postgres-864846d4f5-72t7s\" (UID: \"db2ecbb0-ce0c-436a-99dc-e5c0ee4832b7\") " pod="test-ns-feast/postgres-864846d4f5-72t7s" Sep 11 09:24:09.722526 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:09.722403 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7skxl\" (UniqueName: \"kubernetes.io/projected/a62e8ddb-4603-4797-9e16-63c7b28c2e1a-kube-api-access-7skxl\") pod \"redis-69bbd798b6-rcs47\" (UID: \"a62e8ddb-4603-4797-9e16-63c7b28c2e1a\") " pod="test-ns-feast/redis-69bbd798b6-rcs47" Sep 11 09:24:09.731995 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:09.731976 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7skxl\" (UniqueName: \"kubernetes.io/projected/a62e8ddb-4603-4797-9e16-63c7b28c2e1a-kube-api-access-7skxl\") pod \"redis-69bbd798b6-rcs47\" (UID: \"a62e8ddb-4603-4797-9e16-63c7b28c2e1a\") " pod="test-ns-feast/redis-69bbd798b6-rcs47" Sep 11 09:24:09.789581 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:09.789540 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-864846d4f5-72t7s" Sep 11 09:24:09.813191 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:09.813164 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/redis-69bbd798b6-rcs47" Sep 11 09:24:10.064156 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:10.064115 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/postgres-864846d4f5-72t7s"] Sep 11 09:24:10.067832 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:24:10.067811 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddb2ecbb0_ce0c_436a_99dc_e5c0ee4832b7.slice/crio-3334d98ca2bf738a45f0a963cf55f1092422d91a213dcb6238bc84870e0b4f67 WatchSource:0}: Error finding container 3334d98ca2bf738a45f0a963cf55f1092422d91a213dcb6238bc84870e0b4f67: Status 404 returned error can't find the container with id 3334d98ca2bf738a45f0a963cf55f1092422d91a213dcb6238bc84870e0b4f67 Sep 11 09:24:10.069655 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:10.069638 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 09:24:10.094045 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:10.094024 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/redis-69bbd798b6-rcs47"] Sep 11 09:24:10.095935 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:24:10.095912 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda62e8ddb_4603_4797_9e16_63c7b28c2e1a.slice/crio-d2aacc95d1de870f5a9bb22283faa49e9e8f6496046eeaab5a36763635659400 WatchSource:0}: Error finding container d2aacc95d1de870f5a9bb22283faa49e9e8f6496046eeaab5a36763635659400: Status 404 returned error can't find the container with id d2aacc95d1de870f5a9bb22283faa49e9e8f6496046eeaab5a36763635659400 Sep 11 09:24:10.174735 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:10.174697 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/redis-69bbd798b6-rcs47" event={"ID":"a62e8ddb-4603-4797-9e16-63c7b28c2e1a","Type":"ContainerStarted","Data":"d2aacc95d1de870f5a9bb22283faa49e9e8f6496046eeaab5a36763635659400"} Sep 11 09:24:10.175755 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:10.175729 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-864846d4f5-72t7s" event={"ID":"db2ecbb0-ce0c-436a-99dc-e5c0ee4832b7","Type":"ContainerStarted","Data":"3334d98ca2bf738a45f0a963cf55f1092422d91a213dcb6238bc84870e0b4f67"} Sep 11 09:24:12.187334 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:12.187243 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/redis-69bbd798b6-rcs47" event={"ID":"a62e8ddb-4603-4797-9e16-63c7b28c2e1a","Type":"ContainerStarted","Data":"42b886227ca57c15d07c6e467d89a1c4cb14d1a8a3b91e2838fb97e70512bf77"} Sep 11 09:24:12.202513 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:12.202380 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/redis-69bbd798b6-rcs47" podStartSLOduration=1.480795815 podStartE2EDuration="3.202358139s" totalImagesPullingTime="1.721562324s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 09:24:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 09:24:12.201957516 +0000 UTC m=+668.542567723" watchObservedRunningTime="2026-09-11 09:24:12.202358139 +0000 UTC m=+668.542968346" Sep 11 09:24:15.202047 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:15.202006 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-864846d4f5-72t7s" event={"ID":"db2ecbb0-ce0c-436a-99dc-e5c0ee4832b7","Type":"ContainerStarted","Data":"07d5e8aa0ad50d3e7d6352aae615513f012751c7bf252b8e5396f8a72b619f41"} Sep 11 09:24:15.218244 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:15.218195 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/postgres-864846d4f5-72t7s" podStartSLOduration=1.9580282420000001 podStartE2EDuration="6.218180349s" totalImagesPullingTime="4.260152107s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 09:24:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 09:24:15.217126933 +0000 UTC m=+671.557737140" watchObservedRunningTime="2026-09-11 09:24:15.218180349 +0000 UTC m=+671.558790555" Sep 11 09:24:18.283072 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:18.282978 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-credit-scoring-5c864dc4f-7nhwc"] Sep 11 09:24:18.290028 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:18.289997 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-5c864dc4f-7nhwc" Sep 11 09:24:18.292832 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:18.292807 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-intra-comm\"" Sep 11 09:24:18.292954 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:18.292849 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-registry-tls\"" Sep 11 09:24:18.293102 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:18.293083 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-online-tls\"" Sep 11 09:24:18.293827 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:18.293810 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-dockercfg-8z7nq\"" Sep 11 09:24:18.293970 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:18.293954 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-offline-tls\"" Sep 11 09:24:18.302788 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:18.302767 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-credit-scoring-5c864dc4f-7nhwc"] Sep 11 09:24:18.405346 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:18.405311 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmqcz\" (UniqueName: \"kubernetes.io/projected/8098283c-6486-45f3-a766-27ae54e3ae36-kube-api-access-wmqcz\") pod \"feast-credit-scoring-5c864dc4f-7nhwc\" (UID: \"8098283c-6486-45f3-a766-27ae54e3ae36\") " pod="test-ns-feast/feast-credit-scoring-5c864dc4f-7nhwc" Sep 11 09:24:18.405346 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:18.405351 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/8098283c-6486-45f3-a766-27ae54e3ae36-offline-tls\") pod \"feast-credit-scoring-5c864dc4f-7nhwc\" (UID: \"8098283c-6486-45f3-a766-27ae54e3ae36\") " pod="test-ns-feast/feast-credit-scoring-5c864dc4f-7nhwc" Sep 11 09:24:18.405597 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:18.405371 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/8098283c-6486-45f3-a766-27ae54e3ae36-feast-data\") pod \"feast-credit-scoring-5c864dc4f-7nhwc\" (UID: \"8098283c-6486-45f3-a766-27ae54e3ae36\") " pod="test-ns-feast/feast-credit-scoring-5c864dc4f-7nhwc" Sep 11 09:24:18.405597 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:18.405490 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/8098283c-6486-45f3-a766-27ae54e3ae36-registry-tls\") pod \"feast-credit-scoring-5c864dc4f-7nhwc\" (UID: \"8098283c-6486-45f3-a766-27ae54e3ae36\") " pod="test-ns-feast/feast-credit-scoring-5c864dc4f-7nhwc" Sep 11 09:24:18.405597 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:18.405537 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/8098283c-6486-45f3-a766-27ae54e3ae36-online-tls\") pod \"feast-credit-scoring-5c864dc4f-7nhwc\" (UID: \"8098283c-6486-45f3-a766-27ae54e3ae36\") " pod="test-ns-feast/feast-credit-scoring-5c864dc4f-7nhwc" Sep 11 09:24:18.506648 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:18.506607 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/8098283c-6486-45f3-a766-27ae54e3ae36-feast-data\") pod \"feast-credit-scoring-5c864dc4f-7nhwc\" (UID: \"8098283c-6486-45f3-a766-27ae54e3ae36\") " pod="test-ns-feast/feast-credit-scoring-5c864dc4f-7nhwc" Sep 11 09:24:18.506810 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:18.506674 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/8098283c-6486-45f3-a766-27ae54e3ae36-registry-tls\") pod \"feast-credit-scoring-5c864dc4f-7nhwc\" (UID: \"8098283c-6486-45f3-a766-27ae54e3ae36\") " pod="test-ns-feast/feast-credit-scoring-5c864dc4f-7nhwc" Sep 11 09:24:18.506810 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:18.506707 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/8098283c-6486-45f3-a766-27ae54e3ae36-online-tls\") pod \"feast-credit-scoring-5c864dc4f-7nhwc\" (UID: \"8098283c-6486-45f3-a766-27ae54e3ae36\") " pod="test-ns-feast/feast-credit-scoring-5c864dc4f-7nhwc" Sep 11 09:24:18.506810 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:18.506750 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmqcz\" (UniqueName: \"kubernetes.io/projected/8098283c-6486-45f3-a766-27ae54e3ae36-kube-api-access-wmqcz\") pod \"feast-credit-scoring-5c864dc4f-7nhwc\" (UID: \"8098283c-6486-45f3-a766-27ae54e3ae36\") " pod="test-ns-feast/feast-credit-scoring-5c864dc4f-7nhwc" Sep 11 09:24:18.506810 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:18.506774 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/8098283c-6486-45f3-a766-27ae54e3ae36-offline-tls\") pod \"feast-credit-scoring-5c864dc4f-7nhwc\" (UID: \"8098283c-6486-45f3-a766-27ae54e3ae36\") " pod="test-ns-feast/feast-credit-scoring-5c864dc4f-7nhwc" Sep 11 09:24:18.507048 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:24:18.506882 2575 secret.go:189] Couldn't get secret test-ns-feast/feast-credit-scoring-online-tls: secret "feast-credit-scoring-online-tls" not found Sep 11 09:24:18.507048 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:24:18.506961 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8098283c-6486-45f3-a766-27ae54e3ae36-online-tls podName:8098283c-6486-45f3-a766-27ae54e3ae36 nodeName:}" failed. No retries permitted until 2026-09-11 09:24:19.006939339 +0000 UTC m=+675.347549540 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/8098283c-6486-45f3-a766-27ae54e3ae36-online-tls") pod "feast-credit-scoring-5c864dc4f-7nhwc" (UID: "8098283c-6486-45f3-a766-27ae54e3ae36") : secret "feast-credit-scoring-online-tls" not found Sep 11 09:24:18.507150 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:18.507101 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/8098283c-6486-45f3-a766-27ae54e3ae36-feast-data\") pod \"feast-credit-scoring-5c864dc4f-7nhwc\" (UID: \"8098283c-6486-45f3-a766-27ae54e3ae36\") " pod="test-ns-feast/feast-credit-scoring-5c864dc4f-7nhwc" Sep 11 09:24:18.509591 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:18.509564 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/8098283c-6486-45f3-a766-27ae54e3ae36-offline-tls\") pod \"feast-credit-scoring-5c864dc4f-7nhwc\" (UID: \"8098283c-6486-45f3-a766-27ae54e3ae36\") " pod="test-ns-feast/feast-credit-scoring-5c864dc4f-7nhwc" Sep 11 09:24:18.509687 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:18.509611 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/8098283c-6486-45f3-a766-27ae54e3ae36-registry-tls\") pod \"feast-credit-scoring-5c864dc4f-7nhwc\" (UID: \"8098283c-6486-45f3-a766-27ae54e3ae36\") " pod="test-ns-feast/feast-credit-scoring-5c864dc4f-7nhwc" Sep 11 09:24:18.517216 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:18.517195 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmqcz\" (UniqueName: \"kubernetes.io/projected/8098283c-6486-45f3-a766-27ae54e3ae36-kube-api-access-wmqcz\") pod \"feast-credit-scoring-5c864dc4f-7nhwc\" (UID: \"8098283c-6486-45f3-a766-27ae54e3ae36\") " pod="test-ns-feast/feast-credit-scoring-5c864dc4f-7nhwc" Sep 11 09:24:19.010905 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:19.010871 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/8098283c-6486-45f3-a766-27ae54e3ae36-online-tls\") pod \"feast-credit-scoring-5c864dc4f-7nhwc\" (UID: \"8098283c-6486-45f3-a766-27ae54e3ae36\") " pod="test-ns-feast/feast-credit-scoring-5c864dc4f-7nhwc" Sep 11 09:24:19.013303 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:19.013276 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/8098283c-6486-45f3-a766-27ae54e3ae36-online-tls\") pod \"feast-credit-scoring-5c864dc4f-7nhwc\" (UID: \"8098283c-6486-45f3-a766-27ae54e3ae36\") " pod="test-ns-feast/feast-credit-scoring-5c864dc4f-7nhwc" Sep 11 09:24:19.199684 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:19.199647 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-5c864dc4f-7nhwc" Sep 11 09:24:19.477213 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:19.477186 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-credit-scoring-5c864dc4f-7nhwc"] Sep 11 09:24:19.479075 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:24:19.479048 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8098283c_6486_45f3_a766_27ae54e3ae36.slice/crio-ae1b90d655f4c3158a5818efbf86cf513415825badfa06185d09be355e57d9bc WatchSource:0}: Error finding container ae1b90d655f4c3158a5818efbf86cf513415825badfa06185d09be355e57d9bc: Status 404 returned error can't find the container with id ae1b90d655f4c3158a5818efbf86cf513415825badfa06185d09be355e57d9bc Sep 11 09:24:20.223506 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:20.223454 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-5c864dc4f-7nhwc" event={"ID":"8098283c-6486-45f3-a766-27ae54e3ae36","Type":"ContainerStarted","Data":"0832a70da175d150e96cf4babc430455d58b1745017c69a1257f6f6e7129b322"} Sep 11 09:24:20.223661 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:20.223528 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-5c864dc4f-7nhwc" event={"ID":"8098283c-6486-45f3-a766-27ae54e3ae36","Type":"ContainerStarted","Data":"ae1b90d655f4c3158a5818efbf86cf513415825badfa06185d09be355e57d9bc"} Sep 11 09:24:21.229085 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:21.229047 2575 generic.go:356] "Generic (PLEG): container finished" podID="8098283c-6486-45f3-a766-27ae54e3ae36" containerID="0832a70da175d150e96cf4babc430455d58b1745017c69a1257f6f6e7129b322" exitCode=0 Sep 11 09:24:21.229487 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:21.229097 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-5c864dc4f-7nhwc" event={"ID":"8098283c-6486-45f3-a766-27ae54e3ae36","Type":"ContainerDied","Data":"0832a70da175d150e96cf4babc430455d58b1745017c69a1257f6f6e7129b322"} Sep 11 09:24:22.236429 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:22.236392 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-5c864dc4f-7nhwc" event={"ID":"8098283c-6486-45f3-a766-27ae54e3ae36","Type":"ContainerStarted","Data":"9f46fe803101effb338407425dc2aeaf0e21e145466d466a65db954663e91b1f"} Sep 11 09:24:28.264046 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:28.263997 2575 generic.go:356] "Generic (PLEG): container finished" podID="8098283c-6486-45f3-a766-27ae54e3ae36" containerID="9f46fe803101effb338407425dc2aeaf0e21e145466d466a65db954663e91b1f" exitCode=0 Sep 11 09:24:28.264557 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:28.264071 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-5c864dc4f-7nhwc" event={"ID":"8098283c-6486-45f3-a766-27ae54e3ae36","Type":"ContainerDied","Data":"9f46fe803101effb338407425dc2aeaf0e21e145466d466a65db954663e91b1f"} Sep 11 09:24:29.273103 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:29.273053 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-5c864dc4f-7nhwc" event={"ID":"8098283c-6486-45f3-a766-27ae54e3ae36","Type":"ContainerStarted","Data":"e949087b99d02dbda0ff02fa35ca5bd095ae4241c74f7292899bc8948db70017"} Sep 11 09:24:29.273514 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:29.273134 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-5c864dc4f-7nhwc" event={"ID":"8098283c-6486-45f3-a766-27ae54e3ae36","Type":"ContainerStarted","Data":"88e513f30dbfa75d8ddaeb5203e69f7a5b60d22db39a52937fcb4a8ab5db50d1"} Sep 11 09:24:29.273514 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:29.273161 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-5c864dc4f-7nhwc" event={"ID":"8098283c-6486-45f3-a766-27ae54e3ae36","Type":"ContainerStarted","Data":"7c70c6475a763eb517ad2d816315d116143ad3d72355d547261a39e8ca5f7d6a"} Sep 11 09:24:29.297282 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:29.297219 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-credit-scoring-5c864dc4f-7nhwc" podStartSLOduration=11.297194262 podStartE2EDuration="11.297194262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 09:24:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 09:24:29.295857687 +0000 UTC m=+685.636467894" watchObservedRunningTime="2026-09-11 09:24:29.297194262 +0000 UTC m=+685.637804469" Sep 11 09:24:31.200677 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:31.200637 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-credit-scoring-5c864dc4f-7nhwc" Sep 11 09:24:31.201316 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:31.201281 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-credit-scoring-5c864dc4f-7nhwc" Sep 11 09:24:31.201496 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:31.201323 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-credit-scoring-5c864dc4f-7nhwc" Sep 11 09:24:31.202563 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:31.202528 2575 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-5c864dc4f-7nhwc" podUID="8098283c-6486-45f3-a766-27ae54e3ae36" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.23:6571: connect: connection refused" Sep 11 09:24:31.202563 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:31.202553 2575 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-5c864dc4f-7nhwc" podUID="8098283c-6486-45f3-a766-27ae54e3ae36" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.23:8816: connect: connection refused" Sep 11 09:24:31.202756 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:31.202639 2575 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-5c864dc4f-7nhwc" podUID="8098283c-6486-45f3-a766-27ae54e3ae36" containerName="online" probeResult="failure" output="Get \"https://10.134.0.23:6567/health\": dial tcp 10.134.0.23:6567: connect: connection refused" Sep 11 09:24:34.200315 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:34.200275 2575 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-5c864dc4f-7nhwc" podUID="8098283c-6486-45f3-a766-27ae54e3ae36" containerName="online" probeResult="failure" output="Get \"https://10.134.0.23:6567/health\": dial tcp 10.134.0.23:6567: connect: connection refused" Sep 11 09:24:34.200887 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:34.200276 2575 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-5c864dc4f-7nhwc" podUID="8098283c-6486-45f3-a766-27ae54e3ae36" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.23:6571: connect: connection refused" Sep 11 09:24:34.200887 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:34.200284 2575 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-5c864dc4f-7nhwc" podUID="8098283c-6486-45f3-a766-27ae54e3ae36" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.23:8816: connect: connection refused" Sep 11 09:24:37.200677 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:37.200624 2575 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-5c864dc4f-7nhwc" podUID="8098283c-6486-45f3-a766-27ae54e3ae36" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.23:6571: connect: connection refused" Sep 11 09:24:37.201657 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:37.201632 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-credit-scoring-5c864dc4f-7nhwc" Sep 11 09:24:37.202167 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:37.202137 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-5c864dc4f-7nhwc" Sep 11 09:24:37.202644 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:37.202624 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-credit-scoring-5c864dc4f-7nhwc" Sep 11 09:24:37.207186 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:37.207166 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-credit-scoring-5c864dc4f-7nhwc" Sep 11 09:24:37.319613 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:37.319577 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-5c864dc4f-7nhwc" Sep 11 09:24:37.322813 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:37.322789 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-credit-scoring-5c864dc4f-7nhwc" Sep 11 09:24:39.200713 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:39.200680 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-5c864dc4f-7nhwc" Sep 11 09:24:40.200190 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:40.200147 2575 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-5c864dc4f-7nhwc" podUID="8098283c-6486-45f3-a766-27ae54e3ae36" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.23:6571: connect: connection refused" Sep 11 09:24:43.200746 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:43.200715 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-credit-scoring-5c864dc4f-7nhwc" Sep 11 09:24:43.201502 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:24:43.201483 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-credit-scoring-5c864dc4f-7nhwc" Sep 11 09:25:47.543574 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:25:47.543536 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-credit-scoring-5c864dc4f-7nhwc"] Sep 11 09:25:47.544158 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:25:47.543809 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-credit-scoring-5c864dc4f-7nhwc" podUID="8098283c-6486-45f3-a766-27ae54e3ae36" containerName="registry" containerID="cri-o://7c70c6475a763eb517ad2d816315d116143ad3d72355d547261a39e8ca5f7d6a" gracePeriod=30 Sep 11 09:25:47.544158 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:25:47.543875 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-credit-scoring-5c864dc4f-7nhwc" podUID="8098283c-6486-45f3-a766-27ae54e3ae36" containerName="offline" containerID="cri-o://e949087b99d02dbda0ff02fa35ca5bd095ae4241c74f7292899bc8948db70017" gracePeriod=30 Sep 11 09:25:47.544158 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:25:47.543875 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-credit-scoring-5c864dc4f-7nhwc" podUID="8098283c-6486-45f3-a766-27ae54e3ae36" containerName="online" containerID="cri-o://88e513f30dbfa75d8ddaeb5203e69f7a5b60d22db39a52937fcb4a8ab5db50d1" gracePeriod=30 Sep 11 09:25:47.559718 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:25:47.559688 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/postgres-864846d4f5-72t7s"] Sep 11 09:25:47.559864 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:25:47.559846 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/postgres-864846d4f5-72t7s" podUID="db2ecbb0-ce0c-436a-99dc-e5c0ee4832b7" containerName="postgres" containerID="cri-o://07d5e8aa0ad50d3e7d6352aae615513f012751c7bf252b8e5396f8a72b619f41" gracePeriod=30 Sep 11 09:25:47.564595 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:25:47.564560 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/redis-69bbd798b6-rcs47"] Sep 11 09:25:47.564826 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:25:47.564750 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/redis-69bbd798b6-rcs47" podUID="a62e8ddb-4603-4797-9e16-63c7b28c2e1a" containerName="redis" containerID="cri-o://42b886227ca57c15d07c6e467d89a1c4cb14d1a8a3b91e2838fb97e70512bf77" gracePeriod=30 Sep 11 09:25:47.814415 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:25:47.814388 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-864846d4f5-72t7s" Sep 11 09:25:47.817581 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:25:47.817561 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/redis-69bbd798b6-rcs47" Sep 11 09:25:47.866753 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:25:47.866716 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-55sn8\" (UniqueName: \"kubernetes.io/projected/db2ecbb0-ce0c-436a-99dc-e5c0ee4832b7-kube-api-access-55sn8\") pod \"db2ecbb0-ce0c-436a-99dc-e5c0ee4832b7\" (UID: \"db2ecbb0-ce0c-436a-99dc-e5c0ee4832b7\") " Sep 11 09:25:47.866928 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:25:47.866781 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/db2ecbb0-ce0c-436a-99dc-e5c0ee4832b7-postgresdata\") pod \"db2ecbb0-ce0c-436a-99dc-e5c0ee4832b7\" (UID: \"db2ecbb0-ce0c-436a-99dc-e5c0ee4832b7\") " Sep 11 09:25:47.866928 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:25:47.866905 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7skxl\" (UniqueName: \"kubernetes.io/projected/a62e8ddb-4603-4797-9e16-63c7b28c2e1a-kube-api-access-7skxl\") pod \"a62e8ddb-4603-4797-9e16-63c7b28c2e1a\" (UID: \"a62e8ddb-4603-4797-9e16-63c7b28c2e1a\") " Sep 11 09:25:47.869328 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:25:47.869277 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db2ecbb0-ce0c-436a-99dc-e5c0ee4832b7-kube-api-access-55sn8" (OuterVolumeSpecName: "kube-api-access-55sn8") pod "db2ecbb0-ce0c-436a-99dc-e5c0ee4832b7" (UID: "db2ecbb0-ce0c-436a-99dc-e5c0ee4832b7"). InnerVolumeSpecName "kube-api-access-55sn8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 09:25:47.869813 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:25:47.869782 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a62e8ddb-4603-4797-9e16-63c7b28c2e1a-kube-api-access-7skxl" (OuterVolumeSpecName: "kube-api-access-7skxl") pod "a62e8ddb-4603-4797-9e16-63c7b28c2e1a" (UID: "a62e8ddb-4603-4797-9e16-63c7b28c2e1a"). InnerVolumeSpecName "kube-api-access-7skxl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 09:25:47.896377 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:25:47.896339 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db2ecbb0-ce0c-436a-99dc-e5c0ee4832b7-postgresdata" (OuterVolumeSpecName: "postgresdata") pod "db2ecbb0-ce0c-436a-99dc-e5c0ee4832b7" (UID: "db2ecbb0-ce0c-436a-99dc-e5c0ee4832b7"). InnerVolumeSpecName "postgresdata". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 09:25:47.967870 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:25:47.967828 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55sn8\" (UniqueName: \"kubernetes.io/projected/db2ecbb0-ce0c-436a-99dc-e5c0ee4832b7-kube-api-access-55sn8\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\"" Sep 11 09:25:47.967870 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:25:47.967873 2575 reconciler_common.go:299] "Volume detached for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/db2ecbb0-ce0c-436a-99dc-e5c0ee4832b7-postgresdata\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\"" Sep 11 09:25:47.968086 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:25:47.967888 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7skxl\" (UniqueName: \"kubernetes.io/projected/a62e8ddb-4603-4797-9e16-63c7b28c2e1a-kube-api-access-7skxl\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\"" Sep 11 09:25:48.623009 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:25:48.622977 2575 generic.go:356] "Generic (PLEG): container finished" podID="a62e8ddb-4603-4797-9e16-63c7b28c2e1a" containerID="42b886227ca57c15d07c6e467d89a1c4cb14d1a8a3b91e2838fb97e70512bf77" exitCode=0 Sep 11 09:25:48.623407 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:25:48.623037 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/redis-69bbd798b6-rcs47" Sep 11 09:25:48.623407 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:25:48.623063 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/redis-69bbd798b6-rcs47" event={"ID":"a62e8ddb-4603-4797-9e16-63c7b28c2e1a","Type":"ContainerDied","Data":"42b886227ca57c15d07c6e467d89a1c4cb14d1a8a3b91e2838fb97e70512bf77"} Sep 11 09:25:48.623407 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:25:48.623102 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/redis-69bbd798b6-rcs47" event={"ID":"a62e8ddb-4603-4797-9e16-63c7b28c2e1a","Type":"ContainerDied","Data":"d2aacc95d1de870f5a9bb22283faa49e9e8f6496046eeaab5a36763635659400"} Sep 11 09:25:48.623407 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:25:48.623124 2575 scope.go:117] "RemoveContainer" containerID="42b886227ca57c15d07c6e467d89a1c4cb14d1a8a3b91e2838fb97e70512bf77" Sep 11 09:25:48.624425 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:25:48.624403 2575 generic.go:356] "Generic (PLEG): container finished" podID="db2ecbb0-ce0c-436a-99dc-e5c0ee4832b7" containerID="07d5e8aa0ad50d3e7d6352aae615513f012751c7bf252b8e5396f8a72b619f41" exitCode=0 Sep 11 09:25:48.624557 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:25:48.624489 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-864846d4f5-72t7s" Sep 11 09:25:48.624557 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:25:48.624509 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-864846d4f5-72t7s" event={"ID":"db2ecbb0-ce0c-436a-99dc-e5c0ee4832b7","Type":"ContainerDied","Data":"07d5e8aa0ad50d3e7d6352aae615513f012751c7bf252b8e5396f8a72b619f41"} Sep 11 09:25:48.624557 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:25:48.624543 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-864846d4f5-72t7s" event={"ID":"db2ecbb0-ce0c-436a-99dc-e5c0ee4832b7","Type":"ContainerDied","Data":"3334d98ca2bf738a45f0a963cf55f1092422d91a213dcb6238bc84870e0b4f67"} Sep 11 09:25:48.628023 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:25:48.628000 2575 generic.go:356] "Generic (PLEG): container finished" podID="8098283c-6486-45f3-a766-27ae54e3ae36" containerID="88e513f30dbfa75d8ddaeb5203e69f7a5b60d22db39a52937fcb4a8ab5db50d1" exitCode=0 Sep 11 09:25:48.628123 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:25:48.628026 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-5c864dc4f-7nhwc" event={"ID":"8098283c-6486-45f3-a766-27ae54e3ae36","Type":"ContainerDied","Data":"88e513f30dbfa75d8ddaeb5203e69f7a5b60d22db39a52937fcb4a8ab5db50d1"} Sep 11 09:25:48.632060 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:25:48.632041 2575 scope.go:117] "RemoveContainer" containerID="42b886227ca57c15d07c6e467d89a1c4cb14d1a8a3b91e2838fb97e70512bf77" Sep 11 09:25:48.632310 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:25:48.632294 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42b886227ca57c15d07c6e467d89a1c4cb14d1a8a3b91e2838fb97e70512bf77\": container with ID starting with 42b886227ca57c15d07c6e467d89a1c4cb14d1a8a3b91e2838fb97e70512bf77 not found: ID does not exist" containerID="42b886227ca57c15d07c6e467d89a1c4cb14d1a8a3b91e2838fb97e70512bf77" Sep 11 09:25:48.632357 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:25:48.632318 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42b886227ca57c15d07c6e467d89a1c4cb14d1a8a3b91e2838fb97e70512bf77"} err="failed to get container status \"42b886227ca57c15d07c6e467d89a1c4cb14d1a8a3b91e2838fb97e70512bf77\": rpc error: code = NotFound desc = could not find container \"42b886227ca57c15d07c6e467d89a1c4cb14d1a8a3b91e2838fb97e70512bf77\": container with ID starting with 42b886227ca57c15d07c6e467d89a1c4cb14d1a8a3b91e2838fb97e70512bf77 not found: ID does not exist" Sep 11 09:25:48.632357 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:25:48.632333 2575 scope.go:117] "RemoveContainer" containerID="07d5e8aa0ad50d3e7d6352aae615513f012751c7bf252b8e5396f8a72b619f41" Sep 11 09:25:48.639576 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:25:48.639558 2575 scope.go:117] "RemoveContainer" containerID="07d5e8aa0ad50d3e7d6352aae615513f012751c7bf252b8e5396f8a72b619f41" Sep 11 09:25:48.639874 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:25:48.639847 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07d5e8aa0ad50d3e7d6352aae615513f012751c7bf252b8e5396f8a72b619f41\": container with ID starting with 07d5e8aa0ad50d3e7d6352aae615513f012751c7bf252b8e5396f8a72b619f41 not found: ID does not exist" containerID="07d5e8aa0ad50d3e7d6352aae615513f012751c7bf252b8e5396f8a72b619f41" Sep 11 09:25:48.639966 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:25:48.639883 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07d5e8aa0ad50d3e7d6352aae615513f012751c7bf252b8e5396f8a72b619f41"} err="failed to get container status \"07d5e8aa0ad50d3e7d6352aae615513f012751c7bf252b8e5396f8a72b619f41\": rpc error: code = NotFound desc = could not find container \"07d5e8aa0ad50d3e7d6352aae615513f012751c7bf252b8e5396f8a72b619f41\": container with ID starting with 07d5e8aa0ad50d3e7d6352aae615513f012751c7bf252b8e5396f8a72b619f41 not found: ID does not exist" Sep 11 09:25:48.642369 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:25:48.642352 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/postgres-864846d4f5-72t7s"] Sep 11 09:25:48.643827 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:25:48.643808 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/postgres-864846d4f5-72t7s"] Sep 11 09:25:48.654848 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:25:48.654827 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/redis-69bbd798b6-rcs47"] Sep 11 09:25:48.660320 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:25:48.660296 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/redis-69bbd798b6-rcs47"] Sep 11 09:25:50.114449 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:25:50.114413 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a62e8ddb-4603-4797-9e16-63c7b28c2e1a" path="/var/lib/kubelet/pods/a62e8ddb-4603-4797-9e16-63c7b28c2e1a/volumes" Sep 11 09:25:50.114965 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:25:50.114901 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="db2ecbb0-ce0c-436a-99dc-e5c0ee4832b7" path="/var/lib/kubelet/pods/db2ecbb0-ce0c-436a-99dc-e5c0ee4832b7/volumes" Sep 11 09:25:57.319688 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:25:57.319646 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-5c864dc4f-7nhwc" podUID="8098283c-6486-45f3-a766-27ae54e3ae36" containerName="online" probeResult="failure" output="Get \"https://10.134.0.23:6567/health\": dial tcp 10.134.0.23:6567: connect: connection refused" Sep 11 09:26:07.320147 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:26:07.320098 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-5c864dc4f-7nhwc" podUID="8098283c-6486-45f3-a766-27ae54e3ae36" containerName="online" probeResult="failure" output="Get \"https://10.134.0.23:6567/health\": dial tcp 10.134.0.23:6567: connect: connection refused" Sep 11 09:26:17.320424 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:26:17.320376 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-5c864dc4f-7nhwc" podUID="8098283c-6486-45f3-a766-27ae54e3ae36" containerName="online" probeResult="failure" output="Get \"https://10.134.0.23:6567/health\": dial tcp 10.134.0.23:6567: connect: connection refused" Sep 11 09:26:17.320832 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:26:17.320523 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-5c864dc4f-7nhwc" Sep 11 09:26:17.752246 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:26:17.752217 2575 generic.go:356] "Generic (PLEG): container finished" podID="8098283c-6486-45f3-a766-27ae54e3ae36" containerID="e949087b99d02dbda0ff02fa35ca5bd095ae4241c74f7292899bc8948db70017" exitCode=137 Sep 11 09:26:17.752246 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:26:17.752239 2575 generic.go:356] "Generic (PLEG): container finished" podID="8098283c-6486-45f3-a766-27ae54e3ae36" containerID="7c70c6475a763eb517ad2d816315d116143ad3d72355d547261a39e8ca5f7d6a" exitCode=137 Sep 11 09:26:17.752439 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:26:17.752278 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-5c864dc4f-7nhwc" event={"ID":"8098283c-6486-45f3-a766-27ae54e3ae36","Type":"ContainerDied","Data":"e949087b99d02dbda0ff02fa35ca5bd095ae4241c74f7292899bc8948db70017"} Sep 11 09:26:17.752439 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:26:17.752322 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-5c864dc4f-7nhwc" event={"ID":"8098283c-6486-45f3-a766-27ae54e3ae36","Type":"ContainerDied","Data":"7c70c6475a763eb517ad2d816315d116143ad3d72355d547261a39e8ca5f7d6a"} Sep 11 09:26:18.187917 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:26:18.187885 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-5c864dc4f-7nhwc" Sep 11 09:26:18.341010 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:26:18.340914 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/8098283c-6486-45f3-a766-27ae54e3ae36-offline-tls\") pod \"8098283c-6486-45f3-a766-27ae54e3ae36\" (UID: \"8098283c-6486-45f3-a766-27ae54e3ae36\") " Sep 11 09:26:18.341010 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:26:18.340969 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wmqcz\" (UniqueName: \"kubernetes.io/projected/8098283c-6486-45f3-a766-27ae54e3ae36-kube-api-access-wmqcz\") pod \"8098283c-6486-45f3-a766-27ae54e3ae36\" (UID: \"8098283c-6486-45f3-a766-27ae54e3ae36\") " Sep 11 09:26:18.341575 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:26:18.341098 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/8098283c-6486-45f3-a766-27ae54e3ae36-registry-tls\") pod \"8098283c-6486-45f3-a766-27ae54e3ae36\" (UID: \"8098283c-6486-45f3-a766-27ae54e3ae36\") " Sep 11 09:26:18.341575 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:26:18.341174 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/8098283c-6486-45f3-a766-27ae54e3ae36-feast-data\") pod \"8098283c-6486-45f3-a766-27ae54e3ae36\" (UID: \"8098283c-6486-45f3-a766-27ae54e3ae36\") " Sep 11 09:26:18.341575 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:26:18.341219 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/8098283c-6486-45f3-a766-27ae54e3ae36-online-tls\") pod \"8098283c-6486-45f3-a766-27ae54e3ae36\" (UID: \"8098283c-6486-45f3-a766-27ae54e3ae36\") " Sep 11 09:26:18.343282 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:26:18.343253 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8098283c-6486-45f3-a766-27ae54e3ae36-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "8098283c-6486-45f3-a766-27ae54e3ae36" (UID: "8098283c-6486-45f3-a766-27ae54e3ae36"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 09:26:18.343395 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:26:18.343276 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8098283c-6486-45f3-a766-27ae54e3ae36-online-tls" (OuterVolumeSpecName: "online-tls") pod "8098283c-6486-45f3-a766-27ae54e3ae36" (UID: "8098283c-6486-45f3-a766-27ae54e3ae36"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 09:26:18.343395 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:26:18.343323 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8098283c-6486-45f3-a766-27ae54e3ae36-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "8098283c-6486-45f3-a766-27ae54e3ae36" (UID: "8098283c-6486-45f3-a766-27ae54e3ae36"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 09:26:18.343492 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:26:18.343416 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8098283c-6486-45f3-a766-27ae54e3ae36-kube-api-access-wmqcz" (OuterVolumeSpecName: "kube-api-access-wmqcz") pod "8098283c-6486-45f3-a766-27ae54e3ae36" (UID: "8098283c-6486-45f3-a766-27ae54e3ae36"). InnerVolumeSpecName "kube-api-access-wmqcz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 09:26:18.349216 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:26:18.349192 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8098283c-6486-45f3-a766-27ae54e3ae36-feast-data" (OuterVolumeSpecName: "feast-data") pod "8098283c-6486-45f3-a766-27ae54e3ae36" (UID: "8098283c-6486-45f3-a766-27ae54e3ae36"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 09:26:18.442848 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:26:18.442810 2575 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/8098283c-6486-45f3-a766-27ae54e3ae36-registry-tls\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\"" Sep 11 09:26:18.442848 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:26:18.442839 2575 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/8098283c-6486-45f3-a766-27ae54e3ae36-feast-data\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\"" Sep 11 09:26:18.442848 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:26:18.442849 2575 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/8098283c-6486-45f3-a766-27ae54e3ae36-online-tls\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\"" Sep 11 09:26:18.442848 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:26:18.442857 2575 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/8098283c-6486-45f3-a766-27ae54e3ae36-offline-tls\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\"" Sep 11 09:26:18.443109 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:26:18.442867 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmqcz\" (UniqueName: \"kubernetes.io/projected/8098283c-6486-45f3-a766-27ae54e3ae36-kube-api-access-wmqcz\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\"" Sep 11 09:26:18.758339 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:26:18.758295 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-5c864dc4f-7nhwc" event={"ID":"8098283c-6486-45f3-a766-27ae54e3ae36","Type":"ContainerDied","Data":"ae1b90d655f4c3158a5818efbf86cf513415825badfa06185d09be355e57d9bc"} Sep 11 09:26:18.758339 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:26:18.758331 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-5c864dc4f-7nhwc" Sep 11 09:26:18.758658 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:26:18.758372 2575 scope.go:117] "RemoveContainer" containerID="e949087b99d02dbda0ff02fa35ca5bd095ae4241c74f7292899bc8948db70017" Sep 11 09:26:18.767366 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:26:18.767350 2575 scope.go:117] "RemoveContainer" containerID="88e513f30dbfa75d8ddaeb5203e69f7a5b60d22db39a52937fcb4a8ab5db50d1" Sep 11 09:26:18.774910 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:26:18.774893 2575 scope.go:117] "RemoveContainer" containerID="7c70c6475a763eb517ad2d816315d116143ad3d72355d547261a39e8ca5f7d6a" Sep 11 09:26:18.782882 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:26:18.782860 2575 scope.go:117] "RemoveContainer" containerID="9f46fe803101effb338407425dc2aeaf0e21e145466d466a65db954663e91b1f" Sep 11 09:26:18.783891 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:26:18.783871 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-credit-scoring-5c864dc4f-7nhwc"] Sep 11 09:26:18.792328 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:26:18.792306 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-credit-scoring-5c864dc4f-7nhwc"] Sep 11 09:26:18.792460 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:26:18.792329 2575 scope.go:117] "RemoveContainer" containerID="0832a70da175d150e96cf4babc430455d58b1745017c69a1257f6f6e7129b322" Sep 11 09:26:20.113765 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:26:20.113724 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8098283c-6486-45f3-a766-27ae54e3ae36" path="/var/lib/kubelet/pods/8098283c-6486-45f3-a766-27ae54e3ae36/volumes" Sep 11 09:30:24.890225 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:30:24.890191 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-lq8cf"] Sep 11 09:30:24.892710 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:30:24.890526 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8098283c-6486-45f3-a766-27ae54e3ae36" containerName="feast-init" Sep 11 09:30:24.892710 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:30:24.890539 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="8098283c-6486-45f3-a766-27ae54e3ae36" containerName="feast-init" Sep 11 09:30:24.892710 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:30:24.890548 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8098283c-6486-45f3-a766-27ae54e3ae36" containerName="feast-apply" Sep 11 09:30:24.892710 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:30:24.890554 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="8098283c-6486-45f3-a766-27ae54e3ae36" containerName="feast-apply" Sep 11 09:30:24.892710 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:30:24.890561 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8098283c-6486-45f3-a766-27ae54e3ae36" containerName="online" Sep 11 09:30:24.892710 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:30:24.890567 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="8098283c-6486-45f3-a766-27ae54e3ae36" containerName="online" Sep 11 09:30:24.892710 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:30:24.890578 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db2ecbb0-ce0c-436a-99dc-e5c0ee4832b7" containerName="postgres" Sep 11 09:30:24.892710 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:30:24.890583 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="db2ecbb0-ce0c-436a-99dc-e5c0ee4832b7" containerName="postgres" Sep 11 09:30:24.892710 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:30:24.890589 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8098283c-6486-45f3-a766-27ae54e3ae36" containerName="offline" Sep 11 09:30:24.892710 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:30:24.890594 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="8098283c-6486-45f3-a766-27ae54e3ae36" containerName="offline" Sep 11 09:30:24.892710 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:30:24.890604 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8098283c-6486-45f3-a766-27ae54e3ae36" containerName="registry" Sep 11 09:30:24.892710 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:30:24.890609 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="8098283c-6486-45f3-a766-27ae54e3ae36" containerName="registry" Sep 11 09:30:24.892710 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:30:24.890619 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a62e8ddb-4603-4797-9e16-63c7b28c2e1a" containerName="redis" Sep 11 09:30:24.892710 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:30:24.890624 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="a62e8ddb-4603-4797-9e16-63c7b28c2e1a" containerName="redis" Sep 11 09:30:24.892710 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:30:24.890691 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="8098283c-6486-45f3-a766-27ae54e3ae36" containerName="registry" Sep 11 09:30:24.892710 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:30:24.890699 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="8098283c-6486-45f3-a766-27ae54e3ae36" containerName="online" Sep 11 09:30:24.892710 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:30:24.890706 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="8098283c-6486-45f3-a766-27ae54e3ae36" containerName="offline" Sep 11 09:30:24.892710 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:30:24.890713 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="a62e8ddb-4603-4797-9e16-63c7b28c2e1a" containerName="redis" Sep 11 09:30:24.892710 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:30:24.890718 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="db2ecbb0-ce0c-436a-99dc-e5c0ee4832b7" containerName="postgres" Sep 11 09:30:24.893728 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:30:24.893509 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-69bbd798b6-lq8cf" Sep 11 09:30:24.896145 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:30:24.896117 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"kube-root-ca.crt\"" Sep 11 09:30:24.896274 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:30:24.896174 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"openshift-service-ca.crt\"" Sep 11 09:30:24.898135 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:30:24.898119 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"default-dockercfg-bbm92\"" Sep 11 09:30:24.913393 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:30:24.913361 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-lq8cf"] Sep 11 09:30:24.942879 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:30:24.942842 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxtzw\" (UniqueName: \"kubernetes.io/projected/48f714a0-2a14-4a38-a3c6-889d72233088-kube-api-access-rxtzw\") pod \"redis-69bbd798b6-lq8cf\" (UID: \"48f714a0-2a14-4a38-a3c6-889d72233088\") " pod="test-ns-feast-rest/redis-69bbd798b6-lq8cf" Sep 11 09:30:25.043330 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:30:25.043290 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxtzw\" (UniqueName: \"kubernetes.io/projected/48f714a0-2a14-4a38-a3c6-889d72233088-kube-api-access-rxtzw\") pod \"redis-69bbd798b6-lq8cf\" (UID: \"48f714a0-2a14-4a38-a3c6-889d72233088\") " pod="test-ns-feast-rest/redis-69bbd798b6-lq8cf" Sep 11 09:30:25.054652 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:30:25.054631 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxtzw\" (UniqueName: \"kubernetes.io/projected/48f714a0-2a14-4a38-a3c6-889d72233088-kube-api-access-rxtzw\") pod \"redis-69bbd798b6-lq8cf\" (UID: \"48f714a0-2a14-4a38-a3c6-889d72233088\") " pod="test-ns-feast-rest/redis-69bbd798b6-lq8cf" Sep 11 09:30:25.214022 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:30:25.213929 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-69bbd798b6-lq8cf" Sep 11 09:30:25.483805 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:30:25.483705 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-lq8cf"] Sep 11 09:30:25.487693 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:30:25.487663 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod48f714a0_2a14_4a38_a3c6_889d72233088.slice/crio-be937dee0c305551086bf9f4a3b5851b4ae37b133e7f6a61ebbe10f98db0b375 WatchSource:0}: Error finding container be937dee0c305551086bf9f4a3b5851b4ae37b133e7f6a61ebbe10f98db0b375: Status 404 returned error can't find the container with id be937dee0c305551086bf9f4a3b5851b4ae37b133e7f6a61ebbe10f98db0b375 Sep 11 09:30:25.721209 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:30:25.721171 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-69bbd798b6-lq8cf" event={"ID":"48f714a0-2a14-4a38-a3c6-889d72233088","Type":"ContainerStarted","Data":"9cc3ff8fdbfbf7c3aceedc7823f3af025ddf9a8e096fb60f8ee389805cc1b1c2"} Sep 11 09:30:25.721209 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:30:25.721212 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-69bbd798b6-lq8cf" event={"ID":"48f714a0-2a14-4a38-a3c6-889d72233088","Type":"ContainerStarted","Data":"be937dee0c305551086bf9f4a3b5851b4ae37b133e7f6a61ebbe10f98db0b375"} Sep 11 09:30:25.737492 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:30:25.737383 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast-rest/redis-69bbd798b6-lq8cf" podStartSLOduration=1.7373630850000001 podStartE2EDuration="1.737363085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 09:30:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 09:30:25.736532426 +0000 UTC m=+1042.077142636" watchObservedRunningTime="2026-09-11 09:30:25.737363085 +0000 UTC m=+1042.077973294" Sep 11 09:30:30.320771 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:30:30.320737 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-4lszd"] Sep 11 09:30:30.324198 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:30:30.324177 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-864846d4f5-4lszd" Sep 11 09:30:30.326876 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:30:30.326852 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"postgres-secret\"" Sep 11 09:30:30.333050 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:30:30.333020 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-4lszd"] Sep 11 09:30:30.382590 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:30:30.382540 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dwjw\" (UniqueName: \"kubernetes.io/projected/d1a0a26d-6776-464e-9f1a-75dc8761f955-kube-api-access-2dwjw\") pod \"postgres-864846d4f5-4lszd\" (UID: \"d1a0a26d-6776-464e-9f1a-75dc8761f955\") " pod="test-ns-feast-rest/postgres-864846d4f5-4lszd" Sep 11 09:30:30.382799 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:30:30.382660 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/d1a0a26d-6776-464e-9f1a-75dc8761f955-postgresdata\") pod \"postgres-864846d4f5-4lszd\" (UID: \"d1a0a26d-6776-464e-9f1a-75dc8761f955\") " pod="test-ns-feast-rest/postgres-864846d4f5-4lszd" Sep 11 09:30:30.483871 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:30:30.483834 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dwjw\" (UniqueName: \"kubernetes.io/projected/d1a0a26d-6776-464e-9f1a-75dc8761f955-kube-api-access-2dwjw\") pod \"postgres-864846d4f5-4lszd\" (UID: \"d1a0a26d-6776-464e-9f1a-75dc8761f955\") " pod="test-ns-feast-rest/postgres-864846d4f5-4lszd" Sep 11 09:30:30.484104 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:30:30.483910 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/d1a0a26d-6776-464e-9f1a-75dc8761f955-postgresdata\") pod \"postgres-864846d4f5-4lszd\" (UID: \"d1a0a26d-6776-464e-9f1a-75dc8761f955\") " pod="test-ns-feast-rest/postgres-864846d4f5-4lszd" Sep 11 09:30:30.484255 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:30:30.484233 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/d1a0a26d-6776-464e-9f1a-75dc8761f955-postgresdata\") pod \"postgres-864846d4f5-4lszd\" (UID: \"d1a0a26d-6776-464e-9f1a-75dc8761f955\") " pod="test-ns-feast-rest/postgres-864846d4f5-4lszd" Sep 11 09:30:30.493120 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:30:30.493089 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dwjw\" (UniqueName: \"kubernetes.io/projected/d1a0a26d-6776-464e-9f1a-75dc8761f955-kube-api-access-2dwjw\") pod \"postgres-864846d4f5-4lszd\" (UID: \"d1a0a26d-6776-464e-9f1a-75dc8761f955\") " pod="test-ns-feast-rest/postgres-864846d4f5-4lszd" Sep 11 09:30:30.634710 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:30:30.634618 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-864846d4f5-4lszd" Sep 11 09:30:30.903497 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:30:30.903421 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-4lszd"] Sep 11 09:30:30.905697 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:30:30.905658 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd1a0a26d_6776_464e_9f1a_75dc8761f955.slice/crio-bdf802183f0f8656dd8805916aa075e3a206b3bf216906288fc5515b8c58079d WatchSource:0}: Error finding container bdf802183f0f8656dd8805916aa075e3a206b3bf216906288fc5515b8c58079d: Status 404 returned error can't find the container with id bdf802183f0f8656dd8805916aa075e3a206b3bf216906288fc5515b8c58079d Sep 11 09:30:31.750239 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:30:31.750197 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-864846d4f5-4lszd" event={"ID":"d1a0a26d-6776-464e-9f1a-75dc8761f955","Type":"ContainerStarted","Data":"03216769f2e2e86b1f0348c5177cb50c7271b1c8fdf9bea6b1b93806eb563b2f"} Sep 11 09:30:31.750239 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:30:31.750241 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-864846d4f5-4lszd" event={"ID":"d1a0a26d-6776-464e-9f1a-75dc8761f955","Type":"ContainerStarted","Data":"bdf802183f0f8656dd8805916aa075e3a206b3bf216906288fc5515b8c58079d"} Sep 11 09:30:31.766798 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:30:31.766742 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast-rest/postgres-864846d4f5-4lszd" podStartSLOduration=1.7667230269999998 podStartE2EDuration="1.766723027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 09:30:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 09:30:31.766417751 +0000 UTC m=+1048.107027960" watchObservedRunningTime="2026-09-11 09:30:31.766723027 +0000 UTC m=+1048.107333232" Sep 11 09:30:35.891716 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:30:35.891679 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-794b46d689-wkcxh"] Sep 11 09:30:35.895297 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:30:35.895278 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-794b46d689-wkcxh" Sep 11 09:30:35.897732 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:30:35.897707 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-online-tls\"" Sep 11 09:30:35.898446 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:30:35.898426 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-dockercfg-qq6vc\"" Sep 11 09:30:35.898634 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:30:35.898619 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-intra-comm\"" Sep 11 09:30:35.898853 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:30:35.898836 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-registry-tls\"" Sep 11 09:30:35.910714 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:30:35.910690 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-794b46d689-wkcxh"] Sep 11 09:30:35.926962 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:30:35.926934 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/23702c44-e27d-4cea-9191-dc7e9bf07ae5-feast-data\") pod \"feast-credit-scoring-794b46d689-wkcxh\" (UID: \"23702c44-e27d-4cea-9191-dc7e9bf07ae5\") " pod="test-ns-feast-rest/feast-credit-scoring-794b46d689-wkcxh" Sep 11 09:30:35.927109 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:30:35.926987 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/23702c44-e27d-4cea-9191-dc7e9bf07ae5-registry-tls\") pod \"feast-credit-scoring-794b46d689-wkcxh\" (UID: \"23702c44-e27d-4cea-9191-dc7e9bf07ae5\") " pod="test-ns-feast-rest/feast-credit-scoring-794b46d689-wkcxh" Sep 11 09:30:35.927109 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:30:35.927016 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/23702c44-e27d-4cea-9191-dc7e9bf07ae5-online-tls\") pod \"feast-credit-scoring-794b46d689-wkcxh\" (UID: \"23702c44-e27d-4cea-9191-dc7e9bf07ae5\") " pod="test-ns-feast-rest/feast-credit-scoring-794b46d689-wkcxh" Sep 11 09:30:35.927109 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:30:35.927084 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqs8w\" (UniqueName: \"kubernetes.io/projected/23702c44-e27d-4cea-9191-dc7e9bf07ae5-kube-api-access-pqs8w\") pod \"feast-credit-scoring-794b46d689-wkcxh\" (UID: \"23702c44-e27d-4cea-9191-dc7e9bf07ae5\") " pod="test-ns-feast-rest/feast-credit-scoring-794b46d689-wkcxh" Sep 11 09:30:36.027864 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:30:36.027828 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/23702c44-e27d-4cea-9191-dc7e9bf07ae5-feast-data\") pod \"feast-credit-scoring-794b46d689-wkcxh\" (UID: \"23702c44-e27d-4cea-9191-dc7e9bf07ae5\") " pod="test-ns-feast-rest/feast-credit-scoring-794b46d689-wkcxh" Sep 11 09:30:36.028073 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:30:36.027889 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/23702c44-e27d-4cea-9191-dc7e9bf07ae5-registry-tls\") pod \"feast-credit-scoring-794b46d689-wkcxh\" (UID: \"23702c44-e27d-4cea-9191-dc7e9bf07ae5\") " pod="test-ns-feast-rest/feast-credit-scoring-794b46d689-wkcxh" Sep 11 09:30:36.028073 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:30:36.028008 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/23702c44-e27d-4cea-9191-dc7e9bf07ae5-online-tls\") pod \"feast-credit-scoring-794b46d689-wkcxh\" (UID: \"23702c44-e27d-4cea-9191-dc7e9bf07ae5\") " pod="test-ns-feast-rest/feast-credit-scoring-794b46d689-wkcxh" Sep 11 09:30:36.028185 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:30:36.028070 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqs8w\" (UniqueName: \"kubernetes.io/projected/23702c44-e27d-4cea-9191-dc7e9bf07ae5-kube-api-access-pqs8w\") pod \"feast-credit-scoring-794b46d689-wkcxh\" (UID: \"23702c44-e27d-4cea-9191-dc7e9bf07ae5\") " pod="test-ns-feast-rest/feast-credit-scoring-794b46d689-wkcxh" Sep 11 09:30:36.028275 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:30:36.028256 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/23702c44-e27d-4cea-9191-dc7e9bf07ae5-feast-data\") pod \"feast-credit-scoring-794b46d689-wkcxh\" (UID: \"23702c44-e27d-4cea-9191-dc7e9bf07ae5\") " pod="test-ns-feast-rest/feast-credit-scoring-794b46d689-wkcxh" Sep 11 09:30:36.030347 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:30:36.030325 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/23702c44-e27d-4cea-9191-dc7e9bf07ae5-online-tls\") pod \"feast-credit-scoring-794b46d689-wkcxh\" (UID: \"23702c44-e27d-4cea-9191-dc7e9bf07ae5\") " pod="test-ns-feast-rest/feast-credit-scoring-794b46d689-wkcxh" Sep 11 09:30:36.030550 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:30:36.030529 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/23702c44-e27d-4cea-9191-dc7e9bf07ae5-registry-tls\") pod \"feast-credit-scoring-794b46d689-wkcxh\" (UID: \"23702c44-e27d-4cea-9191-dc7e9bf07ae5\") " pod="test-ns-feast-rest/feast-credit-scoring-794b46d689-wkcxh" Sep 11 09:30:36.037786 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:30:36.037765 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqs8w\" (UniqueName: \"kubernetes.io/projected/23702c44-e27d-4cea-9191-dc7e9bf07ae5-kube-api-access-pqs8w\") pod \"feast-credit-scoring-794b46d689-wkcxh\" (UID: \"23702c44-e27d-4cea-9191-dc7e9bf07ae5\") " pod="test-ns-feast-rest/feast-credit-scoring-794b46d689-wkcxh" Sep 11 09:30:36.205151 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:30:36.205063 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-794b46d689-wkcxh" Sep 11 09:30:36.489201 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:30:36.489173 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-794b46d689-wkcxh"] Sep 11 09:30:36.490076 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:30:36.490042 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23702c44_e27d_4cea_9191_dc7e9bf07ae5.slice/crio-d7af31da6c3b8d30f9c57f330d57a148e35a7bf485c1b9ea8a426f1c1e5a0767 WatchSource:0}: Error finding container d7af31da6c3b8d30f9c57f330d57a148e35a7bf485c1b9ea8a426f1c1e5a0767: Status 404 returned error can't find the container with id d7af31da6c3b8d30f9c57f330d57a148e35a7bf485c1b9ea8a426f1c1e5a0767 Sep 11 09:30:36.771126 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:30:36.771097 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-794b46d689-wkcxh" event={"ID":"23702c44-e27d-4cea-9191-dc7e9bf07ae5","Type":"ContainerStarted","Data":"69c07811a51693f7bd13b079c35fb27400805ea39f51f278c230475796032c5d"} Sep 11 09:30:36.771272 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:30:36.771145 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-794b46d689-wkcxh" event={"ID":"23702c44-e27d-4cea-9191-dc7e9bf07ae5","Type":"ContainerStarted","Data":"d7af31da6c3b8d30f9c57f330d57a148e35a7bf485c1b9ea8a426f1c1e5a0767"} Sep 11 09:30:37.777290 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:30:37.777260 2575 generic.go:356] "Generic (PLEG): container finished" podID="23702c44-e27d-4cea-9191-dc7e9bf07ae5" containerID="69c07811a51693f7bd13b079c35fb27400805ea39f51f278c230475796032c5d" exitCode=0 Sep 11 09:30:37.777674 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:30:37.777356 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-794b46d689-wkcxh" event={"ID":"23702c44-e27d-4cea-9191-dc7e9bf07ae5","Type":"ContainerDied","Data":"69c07811a51693f7bd13b079c35fb27400805ea39f51f278c230475796032c5d"} Sep 11 09:30:38.787135 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:30:38.787029 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-794b46d689-wkcxh" event={"ID":"23702c44-e27d-4cea-9191-dc7e9bf07ae5","Type":"ContainerStarted","Data":"ef135d6cde0be1f0e6a2fbb7f3f0891af7c70d2fe3d3b62ac0d5c8e82cf2ecee"} Sep 11 09:30:45.820754 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:30:45.820718 2575 generic.go:356] "Generic (PLEG): container finished" podID="23702c44-e27d-4cea-9191-dc7e9bf07ae5" containerID="ef135d6cde0be1f0e6a2fbb7f3f0891af7c70d2fe3d3b62ac0d5c8e82cf2ecee" exitCode=0 Sep 11 09:30:45.821175 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:30:45.820786 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-794b46d689-wkcxh" event={"ID":"23702c44-e27d-4cea-9191-dc7e9bf07ae5","Type":"ContainerDied","Data":"ef135d6cde0be1f0e6a2fbb7f3f0891af7c70d2fe3d3b62ac0d5c8e82cf2ecee"} Sep 11 09:30:46.829023 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:30:46.828985 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-794b46d689-wkcxh" event={"ID":"23702c44-e27d-4cea-9191-dc7e9bf07ae5","Type":"ContainerStarted","Data":"ec18b69b8686f5e1497e2d4d291c752e608497e44e1d8a381399fb11405693ca"} Sep 11 09:30:46.829395 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:30:46.829042 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-794b46d689-wkcxh" event={"ID":"23702c44-e27d-4cea-9191-dc7e9bf07ae5","Type":"ContainerStarted","Data":"6cccee43b082a2c0cc1dedad7f6acb8249ee3fb5ed4828e1c64f99d3deaab6c8"} Sep 11 09:30:46.854924 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:30:46.854854 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast-rest/feast-credit-scoring-794b46d689-wkcxh" podStartSLOduration=11.854833466 podStartE2EDuration="11.854833466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 09:30:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 09:30:46.853947941 +0000 UTC m=+1063.194558179" watchObservedRunningTime="2026-09-11 09:30:46.854833466 +0000 UTC m=+1063.195443676" Sep 11 09:30:48.205478 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:30:48.205427 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-credit-scoring-794b46d689-wkcxh" Sep 11 09:30:48.205869 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:30:48.205516 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-credit-scoring-794b46d689-wkcxh" Sep 11 09:30:48.207032 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:30:48.206986 2575 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-794b46d689-wkcxh" podUID="23702c44-e27d-4cea-9191-dc7e9bf07ae5" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.26:6571: connect: connection refused" Sep 11 09:30:48.207032 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:30:48.207019 2575 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-794b46d689-wkcxh" podUID="23702c44-e27d-4cea-9191-dc7e9bf07ae5" containerName="online" probeResult="failure" output="Get \"https://10.134.0.26:6567/health\": dial tcp 10.134.0.26:6567: connect: connection refused" Sep 11 09:30:51.205672 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:30:51.205625 2575 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-794b46d689-wkcxh" podUID="23702c44-e27d-4cea-9191-dc7e9bf07ae5" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.26:6571: connect: connection refused" Sep 11 09:30:51.205672 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:30:51.205645 2575 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-794b46d689-wkcxh" podUID="23702c44-e27d-4cea-9191-dc7e9bf07ae5" containerName="online" probeResult="failure" output="Get \"https://10.134.0.26:6567/health\": dial tcp 10.134.0.26:6567: connect: connection refused" Sep 11 09:30:54.206323 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:30:54.206276 2575 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-794b46d689-wkcxh" podUID="23702c44-e27d-4cea-9191-dc7e9bf07ae5" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.26:6571: connect: connection refused" Sep 11 09:30:54.212585 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:30:54.212552 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-credit-scoring-794b46d689-wkcxh" Sep 11 09:30:54.213153 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:30:54.213130 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-credit-scoring-794b46d689-wkcxh" Sep 11 09:30:54.216359 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:30:54.216334 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-credit-scoring-794b46d689-wkcxh" Sep 11 09:30:56.205226 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:30:56.205141 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-credit-scoring-794b46d689-wkcxh" Sep 11 09:30:57.206091 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:30:57.206051 2575 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-794b46d689-wkcxh" podUID="23702c44-e27d-4cea-9191-dc7e9bf07ae5" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.26:6571: connect: connection refused" Sep 11 09:31:00.206071 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:31:00.206039 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-credit-scoring-794b46d689-wkcxh" Sep 11 09:31:00.206786 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:31:00.206765 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-credit-scoring-794b46d689-wkcxh" Sep 11 09:31:01.792297 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:31:01.792259 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-6977969646-mbzdm"] Sep 11 09:31:01.795771 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:31:01.795749 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-6977969646-mbzdm" Sep 11 09:31:01.798431 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:31:01.798404 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-registry-tls\"" Sep 11 09:31:01.798593 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:31:01.798405 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-intra-comm\"" Sep 11 09:31:01.798593 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:31:01.798408 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-online-tls\"" Sep 11 09:31:01.798593 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:31:01.798406 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-dockercfg-vl7x5\"" Sep 11 09:31:01.807526 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:31:01.807501 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-6977969646-mbzdm"] Sep 11 09:31:01.870522 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:31:01.870487 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/00d79233-d405-4abe-9e5d-e010536fc3d3-online-tls\") pod \"feast-driver-ranking-6977969646-mbzdm\" (UID: \"00d79233-d405-4abe-9e5d-e010536fc3d3\") " pod="test-ns-feast-rest/feast-driver-ranking-6977969646-mbzdm" Sep 11 09:31:01.870522 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:31:01.870526 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/00d79233-d405-4abe-9e5d-e010536fc3d3-registry-tls\") pod \"feast-driver-ranking-6977969646-mbzdm\" (UID: \"00d79233-d405-4abe-9e5d-e010536fc3d3\") " pod="test-ns-feast-rest/feast-driver-ranking-6977969646-mbzdm" Sep 11 09:31:01.870740 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:31:01.870582 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/00d79233-d405-4abe-9e5d-e010536fc3d3-feast-data\") pod \"feast-driver-ranking-6977969646-mbzdm\" (UID: \"00d79233-d405-4abe-9e5d-e010536fc3d3\") " pod="test-ns-feast-rest/feast-driver-ranking-6977969646-mbzdm" Sep 11 09:31:01.870740 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:31:01.870685 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bf6v6\" (UniqueName: \"kubernetes.io/projected/00d79233-d405-4abe-9e5d-e010536fc3d3-kube-api-access-bf6v6\") pod \"feast-driver-ranking-6977969646-mbzdm\" (UID: \"00d79233-d405-4abe-9e5d-e010536fc3d3\") " pod="test-ns-feast-rest/feast-driver-ranking-6977969646-mbzdm" Sep 11 09:31:01.971639 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:31:01.971604 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bf6v6\" (UniqueName: \"kubernetes.io/projected/00d79233-d405-4abe-9e5d-e010536fc3d3-kube-api-access-bf6v6\") pod \"feast-driver-ranking-6977969646-mbzdm\" (UID: \"00d79233-d405-4abe-9e5d-e010536fc3d3\") " pod="test-ns-feast-rest/feast-driver-ranking-6977969646-mbzdm" Sep 11 09:31:01.971639 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:31:01.971644 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/00d79233-d405-4abe-9e5d-e010536fc3d3-online-tls\") pod \"feast-driver-ranking-6977969646-mbzdm\" (UID: \"00d79233-d405-4abe-9e5d-e010536fc3d3\") " pod="test-ns-feast-rest/feast-driver-ranking-6977969646-mbzdm" Sep 11 09:31:01.971904 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:31:01.971669 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/00d79233-d405-4abe-9e5d-e010536fc3d3-registry-tls\") pod \"feast-driver-ranking-6977969646-mbzdm\" (UID: \"00d79233-d405-4abe-9e5d-e010536fc3d3\") " pod="test-ns-feast-rest/feast-driver-ranking-6977969646-mbzdm" Sep 11 09:31:01.971904 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:31:01.971702 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/00d79233-d405-4abe-9e5d-e010536fc3d3-feast-data\") pod \"feast-driver-ranking-6977969646-mbzdm\" (UID: \"00d79233-d405-4abe-9e5d-e010536fc3d3\") " pod="test-ns-feast-rest/feast-driver-ranking-6977969646-mbzdm" Sep 11 09:31:01.972089 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:31:01.972071 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/00d79233-d405-4abe-9e5d-e010536fc3d3-feast-data\") pod \"feast-driver-ranking-6977969646-mbzdm\" (UID: \"00d79233-d405-4abe-9e5d-e010536fc3d3\") " pod="test-ns-feast-rest/feast-driver-ranking-6977969646-mbzdm" Sep 11 09:31:01.974143 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:31:01.974122 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/00d79233-d405-4abe-9e5d-e010536fc3d3-online-tls\") pod \"feast-driver-ranking-6977969646-mbzdm\" (UID: \"00d79233-d405-4abe-9e5d-e010536fc3d3\") " pod="test-ns-feast-rest/feast-driver-ranking-6977969646-mbzdm" Sep 11 09:31:01.974208 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:31:01.974159 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/00d79233-d405-4abe-9e5d-e010536fc3d3-registry-tls\") pod \"feast-driver-ranking-6977969646-mbzdm\" (UID: \"00d79233-d405-4abe-9e5d-e010536fc3d3\") " pod="test-ns-feast-rest/feast-driver-ranking-6977969646-mbzdm" Sep 11 09:31:01.980142 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:31:01.980117 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bf6v6\" (UniqueName: \"kubernetes.io/projected/00d79233-d405-4abe-9e5d-e010536fc3d3-kube-api-access-bf6v6\") pod \"feast-driver-ranking-6977969646-mbzdm\" (UID: \"00d79233-d405-4abe-9e5d-e010536fc3d3\") " pod="test-ns-feast-rest/feast-driver-ranking-6977969646-mbzdm" Sep 11 09:31:02.105248 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:31:02.105140 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-6977969646-mbzdm" Sep 11 09:31:02.383805 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:31:02.383766 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-6977969646-mbzdm"] Sep 11 09:31:02.385685 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:31:02.385662 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00d79233_d405_4abe_9e5d_e010536fc3d3.slice/crio-c3ccb437ca3f0679edbc5a262d78a7360447565c79e934f28cdb390cb3310ba8 WatchSource:0}: Error finding container c3ccb437ca3f0679edbc5a262d78a7360447565c79e934f28cdb390cb3310ba8: Status 404 returned error can't find the container with id c3ccb437ca3f0679edbc5a262d78a7360447565c79e934f28cdb390cb3310ba8 Sep 11 09:31:02.902854 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:31:02.902823 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-6977969646-mbzdm" event={"ID":"00d79233-d405-4abe-9e5d-e010536fc3d3","Type":"ContainerStarted","Data":"5d91e016dda1d8a3a7236f31bd50a2b4451fb29bcbe1aae4b9dc4d6a178584e0"} Sep 11 09:31:02.903239 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:31:02.902875 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-6977969646-mbzdm" event={"ID":"00d79233-d405-4abe-9e5d-e010536fc3d3","Type":"ContainerStarted","Data":"c3ccb437ca3f0679edbc5a262d78a7360447565c79e934f28cdb390cb3310ba8"} Sep 11 09:31:07.927619 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:31:07.927583 2575 generic.go:356] "Generic (PLEG): container finished" podID="00d79233-d405-4abe-9e5d-e010536fc3d3" containerID="5d91e016dda1d8a3a7236f31bd50a2b4451fb29bcbe1aae4b9dc4d6a178584e0" exitCode=0 Sep 11 09:31:07.928001 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:31:07.927656 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-6977969646-mbzdm" event={"ID":"00d79233-d405-4abe-9e5d-e010536fc3d3","Type":"ContainerDied","Data":"5d91e016dda1d8a3a7236f31bd50a2b4451fb29bcbe1aae4b9dc4d6a178584e0"} Sep 11 09:31:08.934515 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:31:08.934446 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-6977969646-mbzdm" event={"ID":"00d79233-d405-4abe-9e5d-e010536fc3d3","Type":"ContainerStarted","Data":"8056006ee4745e25d36e1017f2e7607178023cbb5d41dbffd1601e2b3c503586"} Sep 11 09:31:15.969625 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:31:15.969588 2575 generic.go:356] "Generic (PLEG): container finished" podID="00d79233-d405-4abe-9e5d-e010536fc3d3" containerID="8056006ee4745e25d36e1017f2e7607178023cbb5d41dbffd1601e2b3c503586" exitCode=0 Sep 11 09:31:15.970003 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:31:15.969657 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-6977969646-mbzdm" event={"ID":"00d79233-d405-4abe-9e5d-e010536fc3d3","Type":"ContainerDied","Data":"8056006ee4745e25d36e1017f2e7607178023cbb5d41dbffd1601e2b3c503586"} Sep 11 09:31:15.973582 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:31:15.973563 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-intra-comm\"" Sep 11 09:31:16.977059 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:31:16.977015 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-6977969646-mbzdm" event={"ID":"00d79233-d405-4abe-9e5d-e010536fc3d3","Type":"ContainerStarted","Data":"9ad695d25cdfa1cf4839d3af938f14876d4812fed2a3f750cc6a37f361bdc170"} Sep 11 09:31:16.977417 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:31:16.977096 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-6977969646-mbzdm" event={"ID":"00d79233-d405-4abe-9e5d-e010536fc3d3","Type":"ContainerStarted","Data":"b1ab91b8b205d2d3a3eef9488c00619c435dd15db1ef37b27d0be77a0454986f"} Sep 11 09:31:17.105828 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:31:17.105794 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-driver-ranking-6977969646-mbzdm" Sep 11 09:31:17.106062 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:31:17.105894 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-driver-ranking-6977969646-mbzdm" Sep 11 09:31:17.107725 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:31:17.107678 2575 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-6977969646-mbzdm" podUID="00d79233-d405-4abe-9e5d-e010536fc3d3" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.27:6571: connect: connection refused" Sep 11 09:31:17.107725 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:31:17.107707 2575 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-6977969646-mbzdm" podUID="00d79233-d405-4abe-9e5d-e010536fc3d3" containerName="online" probeResult="failure" output="Get \"https://10.134.0.27:6567/health\": dial tcp 10.134.0.27:6567: connect: connection refused" Sep 11 09:31:20.106528 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:31:20.106459 2575 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-6977969646-mbzdm" podUID="00d79233-d405-4abe-9e5d-e010536fc3d3" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.27:6571: connect: connection refused" Sep 11 09:31:20.107054 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:31:20.106548 2575 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-6977969646-mbzdm" podUID="00d79233-d405-4abe-9e5d-e010536fc3d3" containerName="online" probeResult="failure" output="Get \"https://10.134.0.27:6567/health\": dial tcp 10.134.0.27:6567: connect: connection refused" Sep 11 09:31:22.107021 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:31:22.106151 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-driver-ranking-6977969646-mbzdm" Sep 11 09:31:22.107637 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:31:22.107607 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-driver-ranking-6977969646-mbzdm" Sep 11 09:31:23.105767 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:31:23.105720 2575 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-6977969646-mbzdm" podUID="00d79233-d405-4abe-9e5d-e010536fc3d3" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.27:6571: connect: connection refused" Sep 11 09:31:23.111932 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:31:23.111898 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-driver-ranking-6977969646-mbzdm" Sep 11 09:31:23.137830 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:31:23.137772 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast-rest/feast-driver-ranking-6977969646-mbzdm" podStartSLOduration=22.13775351 podStartE2EDuration="22.13775351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 09:31:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 09:31:17.011854228 +0000 UTC m=+1093.352464435" watchObservedRunningTime="2026-09-11 09:31:23.13775351 +0000 UTC m=+1099.478363715" Sep 11 09:31:24.017335 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:31:24.017295 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-driver-ranking-6977969646-mbzdm" Sep 11 09:31:26.105883 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:31:26.105823 2575 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-6977969646-mbzdm" podUID="00d79233-d405-4abe-9e5d-e010536fc3d3" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.27:6571: connect: connection refused" Sep 11 09:31:29.105733 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:31:29.105703 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-driver-ranking-6977969646-mbzdm" Sep 11 09:31:29.106386 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:31:29.106366 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-driver-ranking-6977969646-mbzdm" Sep 11 09:32:53.545692 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:32:53.545655 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-794b46d689-wkcxh"] Sep 11 09:32:53.546205 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:32:53.545913 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast-rest/feast-credit-scoring-794b46d689-wkcxh" podUID="23702c44-e27d-4cea-9191-dc7e9bf07ae5" containerName="registry" containerID="cri-o://6cccee43b082a2c0cc1dedad7f6acb8249ee3fb5ed4828e1c64f99d3deaab6c8" gracePeriod=30 Sep 11 09:32:53.546205 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:32:53.545955 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast-rest/feast-credit-scoring-794b46d689-wkcxh" podUID="23702c44-e27d-4cea-9191-dc7e9bf07ae5" containerName="online" containerID="cri-o://ec18b69b8686f5e1497e2d4d291c752e608497e44e1d8a381399fb11405693ca" gracePeriod=30 Sep 11 09:32:53.550001 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:32:53.549972 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-6977969646-mbzdm"] Sep 11 09:32:53.550285 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:32:53.550242 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast-rest/feast-driver-ranking-6977969646-mbzdm" podUID="00d79233-d405-4abe-9e5d-e010536fc3d3" containerName="registry" containerID="cri-o://b1ab91b8b205d2d3a3eef9488c00619c435dd15db1ef37b27d0be77a0454986f" gracePeriod=30 Sep 11 09:32:53.550391 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:32:53.550261 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast-rest/feast-driver-ranking-6977969646-mbzdm" podUID="00d79233-d405-4abe-9e5d-e010536fc3d3" containerName="online" containerID="cri-o://9ad695d25cdfa1cf4839d3af938f14876d4812fed2a3f750cc6a37f361bdc170" gracePeriod=30 Sep 11 09:32:54.014447 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:32:54.014391 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-driver-ranking-6977969646-mbzdm" podUID="00d79233-d405-4abe-9e5d-e010536fc3d3" containerName="online" probeResult="failure" output="Get \"https://10.134.0.27:6567/health\": dial tcp 10.134.0.27:6567: connect: connection refused" Sep 11 09:32:54.214254 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:32:54.214207 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-credit-scoring-794b46d689-wkcxh" podUID="23702c44-e27d-4cea-9191-dc7e9bf07ae5" containerName="online" probeResult="failure" output="Get \"https://10.134.0.26:6567/health\": dial tcp 10.134.0.26:6567: connect: connection refused" Sep 11 09:32:54.924394 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:32:54.924358 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-4lszd"] Sep 11 09:32:54.924792 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:32:54.924559 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast-rest/postgres-864846d4f5-4lszd" podUID="d1a0a26d-6776-464e-9f1a-75dc8761f955" containerName="postgres" containerID="cri-o://03216769f2e2e86b1f0348c5177cb50c7271b1c8fdf9bea6b1b93806eb563b2f" gracePeriod=30 Sep 11 09:32:54.929159 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:32:54.929131 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-lq8cf"] Sep 11 09:32:54.929293 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:32:54.929269 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast-rest/redis-69bbd798b6-lq8cf" podUID="48f714a0-2a14-4a38-a3c6-889d72233088" containerName="redis" containerID="cri-o://9cc3ff8fdbfbf7c3aceedc7823f3af025ddf9a8e096fb60f8ee389805cc1b1c2" gracePeriod=30 Sep 11 09:32:55.177688 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:32:55.177669 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-864846d4f5-4lszd" Sep 11 09:32:55.184443 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:32:55.184419 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-69bbd798b6-lq8cf" Sep 11 09:32:55.301585 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:32:55.301540 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/d1a0a26d-6776-464e-9f1a-75dc8761f955-postgresdata\") pod \"d1a0a26d-6776-464e-9f1a-75dc8761f955\" (UID: \"d1a0a26d-6776-464e-9f1a-75dc8761f955\") " Sep 11 09:32:55.301585 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:32:55.301585 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2dwjw\" (UniqueName: \"kubernetes.io/projected/d1a0a26d-6776-464e-9f1a-75dc8761f955-kube-api-access-2dwjw\") pod \"d1a0a26d-6776-464e-9f1a-75dc8761f955\" (UID: \"d1a0a26d-6776-464e-9f1a-75dc8761f955\") " Sep 11 09:32:55.301845 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:32:55.301642 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rxtzw\" (UniqueName: \"kubernetes.io/projected/48f714a0-2a14-4a38-a3c6-889d72233088-kube-api-access-rxtzw\") pod \"48f714a0-2a14-4a38-a3c6-889d72233088\" (UID: \"48f714a0-2a14-4a38-a3c6-889d72233088\") " Sep 11 09:32:55.303848 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:32:55.303822 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1a0a26d-6776-464e-9f1a-75dc8761f955-kube-api-access-2dwjw" (OuterVolumeSpecName: "kube-api-access-2dwjw") pod "d1a0a26d-6776-464e-9f1a-75dc8761f955" (UID: "d1a0a26d-6776-464e-9f1a-75dc8761f955"). InnerVolumeSpecName "kube-api-access-2dwjw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 09:32:55.303981 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:32:55.303864 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48f714a0-2a14-4a38-a3c6-889d72233088-kube-api-access-rxtzw" (OuterVolumeSpecName: "kube-api-access-rxtzw") pod "48f714a0-2a14-4a38-a3c6-889d72233088" (UID: "48f714a0-2a14-4a38-a3c6-889d72233088"). InnerVolumeSpecName "kube-api-access-rxtzw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 09:32:55.332485 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:32:55.332425 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1a0a26d-6776-464e-9f1a-75dc8761f955-postgresdata" (OuterVolumeSpecName: "postgresdata") pod "d1a0a26d-6776-464e-9f1a-75dc8761f955" (UID: "d1a0a26d-6776-464e-9f1a-75dc8761f955"). InnerVolumeSpecName "postgresdata". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 09:32:55.402954 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:32:55.402913 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dwjw\" (UniqueName: \"kubernetes.io/projected/d1a0a26d-6776-464e-9f1a-75dc8761f955-kube-api-access-2dwjw\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\"" Sep 11 09:32:55.402954 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:32:55.402945 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxtzw\" (UniqueName: \"kubernetes.io/projected/48f714a0-2a14-4a38-a3c6-889d72233088-kube-api-access-rxtzw\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\"" Sep 11 09:32:55.402954 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:32:55.402955 2575 reconciler_common.go:299] "Volume detached for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/d1a0a26d-6776-464e-9f1a-75dc8761f955-postgresdata\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\"" Sep 11 09:32:55.440004 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:32:55.439917 2575 generic.go:356] "Generic (PLEG): container finished" podID="48f714a0-2a14-4a38-a3c6-889d72233088" containerID="9cc3ff8fdbfbf7c3aceedc7823f3af025ddf9a8e096fb60f8ee389805cc1b1c2" exitCode=0 Sep 11 09:32:55.440004 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:32:55.439962 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-69bbd798b6-lq8cf" Sep 11 09:32:55.440208 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:32:55.440001 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-69bbd798b6-lq8cf" event={"ID":"48f714a0-2a14-4a38-a3c6-889d72233088","Type":"ContainerDied","Data":"9cc3ff8fdbfbf7c3aceedc7823f3af025ddf9a8e096fb60f8ee389805cc1b1c2"} Sep 11 09:32:55.440208 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:32:55.440056 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-69bbd798b6-lq8cf" event={"ID":"48f714a0-2a14-4a38-a3c6-889d72233088","Type":"ContainerDied","Data":"be937dee0c305551086bf9f4a3b5851b4ae37b133e7f6a61ebbe10f98db0b375"} Sep 11 09:32:55.440208 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:32:55.440082 2575 scope.go:117] "RemoveContainer" containerID="9cc3ff8fdbfbf7c3aceedc7823f3af025ddf9a8e096fb60f8ee389805cc1b1c2" Sep 11 09:32:55.441543 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:32:55.441520 2575 generic.go:356] "Generic (PLEG): container finished" podID="d1a0a26d-6776-464e-9f1a-75dc8761f955" containerID="03216769f2e2e86b1f0348c5177cb50c7271b1c8fdf9bea6b1b93806eb563b2f" exitCode=0 Sep 11 09:32:55.441673 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:32:55.441573 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-864846d4f5-4lszd" Sep 11 09:32:55.441673 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:32:55.441610 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-864846d4f5-4lszd" event={"ID":"d1a0a26d-6776-464e-9f1a-75dc8761f955","Type":"ContainerDied","Data":"03216769f2e2e86b1f0348c5177cb50c7271b1c8fdf9bea6b1b93806eb563b2f"} Sep 11 09:32:55.441673 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:32:55.441662 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-864846d4f5-4lszd" event={"ID":"d1a0a26d-6776-464e-9f1a-75dc8761f955","Type":"ContainerDied","Data":"bdf802183f0f8656dd8805916aa075e3a206b3bf216906288fc5515b8c58079d"} Sep 11 09:32:55.445028 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:32:55.445003 2575 generic.go:356] "Generic (PLEG): container finished" podID="00d79233-d405-4abe-9e5d-e010536fc3d3" containerID="9ad695d25cdfa1cf4839d3af938f14876d4812fed2a3f750cc6a37f361bdc170" exitCode=0 Sep 11 09:32:55.445161 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:32:55.445076 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-6977969646-mbzdm" event={"ID":"00d79233-d405-4abe-9e5d-e010536fc3d3","Type":"ContainerDied","Data":"9ad695d25cdfa1cf4839d3af938f14876d4812fed2a3f750cc6a37f361bdc170"} Sep 11 09:32:55.448131 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:32:55.448108 2575 generic.go:356] "Generic (PLEG): container finished" podID="23702c44-e27d-4cea-9191-dc7e9bf07ae5" containerID="ec18b69b8686f5e1497e2d4d291c752e608497e44e1d8a381399fb11405693ca" exitCode=0 Sep 11 09:32:55.448231 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:32:55.448160 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-794b46d689-wkcxh" event={"ID":"23702c44-e27d-4cea-9191-dc7e9bf07ae5","Type":"ContainerDied","Data":"ec18b69b8686f5e1497e2d4d291c752e608497e44e1d8a381399fb11405693ca"} Sep 11 09:32:55.455438 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:32:55.455419 2575 scope.go:117] "RemoveContainer" containerID="9cc3ff8fdbfbf7c3aceedc7823f3af025ddf9a8e096fb60f8ee389805cc1b1c2" Sep 11 09:32:55.455760 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:32:55.455741 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9cc3ff8fdbfbf7c3aceedc7823f3af025ddf9a8e096fb60f8ee389805cc1b1c2\": container with ID starting with 9cc3ff8fdbfbf7c3aceedc7823f3af025ddf9a8e096fb60f8ee389805cc1b1c2 not found: ID does not exist" containerID="9cc3ff8fdbfbf7c3aceedc7823f3af025ddf9a8e096fb60f8ee389805cc1b1c2" Sep 11 09:32:55.455816 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:32:55.455770 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cc3ff8fdbfbf7c3aceedc7823f3af025ddf9a8e096fb60f8ee389805cc1b1c2"} err="failed to get container status \"9cc3ff8fdbfbf7c3aceedc7823f3af025ddf9a8e096fb60f8ee389805cc1b1c2\": rpc error: code = NotFound desc = could not find container \"9cc3ff8fdbfbf7c3aceedc7823f3af025ddf9a8e096fb60f8ee389805cc1b1c2\": container with ID starting with 9cc3ff8fdbfbf7c3aceedc7823f3af025ddf9a8e096fb60f8ee389805cc1b1c2 not found: ID does not exist" Sep 11 09:32:55.455816 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:32:55.455787 2575 scope.go:117] "RemoveContainer" containerID="03216769f2e2e86b1f0348c5177cb50c7271b1c8fdf9bea6b1b93806eb563b2f" Sep 11 09:32:55.463656 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:32:55.463633 2575 scope.go:117] "RemoveContainer" containerID="03216769f2e2e86b1f0348c5177cb50c7271b1c8fdf9bea6b1b93806eb563b2f" Sep 11 09:32:55.464181 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:32:55.464160 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03216769f2e2e86b1f0348c5177cb50c7271b1c8fdf9bea6b1b93806eb563b2f\": container with ID starting with 03216769f2e2e86b1f0348c5177cb50c7271b1c8fdf9bea6b1b93806eb563b2f not found: ID does not exist" containerID="03216769f2e2e86b1f0348c5177cb50c7271b1c8fdf9bea6b1b93806eb563b2f" Sep 11 09:32:55.464265 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:32:55.464193 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03216769f2e2e86b1f0348c5177cb50c7271b1c8fdf9bea6b1b93806eb563b2f"} err="failed to get container status \"03216769f2e2e86b1f0348c5177cb50c7271b1c8fdf9bea6b1b93806eb563b2f\": rpc error: code = NotFound desc = could not find container \"03216769f2e2e86b1f0348c5177cb50c7271b1c8fdf9bea6b1b93806eb563b2f\": container with ID starting with 03216769f2e2e86b1f0348c5177cb50c7271b1c8fdf9bea6b1b93806eb563b2f not found: ID does not exist" Sep 11 09:32:55.465908 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:32:55.465887 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-lq8cf"] Sep 11 09:32:55.467333 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:32:55.467316 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-lq8cf"] Sep 11 09:32:55.479577 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:32:55.479551 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-4lszd"] Sep 11 09:32:55.485121 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:32:55.485097 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-4lszd"] Sep 11 09:32:56.114003 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:32:56.113973 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="48f714a0-2a14-4a38-a3c6-889d72233088" path="/var/lib/kubelet/pods/48f714a0-2a14-4a38-a3c6-889d72233088/volumes" Sep 11 09:32:56.114367 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:32:56.114297 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d1a0a26d-6776-464e-9f1a-75dc8761f955" path="/var/lib/kubelet/pods/d1a0a26d-6776-464e-9f1a-75dc8761f955/volumes" Sep 11 09:33:04.014584 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:04.014535 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-driver-ranking-6977969646-mbzdm" podUID="00d79233-d405-4abe-9e5d-e010536fc3d3" containerName="online" probeResult="failure" output="Get \"https://10.134.0.27:6567/health\": dial tcp 10.134.0.27:6567: connect: connection refused" Sep 11 09:33:04.214073 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:04.214026 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-credit-scoring-794b46d689-wkcxh" podUID="23702c44-e27d-4cea-9191-dc7e9bf07ae5" containerName="online" probeResult="failure" output="Get \"https://10.134.0.26:6567/health\": dial tcp 10.134.0.26:6567: connect: connection refused" Sep 11 09:33:14.014683 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:14.014636 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-driver-ranking-6977969646-mbzdm" podUID="00d79233-d405-4abe-9e5d-e010536fc3d3" containerName="online" probeResult="failure" output="Get \"https://10.134.0.27:6567/health\": dial tcp 10.134.0.27:6567: connect: connection refused" Sep 11 09:33:14.015092 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:14.014810 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-driver-ranking-6977969646-mbzdm" Sep 11 09:33:14.213645 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:14.213596 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-credit-scoring-794b46d689-wkcxh" podUID="23702c44-e27d-4cea-9191-dc7e9bf07ae5" containerName="online" probeResult="failure" output="Get \"https://10.134.0.26:6567/health\": dial tcp 10.134.0.26:6567: connect: connection refused" Sep 11 09:33:14.213832 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:14.213725 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-credit-scoring-794b46d689-wkcxh" Sep 11 09:33:24.014230 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:24.014195 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-driver-ranking-6977969646-mbzdm" podUID="00d79233-d405-4abe-9e5d-e010536fc3d3" containerName="online" probeResult="failure" output="Get \"https://10.134.0.27:6567/health\": dial tcp 10.134.0.27:6567: connect: connection refused" Sep 11 09:33:24.224104 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:24.224076 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-794b46d689-wkcxh" Sep 11 09:33:24.226873 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:24.226849 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-6977969646-mbzdm" Sep 11 09:33:24.356033 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:24.355931 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pqs8w\" (UniqueName: \"kubernetes.io/projected/23702c44-e27d-4cea-9191-dc7e9bf07ae5-kube-api-access-pqs8w\") pod \"23702c44-e27d-4cea-9191-dc7e9bf07ae5\" (UID: \"23702c44-e27d-4cea-9191-dc7e9bf07ae5\") " Sep 11 09:33:24.356033 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:24.355976 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/23702c44-e27d-4cea-9191-dc7e9bf07ae5-feast-data\") pod \"23702c44-e27d-4cea-9191-dc7e9bf07ae5\" (UID: \"23702c44-e27d-4cea-9191-dc7e9bf07ae5\") " Sep 11 09:33:24.356033 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:24.356014 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/00d79233-d405-4abe-9e5d-e010536fc3d3-feast-data\") pod \"00d79233-d405-4abe-9e5d-e010536fc3d3\" (UID: \"00d79233-d405-4abe-9e5d-e010536fc3d3\") " Sep 11 09:33:24.356301 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:24.356043 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/23702c44-e27d-4cea-9191-dc7e9bf07ae5-registry-tls\") pod \"23702c44-e27d-4cea-9191-dc7e9bf07ae5\" (UID: \"23702c44-e27d-4cea-9191-dc7e9bf07ae5\") " Sep 11 09:33:24.356301 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:24.356073 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bf6v6\" (UniqueName: \"kubernetes.io/projected/00d79233-d405-4abe-9e5d-e010536fc3d3-kube-api-access-bf6v6\") pod \"00d79233-d405-4abe-9e5d-e010536fc3d3\" (UID: \"00d79233-d405-4abe-9e5d-e010536fc3d3\") " Sep 11 09:33:24.356301 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:24.356141 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/23702c44-e27d-4cea-9191-dc7e9bf07ae5-online-tls\") pod \"23702c44-e27d-4cea-9191-dc7e9bf07ae5\" (UID: \"23702c44-e27d-4cea-9191-dc7e9bf07ae5\") " Sep 11 09:33:24.356301 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:24.356186 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/00d79233-d405-4abe-9e5d-e010536fc3d3-registry-tls\") pod \"00d79233-d405-4abe-9e5d-e010536fc3d3\" (UID: \"00d79233-d405-4abe-9e5d-e010536fc3d3\") " Sep 11 09:33:24.356301 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:24.356220 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/00d79233-d405-4abe-9e5d-e010536fc3d3-online-tls\") pod \"00d79233-d405-4abe-9e5d-e010536fc3d3\" (UID: \"00d79233-d405-4abe-9e5d-e010536fc3d3\") " Sep 11 09:33:24.356692 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:24.356641 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00d79233-d405-4abe-9e5d-e010536fc3d3-feast-data" (OuterVolumeSpecName: "feast-data") pod "00d79233-d405-4abe-9e5d-e010536fc3d3" (UID: "00d79233-d405-4abe-9e5d-e010536fc3d3"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 09:33:24.358436 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:24.358405 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00d79233-d405-4abe-9e5d-e010536fc3d3-kube-api-access-bf6v6" (OuterVolumeSpecName: "kube-api-access-bf6v6") pod "00d79233-d405-4abe-9e5d-e010536fc3d3" (UID: "00d79233-d405-4abe-9e5d-e010536fc3d3"). InnerVolumeSpecName "kube-api-access-bf6v6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 09:33:24.358706 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:24.358685 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23702c44-e27d-4cea-9191-dc7e9bf07ae5-kube-api-access-pqs8w" (OuterVolumeSpecName: "kube-api-access-pqs8w") pod "23702c44-e27d-4cea-9191-dc7e9bf07ae5" (UID: "23702c44-e27d-4cea-9191-dc7e9bf07ae5"). InnerVolumeSpecName "kube-api-access-pqs8w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 09:33:24.358706 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:24.358689 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23702c44-e27d-4cea-9191-dc7e9bf07ae5-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "23702c44-e27d-4cea-9191-dc7e9bf07ae5" (UID: "23702c44-e27d-4cea-9191-dc7e9bf07ae5"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 09:33:24.358891 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:24.358870 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00d79233-d405-4abe-9e5d-e010536fc3d3-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "00d79233-d405-4abe-9e5d-e010536fc3d3" (UID: "00d79233-d405-4abe-9e5d-e010536fc3d3"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 09:33:24.358951 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:24.358922 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00d79233-d405-4abe-9e5d-e010536fc3d3-online-tls" (OuterVolumeSpecName: "online-tls") pod "00d79233-d405-4abe-9e5d-e010536fc3d3" (UID: "00d79233-d405-4abe-9e5d-e010536fc3d3"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 09:33:24.358984 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:24.358923 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23702c44-e27d-4cea-9191-dc7e9bf07ae5-online-tls" (OuterVolumeSpecName: "online-tls") pod "23702c44-e27d-4cea-9191-dc7e9bf07ae5" (UID: "23702c44-e27d-4cea-9191-dc7e9bf07ae5"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 09:33:24.365830 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:24.365803 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23702c44-e27d-4cea-9191-dc7e9bf07ae5-feast-data" (OuterVolumeSpecName: "feast-data") pod "23702c44-e27d-4cea-9191-dc7e9bf07ae5" (UID: "23702c44-e27d-4cea-9191-dc7e9bf07ae5"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 09:33:24.457019 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:24.456986 2575 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/00d79233-d405-4abe-9e5d-e010536fc3d3-feast-data\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\"" Sep 11 09:33:24.457019 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:24.457014 2575 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/23702c44-e27d-4cea-9191-dc7e9bf07ae5-registry-tls\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\"" Sep 11 09:33:24.457019 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:24.457023 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bf6v6\" (UniqueName: \"kubernetes.io/projected/00d79233-d405-4abe-9e5d-e010536fc3d3-kube-api-access-bf6v6\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\"" Sep 11 09:33:24.457245 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:24.457032 2575 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/23702c44-e27d-4cea-9191-dc7e9bf07ae5-online-tls\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\"" Sep 11 09:33:24.457245 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:24.457041 2575 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/00d79233-d405-4abe-9e5d-e010536fc3d3-registry-tls\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\"" Sep 11 09:33:24.457245 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:24.457048 2575 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/00d79233-d405-4abe-9e5d-e010536fc3d3-online-tls\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\"" Sep 11 09:33:24.457245 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:24.457056 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqs8w\" (UniqueName: \"kubernetes.io/projected/23702c44-e27d-4cea-9191-dc7e9bf07ae5-kube-api-access-pqs8w\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\"" Sep 11 09:33:24.457245 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:24.457063 2575 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/23702c44-e27d-4cea-9191-dc7e9bf07ae5-feast-data\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\"" Sep 11 09:33:24.572012 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:24.571978 2575 generic.go:356] "Generic (PLEG): container finished" podID="00d79233-d405-4abe-9e5d-e010536fc3d3" containerID="b1ab91b8b205d2d3a3eef9488c00619c435dd15db1ef37b27d0be77a0454986f" exitCode=137 Sep 11 09:33:24.572173 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:24.572032 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-6977969646-mbzdm" Sep 11 09:33:24.572173 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:24.572058 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-6977969646-mbzdm" event={"ID":"00d79233-d405-4abe-9e5d-e010536fc3d3","Type":"ContainerDied","Data":"b1ab91b8b205d2d3a3eef9488c00619c435dd15db1ef37b27d0be77a0454986f"} Sep 11 09:33:24.572173 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:24.572106 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-6977969646-mbzdm" event={"ID":"00d79233-d405-4abe-9e5d-e010536fc3d3","Type":"ContainerDied","Data":"c3ccb437ca3f0679edbc5a262d78a7360447565c79e934f28cdb390cb3310ba8"} Sep 11 09:33:24.572173 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:24.572133 2575 scope.go:117] "RemoveContainer" containerID="9ad695d25cdfa1cf4839d3af938f14876d4812fed2a3f750cc6a37f361bdc170" Sep 11 09:33:24.574560 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:24.574524 2575 generic.go:356] "Generic (PLEG): container finished" podID="23702c44-e27d-4cea-9191-dc7e9bf07ae5" containerID="6cccee43b082a2c0cc1dedad7f6acb8249ee3fb5ed4828e1c64f99d3deaab6c8" exitCode=137 Sep 11 09:33:24.574680 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:24.574582 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-794b46d689-wkcxh" event={"ID":"23702c44-e27d-4cea-9191-dc7e9bf07ae5","Type":"ContainerDied","Data":"6cccee43b082a2c0cc1dedad7f6acb8249ee3fb5ed4828e1c64f99d3deaab6c8"} Sep 11 09:33:24.574680 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:24.574638 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-794b46d689-wkcxh" event={"ID":"23702c44-e27d-4cea-9191-dc7e9bf07ae5","Type":"ContainerDied","Data":"d7af31da6c3b8d30f9c57f330d57a148e35a7bf485c1b9ea8a426f1c1e5a0767"} Sep 11 09:33:24.574680 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:24.574594 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-794b46d689-wkcxh" Sep 11 09:33:24.581164 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:24.581128 2575 scope.go:117] "RemoveContainer" containerID="b1ab91b8b205d2d3a3eef9488c00619c435dd15db1ef37b27d0be77a0454986f" Sep 11 09:33:24.589791 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:24.589774 2575 scope.go:117] "RemoveContainer" containerID="8056006ee4745e25d36e1017f2e7607178023cbb5d41dbffd1601e2b3c503586" Sep 11 09:33:24.597575 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:24.597539 2575 scope.go:117] "RemoveContainer" containerID="5d91e016dda1d8a3a7236f31bd50a2b4451fb29bcbe1aae4b9dc4d6a178584e0" Sep 11 09:33:24.597781 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:24.597760 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-794b46d689-wkcxh"] Sep 11 09:33:24.602877 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:24.602850 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-794b46d689-wkcxh"] Sep 11 09:33:24.606071 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:24.606025 2575 scope.go:117] "RemoveContainer" containerID="9ad695d25cdfa1cf4839d3af938f14876d4812fed2a3f750cc6a37f361bdc170" Sep 11 09:33:24.606342 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:33:24.606324 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ad695d25cdfa1cf4839d3af938f14876d4812fed2a3f750cc6a37f361bdc170\": container with ID starting with 9ad695d25cdfa1cf4839d3af938f14876d4812fed2a3f750cc6a37f361bdc170 not found: ID does not exist" containerID="9ad695d25cdfa1cf4839d3af938f14876d4812fed2a3f750cc6a37f361bdc170" Sep 11 09:33:24.606393 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:24.606351 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ad695d25cdfa1cf4839d3af938f14876d4812fed2a3f750cc6a37f361bdc170"} err="failed to get container status \"9ad695d25cdfa1cf4839d3af938f14876d4812fed2a3f750cc6a37f361bdc170\": rpc error: code = NotFound desc = could not find container \"9ad695d25cdfa1cf4839d3af938f14876d4812fed2a3f750cc6a37f361bdc170\": container with ID starting with 9ad695d25cdfa1cf4839d3af938f14876d4812fed2a3f750cc6a37f361bdc170 not found: ID does not exist" Sep 11 09:33:24.606393 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:24.606369 2575 scope.go:117] "RemoveContainer" containerID="b1ab91b8b205d2d3a3eef9488c00619c435dd15db1ef37b27d0be77a0454986f" Sep 11 09:33:24.606687 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:33:24.606664 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1ab91b8b205d2d3a3eef9488c00619c435dd15db1ef37b27d0be77a0454986f\": container with ID starting with b1ab91b8b205d2d3a3eef9488c00619c435dd15db1ef37b27d0be77a0454986f not found: ID does not exist" containerID="b1ab91b8b205d2d3a3eef9488c00619c435dd15db1ef37b27d0be77a0454986f" Sep 11 09:33:24.606786 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:24.606690 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1ab91b8b205d2d3a3eef9488c00619c435dd15db1ef37b27d0be77a0454986f"} err="failed to get container status \"b1ab91b8b205d2d3a3eef9488c00619c435dd15db1ef37b27d0be77a0454986f\": rpc error: code = NotFound desc = could not find container \"b1ab91b8b205d2d3a3eef9488c00619c435dd15db1ef37b27d0be77a0454986f\": container with ID starting with b1ab91b8b205d2d3a3eef9488c00619c435dd15db1ef37b27d0be77a0454986f not found: ID does not exist" Sep 11 09:33:24.606786 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:24.606705 2575 scope.go:117] "RemoveContainer" containerID="8056006ee4745e25d36e1017f2e7607178023cbb5d41dbffd1601e2b3c503586" Sep 11 09:33:24.606970 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:33:24.606952 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8056006ee4745e25d36e1017f2e7607178023cbb5d41dbffd1601e2b3c503586\": container with ID starting with 8056006ee4745e25d36e1017f2e7607178023cbb5d41dbffd1601e2b3c503586 not found: ID does not exist" containerID="8056006ee4745e25d36e1017f2e7607178023cbb5d41dbffd1601e2b3c503586" Sep 11 09:33:24.607020 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:24.606975 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8056006ee4745e25d36e1017f2e7607178023cbb5d41dbffd1601e2b3c503586"} err="failed to get container status \"8056006ee4745e25d36e1017f2e7607178023cbb5d41dbffd1601e2b3c503586\": rpc error: code = NotFound desc = could not find container \"8056006ee4745e25d36e1017f2e7607178023cbb5d41dbffd1601e2b3c503586\": container with ID starting with 8056006ee4745e25d36e1017f2e7607178023cbb5d41dbffd1601e2b3c503586 not found: ID does not exist" Sep 11 09:33:24.607020 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:24.606988 2575 scope.go:117] "RemoveContainer" containerID="5d91e016dda1d8a3a7236f31bd50a2b4451fb29bcbe1aae4b9dc4d6a178584e0" Sep 11 09:33:24.607247 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:33:24.607232 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d91e016dda1d8a3a7236f31bd50a2b4451fb29bcbe1aae4b9dc4d6a178584e0\": container with ID starting with 5d91e016dda1d8a3a7236f31bd50a2b4451fb29bcbe1aae4b9dc4d6a178584e0 not found: ID does not exist" containerID="5d91e016dda1d8a3a7236f31bd50a2b4451fb29bcbe1aae4b9dc4d6a178584e0" Sep 11 09:33:24.607287 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:24.607253 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d91e016dda1d8a3a7236f31bd50a2b4451fb29bcbe1aae4b9dc4d6a178584e0"} err="failed to get container status \"5d91e016dda1d8a3a7236f31bd50a2b4451fb29bcbe1aae4b9dc4d6a178584e0\": rpc error: code = NotFound desc = could not find container \"5d91e016dda1d8a3a7236f31bd50a2b4451fb29bcbe1aae4b9dc4d6a178584e0\": container with ID starting with 5d91e016dda1d8a3a7236f31bd50a2b4451fb29bcbe1aae4b9dc4d6a178584e0 not found: ID does not exist" Sep 11 09:33:24.607287 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:24.607266 2575 scope.go:117] "RemoveContainer" containerID="ec18b69b8686f5e1497e2d4d291c752e608497e44e1d8a381399fb11405693ca" Sep 11 09:33:24.614117 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:24.614093 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-6977969646-mbzdm"] Sep 11 09:33:24.615379 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:24.615360 2575 scope.go:117] "RemoveContainer" containerID="6cccee43b082a2c0cc1dedad7f6acb8249ee3fb5ed4828e1c64f99d3deaab6c8" Sep 11 09:33:24.618841 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:24.618819 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-6977969646-mbzdm"] Sep 11 09:33:24.624031 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:24.624013 2575 scope.go:117] "RemoveContainer" containerID="ef135d6cde0be1f0e6a2fbb7f3f0891af7c70d2fe3d3b62ac0d5c8e82cf2ecee" Sep 11 09:33:24.631457 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:24.631437 2575 scope.go:117] "RemoveContainer" containerID="69c07811a51693f7bd13b079c35fb27400805ea39f51f278c230475796032c5d" Sep 11 09:33:24.638596 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:24.638578 2575 scope.go:117] "RemoveContainer" containerID="ec18b69b8686f5e1497e2d4d291c752e608497e44e1d8a381399fb11405693ca" Sep 11 09:33:24.638879 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:33:24.638853 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec18b69b8686f5e1497e2d4d291c752e608497e44e1d8a381399fb11405693ca\": container with ID starting with ec18b69b8686f5e1497e2d4d291c752e608497e44e1d8a381399fb11405693ca not found: ID does not exist" containerID="ec18b69b8686f5e1497e2d4d291c752e608497e44e1d8a381399fb11405693ca" Sep 11 09:33:24.638944 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:24.638896 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec18b69b8686f5e1497e2d4d291c752e608497e44e1d8a381399fb11405693ca"} err="failed to get container status \"ec18b69b8686f5e1497e2d4d291c752e608497e44e1d8a381399fb11405693ca\": rpc error: code = NotFound desc = could not find container \"ec18b69b8686f5e1497e2d4d291c752e608497e44e1d8a381399fb11405693ca\": container with ID starting with ec18b69b8686f5e1497e2d4d291c752e608497e44e1d8a381399fb11405693ca not found: ID does not exist" Sep 11 09:33:24.638944 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:24.638916 2575 scope.go:117] "RemoveContainer" containerID="6cccee43b082a2c0cc1dedad7f6acb8249ee3fb5ed4828e1c64f99d3deaab6c8" Sep 11 09:33:24.639197 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:33:24.639182 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cccee43b082a2c0cc1dedad7f6acb8249ee3fb5ed4828e1c64f99d3deaab6c8\": container with ID starting with 6cccee43b082a2c0cc1dedad7f6acb8249ee3fb5ed4828e1c64f99d3deaab6c8 not found: ID does not exist" containerID="6cccee43b082a2c0cc1dedad7f6acb8249ee3fb5ed4828e1c64f99d3deaab6c8" Sep 11 09:33:24.639247 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:24.639201 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cccee43b082a2c0cc1dedad7f6acb8249ee3fb5ed4828e1c64f99d3deaab6c8"} err="failed to get container status \"6cccee43b082a2c0cc1dedad7f6acb8249ee3fb5ed4828e1c64f99d3deaab6c8\": rpc error: code = NotFound desc = could not find container \"6cccee43b082a2c0cc1dedad7f6acb8249ee3fb5ed4828e1c64f99d3deaab6c8\": container with ID starting with 6cccee43b082a2c0cc1dedad7f6acb8249ee3fb5ed4828e1c64f99d3deaab6c8 not found: ID does not exist" Sep 11 09:33:24.639247 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:24.639213 2575 scope.go:117] "RemoveContainer" containerID="ef135d6cde0be1f0e6a2fbb7f3f0891af7c70d2fe3d3b62ac0d5c8e82cf2ecee" Sep 11 09:33:24.639440 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:33:24.639425 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef135d6cde0be1f0e6a2fbb7f3f0891af7c70d2fe3d3b62ac0d5c8e82cf2ecee\": container with ID starting with ef135d6cde0be1f0e6a2fbb7f3f0891af7c70d2fe3d3b62ac0d5c8e82cf2ecee not found: ID does not exist" containerID="ef135d6cde0be1f0e6a2fbb7f3f0891af7c70d2fe3d3b62ac0d5c8e82cf2ecee" Sep 11 09:33:24.639502 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:24.639444 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef135d6cde0be1f0e6a2fbb7f3f0891af7c70d2fe3d3b62ac0d5c8e82cf2ecee"} err="failed to get container status \"ef135d6cde0be1f0e6a2fbb7f3f0891af7c70d2fe3d3b62ac0d5c8e82cf2ecee\": rpc error: code = NotFound desc = could not find container \"ef135d6cde0be1f0e6a2fbb7f3f0891af7c70d2fe3d3b62ac0d5c8e82cf2ecee\": container with ID starting with ef135d6cde0be1f0e6a2fbb7f3f0891af7c70d2fe3d3b62ac0d5c8e82cf2ecee not found: ID does not exist" Sep 11 09:33:24.639502 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:24.639455 2575 scope.go:117] "RemoveContainer" containerID="69c07811a51693f7bd13b079c35fb27400805ea39f51f278c230475796032c5d" Sep 11 09:33:24.639715 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:33:24.639698 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69c07811a51693f7bd13b079c35fb27400805ea39f51f278c230475796032c5d\": container with ID starting with 69c07811a51693f7bd13b079c35fb27400805ea39f51f278c230475796032c5d not found: ID does not exist" containerID="69c07811a51693f7bd13b079c35fb27400805ea39f51f278c230475796032c5d" Sep 11 09:33:24.639771 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:24.639717 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69c07811a51693f7bd13b079c35fb27400805ea39f51f278c230475796032c5d"} err="failed to get container status \"69c07811a51693f7bd13b079c35fb27400805ea39f51f278c230475796032c5d\": rpc error: code = NotFound desc = could not find container \"69c07811a51693f7bd13b079c35fb27400805ea39f51f278c230475796032c5d\": container with ID starting with 69c07811a51693f7bd13b079c35fb27400805ea39f51f278c230475796032c5d not found: ID does not exist" Sep 11 09:33:25.214175 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:25.214128 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-credit-scoring-794b46d689-wkcxh" podUID="23702c44-e27d-4cea-9191-dc7e9bf07ae5" containerName="online" probeResult="failure" output="Get \"https://10.134.0.26:6567/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Sep 11 09:33:26.114189 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:26.114154 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="00d79233-d405-4abe-9e5d-e010536fc3d3" path="/var/lib/kubelet/pods/00d79233-d405-4abe-9e5d-e010536fc3d3/volumes" Sep 11 09:33:26.114695 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:26.114680 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="23702c44-e27d-4cea-9191-dc7e9bf07ae5" path="/var/lib/kubelet/pods/23702c44-e27d-4cea-9191-dc7e9bf07ae5/volumes" Sep 11 09:33:31.559629 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:31.559593 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-6kxkr"] Sep 11 09:33:31.560043 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:31.559917 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23702c44-e27d-4cea-9191-dc7e9bf07ae5" containerName="feast-init" Sep 11 09:33:31.560043 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:31.559929 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="23702c44-e27d-4cea-9191-dc7e9bf07ae5" containerName="feast-init" Sep 11 09:33:31.560043 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:31.559942 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23702c44-e27d-4cea-9191-dc7e9bf07ae5" containerName="registry" Sep 11 09:33:31.560043 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:31.559948 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="23702c44-e27d-4cea-9191-dc7e9bf07ae5" containerName="registry" Sep 11 09:33:31.560043 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:31.559957 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48f714a0-2a14-4a38-a3c6-889d72233088" containerName="redis" Sep 11 09:33:31.560043 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:31.559962 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="48f714a0-2a14-4a38-a3c6-889d72233088" containerName="redis" Sep 11 09:33:31.560043 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:31.559971 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23702c44-e27d-4cea-9191-dc7e9bf07ae5" containerName="feast-apply" Sep 11 09:33:31.560043 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:31.559977 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="23702c44-e27d-4cea-9191-dc7e9bf07ae5" containerName="feast-apply" Sep 11 09:33:31.560043 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:31.559982 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00d79233-d405-4abe-9e5d-e010536fc3d3" containerName="registry" Sep 11 09:33:31.560043 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:31.559990 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="00d79233-d405-4abe-9e5d-e010536fc3d3" containerName="registry" Sep 11 09:33:31.560043 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:31.560002 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00d79233-d405-4abe-9e5d-e010536fc3d3" containerName="feast-apply" Sep 11 09:33:31.560043 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:31.560009 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="00d79233-d405-4abe-9e5d-e010536fc3d3" containerName="feast-apply" Sep 11 09:33:31.560043 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:31.560017 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00d79233-d405-4abe-9e5d-e010536fc3d3" containerName="online" Sep 11 09:33:31.560043 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:31.560022 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="00d79233-d405-4abe-9e5d-e010536fc3d3" containerName="online" Sep 11 09:33:31.560043 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:31.560028 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00d79233-d405-4abe-9e5d-e010536fc3d3" containerName="feast-init" Sep 11 09:33:31.560043 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:31.560034 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="00d79233-d405-4abe-9e5d-e010536fc3d3" containerName="feast-init" Sep 11 09:33:31.560043 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:31.560041 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23702c44-e27d-4cea-9191-dc7e9bf07ae5" containerName="online" Sep 11 09:33:31.560043 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:31.560051 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="23702c44-e27d-4cea-9191-dc7e9bf07ae5" containerName="online" Sep 11 09:33:31.560694 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:31.560059 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1a0a26d-6776-464e-9f1a-75dc8761f955" containerName="postgres" Sep 11 09:33:31.560694 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:31.560064 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1a0a26d-6776-464e-9f1a-75dc8761f955" containerName="postgres" Sep 11 09:33:31.560694 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:31.560104 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="23702c44-e27d-4cea-9191-dc7e9bf07ae5" containerName="online" Sep 11 09:33:31.560694 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:31.560110 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="00d79233-d405-4abe-9e5d-e010536fc3d3" containerName="registry" Sep 11 09:33:31.560694 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:31.560116 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="48f714a0-2a14-4a38-a3c6-889d72233088" containerName="redis" Sep 11 09:33:31.560694 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:31.560122 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="23702c44-e27d-4cea-9191-dc7e9bf07ae5" containerName="registry" Sep 11 09:33:31.560694 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:31.560127 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="00d79233-d405-4abe-9e5d-e010536fc3d3" containerName="online" Sep 11 09:33:31.560694 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:31.560133 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1a0a26d-6776-464e-9f1a-75dc8761f955" containerName="postgres" Sep 11 09:33:31.565230 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:31.565211 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6kxkr" Sep 11 09:33:31.568228 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:31.568205 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-offline-tls\"" Sep 11 09:33:31.568377 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:31.568358 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\"" Sep 11 09:33:31.568444 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:31.568428 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\"" Sep 11 09:33:31.568536 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:31.568443 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-online-tls\"" Sep 11 09:33:31.568536 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:31.568489 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-registry-tls\"" Sep 11 09:33:31.568681 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:31.568667 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-ui-tls\"" Sep 11 09:33:31.569171 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:31.569154 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-7f8ft\"" Sep 11 09:33:31.579055 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:31.579020 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-6kxkr"] Sep 11 09:33:31.722739 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:31.722698 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/f3b318f8-393b-4936-abca-27b73ea7c3bd-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-6kxkr\" (UID: \"f3b318f8-393b-4936-abca-27b73ea7c3bd\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6kxkr" Sep 11 09:33:31.722739 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:31.722737 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/f3b318f8-393b-4936-abca-27b73ea7c3bd-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-6kxkr\" (UID: \"f3b318f8-393b-4936-abca-27b73ea7c3bd\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6kxkr" Sep 11 09:33:31.722959 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:31.722757 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/f3b318f8-393b-4936-abca-27b73ea7c3bd-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-6kxkr\" (UID: \"f3b318f8-393b-4936-abca-27b73ea7c3bd\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6kxkr" Sep 11 09:33:31.722959 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:31.722854 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nf5lx\" (UniqueName: \"kubernetes.io/projected/f3b318f8-393b-4936-abca-27b73ea7c3bd-kube-api-access-nf5lx\") pod \"feast-simple-feast-setup-57576b8fcf-6kxkr\" (UID: \"f3b318f8-393b-4936-abca-27b73ea7c3bd\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6kxkr" Sep 11 09:33:31.722959 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:31.722916 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/f3b318f8-393b-4936-abca-27b73ea7c3bd-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-6kxkr\" (UID: \"f3b318f8-393b-4936-abca-27b73ea7c3bd\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6kxkr" Sep 11 09:33:31.722959 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:31.722954 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/f3b318f8-393b-4936-abca-27b73ea7c3bd-feast-data\") pod \"feast-simple-feast-setup-57576b8fcf-6kxkr\" (UID: \"f3b318f8-393b-4936-abca-27b73ea7c3bd\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6kxkr" Sep 11 09:33:31.824303 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:31.824215 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/f3b318f8-393b-4936-abca-27b73ea7c3bd-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-6kxkr\" (UID: \"f3b318f8-393b-4936-abca-27b73ea7c3bd\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6kxkr" Sep 11 09:33:31.824303 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:31.824259 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/f3b318f8-393b-4936-abca-27b73ea7c3bd-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-6kxkr\" (UID: \"f3b318f8-393b-4936-abca-27b73ea7c3bd\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6kxkr" Sep 11 09:33:31.824303 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:31.824276 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/f3b318f8-393b-4936-abca-27b73ea7c3bd-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-6kxkr\" (UID: \"f3b318f8-393b-4936-abca-27b73ea7c3bd\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6kxkr" Sep 11 09:33:31.824303 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:31.824298 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nf5lx\" (UniqueName: \"kubernetes.io/projected/f3b318f8-393b-4936-abca-27b73ea7c3bd-kube-api-access-nf5lx\") pod \"feast-simple-feast-setup-57576b8fcf-6kxkr\" (UID: \"f3b318f8-393b-4936-abca-27b73ea7c3bd\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6kxkr" Sep 11 09:33:31.824643 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:31.824334 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/f3b318f8-393b-4936-abca-27b73ea7c3bd-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-6kxkr\" (UID: \"f3b318f8-393b-4936-abca-27b73ea7c3bd\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6kxkr" Sep 11 09:33:31.824643 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:31.824361 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/f3b318f8-393b-4936-abca-27b73ea7c3bd-feast-data\") pod \"feast-simple-feast-setup-57576b8fcf-6kxkr\" (UID: \"f3b318f8-393b-4936-abca-27b73ea7c3bd\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6kxkr" Sep 11 09:33:31.824817 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:31.824786 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/f3b318f8-393b-4936-abca-27b73ea7c3bd-feast-data\") pod \"feast-simple-feast-setup-57576b8fcf-6kxkr\" (UID: \"f3b318f8-393b-4936-abca-27b73ea7c3bd\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6kxkr" Sep 11 09:33:31.826828 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:31.826797 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/f3b318f8-393b-4936-abca-27b73ea7c3bd-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-6kxkr\" (UID: \"f3b318f8-393b-4936-abca-27b73ea7c3bd\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6kxkr" Sep 11 09:33:31.826924 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:31.826837 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/f3b318f8-393b-4936-abca-27b73ea7c3bd-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-6kxkr\" (UID: \"f3b318f8-393b-4936-abca-27b73ea7c3bd\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6kxkr" Sep 11 09:33:31.827046 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:31.827014 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/f3b318f8-393b-4936-abca-27b73ea7c3bd-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-6kxkr\" (UID: \"f3b318f8-393b-4936-abca-27b73ea7c3bd\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6kxkr" Sep 11 09:33:31.827085 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:31.827014 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/f3b318f8-393b-4936-abca-27b73ea7c3bd-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-6kxkr\" (UID: \"f3b318f8-393b-4936-abca-27b73ea7c3bd\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6kxkr" Sep 11 09:33:31.833383 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:31.833363 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nf5lx\" (UniqueName: \"kubernetes.io/projected/f3b318f8-393b-4936-abca-27b73ea7c3bd-kube-api-access-nf5lx\") pod \"feast-simple-feast-setup-57576b8fcf-6kxkr\" (UID: \"f3b318f8-393b-4936-abca-27b73ea7c3bd\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6kxkr" Sep 11 09:33:31.887835 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:31.887796 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6kxkr" Sep 11 09:33:32.170711 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:32.170503 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-6kxkr"] Sep 11 09:33:32.173658 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:33:32.173631 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf3b318f8_393b_4936_abca_27b73ea7c3bd.slice/crio-78b194f2c3906652f2db98147ac4c986fc2a6256f3308229bb1847ffaf57099f WatchSource:0}: Error finding container 78b194f2c3906652f2db98147ac4c986fc2a6256f3308229bb1847ffaf57099f: Status 404 returned error can't find the container with id 78b194f2c3906652f2db98147ac4c986fc2a6256f3308229bb1847ffaf57099f Sep 11 09:33:32.175730 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:32.175707 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 09:33:32.615160 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:32.615124 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6kxkr" event={"ID":"f3b318f8-393b-4936-abca-27b73ea7c3bd","Type":"ContainerStarted","Data":"78b194f2c3906652f2db98147ac4c986fc2a6256f3308229bb1847ffaf57099f"} Sep 11 09:33:45.686753 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:45.686718 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6kxkr" event={"ID":"f3b318f8-393b-4936-abca-27b73ea7c3bd","Type":"ContainerStarted","Data":"0272785e0b9a5077e04698c65d3f732ee2714d64fcbc6e6730e4e6afd6a2fe6e"} Sep 11 09:33:50.708513 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:50.708386 2575 generic.go:356] "Generic (PLEG): container finished" podID="f3b318f8-393b-4936-abca-27b73ea7c3bd" containerID="0272785e0b9a5077e04698c65d3f732ee2714d64fcbc6e6730e4e6afd6a2fe6e" exitCode=0 Sep 11 09:33:50.709048 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:50.708489 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6kxkr" event={"ID":"f3b318f8-393b-4936-abca-27b73ea7c3bd","Type":"ContainerDied","Data":"0272785e0b9a5077e04698c65d3f732ee2714d64fcbc6e6730e4e6afd6a2fe6e"} Sep 11 09:33:51.715615 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:51.715568 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6kxkr" event={"ID":"f3b318f8-393b-4936-abca-27b73ea7c3bd","Type":"ContainerStarted","Data":"e82320b633535a438cb89ee3fe77af15dad77fb387d94275a145728475cf7bd6"} Sep 11 09:33:56.737592 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:56.737562 2575 generic.go:356] "Generic (PLEG): container finished" podID="f3b318f8-393b-4936-abca-27b73ea7c3bd" containerID="e82320b633535a438cb89ee3fe77af15dad77fb387d94275a145728475cf7bd6" exitCode=0 Sep 11 09:33:56.737951 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:56.737628 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6kxkr" event={"ID":"f3b318f8-393b-4936-abca-27b73ea7c3bd","Type":"ContainerDied","Data":"e82320b633535a438cb89ee3fe77af15dad77fb387d94275a145728475cf7bd6"} Sep 11 09:33:57.748627 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:57.748578 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6kxkr" event={"ID":"f3b318f8-393b-4936-abca-27b73ea7c3bd","Type":"ContainerStarted","Data":"e09dea3354358b0a7426347c9ddd56f1bb26e9a101842da96640f1b581d6d138"} Sep 11 09:33:57.749111 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:57.748647 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6kxkr" event={"ID":"f3b318f8-393b-4936-abca-27b73ea7c3bd","Type":"ContainerStarted","Data":"cabc571b0914a05887a2721ebc48b73ac1faeec8c27001a503e93f06dbd8be5e"} Sep 11 09:33:57.749111 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:57.748673 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6kxkr" event={"ID":"f3b318f8-393b-4936-abca-27b73ea7c3bd","Type":"ContainerStarted","Data":"499c96b9e6799b23ca397ba1bccc1ac4142c0f73f778124910cba5eef676b0a9"} Sep 11 09:33:57.749111 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:57.748692 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6kxkr" event={"ID":"f3b318f8-393b-4936-abca-27b73ea7c3bd","Type":"ContainerStarted","Data":"7cb5016ae8b178f1c7eb31e740932c8b65ecfcebdbb73a70a265b819efe8e91f"} Sep 11 09:33:57.776300 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:57.776233 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6kxkr" podStartSLOduration=14.125494282 podStartE2EDuration="26.776209897s" totalImagesPullingTime="12.650715615s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 09:33:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 09:33:57.774556049 +0000 UTC m=+1254.115166282" watchObservedRunningTime="2026-09-11 09:33:57.776209897 +0000 UTC m=+1254.116820107" Sep 11 09:33:58.888128 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:58.888091 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6kxkr" Sep 11 09:33:58.888613 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:58.888158 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6kxkr" Sep 11 09:33:58.888613 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:58.888179 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6kxkr" Sep 11 09:33:58.888613 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:58.888203 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6kxkr" Sep 11 09:33:58.890336 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:58.890154 2575 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6kxkr" podUID="f3b318f8-393b-4936-abca-27b73ea7c3bd" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.28:8816: connect: connection refused" Sep 11 09:33:58.890336 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:58.890183 2575 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6kxkr" podUID="f3b318f8-393b-4936-abca-27b73ea7c3bd" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.28:8443: connect: connection refused" Sep 11 09:33:58.890336 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:58.890203 2575 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6kxkr" podUID="f3b318f8-393b-4936-abca-27b73ea7c3bd" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.28:6571: connect: connection refused" Sep 11 09:33:58.890336 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:33:58.890259 2575 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6kxkr" podUID="f3b318f8-393b-4936-abca-27b73ea7c3bd" containerName="online" probeResult="failure" output="Get \"https://10.134.0.28:6567/health\": dial tcp 10.134.0.28:6567: connect: connection refused" Sep 11 09:34:01.888672 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:34:01.888368 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6kxkr" Sep 11 09:34:01.889274 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:34:01.889238 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6kxkr" Sep 11 09:34:01.889274 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:34:01.889276 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6kxkr" Sep 11 09:34:01.889460 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:34:01.888718 2575 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6kxkr" podUID="f3b318f8-393b-4936-abca-27b73ea7c3bd" containerName="online" probeResult="failure" output="Get \"https://10.134.0.28:6567/health\": dial tcp 10.134.0.28:6567: connect: connection refused" Sep 11 09:34:01.889460 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:34:01.888708 2575 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6kxkr" podUID="f3b318f8-393b-4936-abca-27b73ea7c3bd" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.28:8816: connect: connection refused" Sep 11 09:34:01.889460 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:34:01.888703 2575 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6kxkr" podUID="f3b318f8-393b-4936-abca-27b73ea7c3bd" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.28:8443: connect: connection refused" Sep 11 09:34:01.889460 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:34:01.888745 2575 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6kxkr" podUID="f3b318f8-393b-4936-abca-27b73ea7c3bd" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.28:6571: connect: connection refused" Sep 11 09:34:01.889733 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:34:01.889713 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6kxkr" Sep 11 09:34:04.889076 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:34:04.889029 2575 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6kxkr" podUID="f3b318f8-393b-4936-abca-27b73ea7c3bd" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.28:8443: connect: connection refused" Sep 11 09:34:04.889779 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:34:04.889754 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6kxkr" Sep 11 09:34:04.890375 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:34:04.890355 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6kxkr" Sep 11 09:34:04.890622 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:34:04.890604 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6kxkr" Sep 11 09:34:04.959655 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:34:04.959623 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6kxkr" Sep 11 09:34:05.793994 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:34:05.793962 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6kxkr" Sep 11 09:34:05.796595 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:34:05.796575 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6kxkr" Sep 11 09:34:07.888748 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:34:07.888712 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6kxkr" Sep 11 09:34:08.806463 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:34:08.806435 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6kxkr" Sep 11 09:35:11.934719 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:35:11.934681 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-6kxkr"] Sep 11 09:35:11.935330 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:35:11.934942 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6kxkr" podUID="f3b318f8-393b-4936-abca-27b73ea7c3bd" containerName="registry" containerID="cri-o://7cb5016ae8b178f1c7eb31e740932c8b65ecfcebdbb73a70a265b819efe8e91f" gracePeriod=30 Sep 11 09:35:11.935330 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:35:11.935010 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6kxkr" podUID="f3b318f8-393b-4936-abca-27b73ea7c3bd" containerName="online" containerID="cri-o://499c96b9e6799b23ca397ba1bccc1ac4142c0f73f778124910cba5eef676b0a9" gracePeriod=30 Sep 11 09:35:11.935330 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:35:11.935050 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6kxkr" podUID="f3b318f8-393b-4936-abca-27b73ea7c3bd" containerName="ui" containerID="cri-o://e09dea3354358b0a7426347c9ddd56f1bb26e9a101842da96640f1b581d6d138" gracePeriod=30 Sep 11 09:35:11.935330 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:35:11.935010 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6kxkr" podUID="f3b318f8-393b-4936-abca-27b73ea7c3bd" containerName="offline" containerID="cri-o://cabc571b0914a05887a2721ebc48b73ac1faeec8c27001a503e93f06dbd8be5e" gracePeriod=30 Sep 11 09:35:12.336528 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:35:12.336463 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-s4zzq"] Sep 11 09:35:12.339973 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:35:12.339946 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-s4zzq" Sep 11 09:35:12.343605 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:35:12.343579 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-plqp2\"" Sep 11 09:35:12.369565 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:35:12.369532 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-s4zzq"] Sep 11 09:35:12.422060 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:35:12.422024 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/48c3c3a9-66e3-4aab-b200-71340090b4b8-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-s4zzq\" (UID: \"48c3c3a9-66e3-4aab-b200-71340090b4b8\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-s4zzq" Sep 11 09:35:12.422060 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:35:12.422063 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w94gb\" (UniqueName: \"kubernetes.io/projected/48c3c3a9-66e3-4aab-b200-71340090b4b8-kube-api-access-w94gb\") pod \"feast-simple-feast-setup-57576b8fcf-s4zzq\" (UID: \"48c3c3a9-66e3-4aab-b200-71340090b4b8\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-s4zzq" Sep 11 09:35:12.422336 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:35:12.422090 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/48c3c3a9-66e3-4aab-b200-71340090b4b8-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-s4zzq\" (UID: \"48c3c3a9-66e3-4aab-b200-71340090b4b8\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-s4zzq" Sep 11 09:35:12.422336 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:35:12.422208 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/48c3c3a9-66e3-4aab-b200-71340090b4b8-feast-data\") pod \"feast-simple-feast-setup-57576b8fcf-s4zzq\" (UID: \"48c3c3a9-66e3-4aab-b200-71340090b4b8\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-s4zzq" Sep 11 09:35:12.422336 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:35:12.422272 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/48c3c3a9-66e3-4aab-b200-71340090b4b8-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-s4zzq\" (UID: \"48c3c3a9-66e3-4aab-b200-71340090b4b8\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-s4zzq" Sep 11 09:35:12.422514 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:35:12.422398 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/48c3c3a9-66e3-4aab-b200-71340090b4b8-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-s4zzq\" (UID: \"48c3c3a9-66e3-4aab-b200-71340090b4b8\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-s4zzq" Sep 11 09:35:12.523305 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:35:12.523259 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/48c3c3a9-66e3-4aab-b200-71340090b4b8-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-s4zzq\" (UID: \"48c3c3a9-66e3-4aab-b200-71340090b4b8\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-s4zzq" Sep 11 09:35:12.523305 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:35:12.523302 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/48c3c3a9-66e3-4aab-b200-71340090b4b8-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-s4zzq\" (UID: \"48c3c3a9-66e3-4aab-b200-71340090b4b8\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-s4zzq" Sep 11 09:35:12.523602 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:35:12.523327 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w94gb\" (UniqueName: \"kubernetes.io/projected/48c3c3a9-66e3-4aab-b200-71340090b4b8-kube-api-access-w94gb\") pod \"feast-simple-feast-setup-57576b8fcf-s4zzq\" (UID: \"48c3c3a9-66e3-4aab-b200-71340090b4b8\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-s4zzq" Sep 11 09:35:12.523602 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:35:12.523361 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/48c3c3a9-66e3-4aab-b200-71340090b4b8-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-s4zzq\" (UID: \"48c3c3a9-66e3-4aab-b200-71340090b4b8\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-s4zzq" Sep 11 09:35:12.523602 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:35:12.523396 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/48c3c3a9-66e3-4aab-b200-71340090b4b8-feast-data\") pod \"feast-simple-feast-setup-57576b8fcf-s4zzq\" (UID: \"48c3c3a9-66e3-4aab-b200-71340090b4b8\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-s4zzq" Sep 11 09:35:12.523602 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:35:12.523482 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/48c3c3a9-66e3-4aab-b200-71340090b4b8-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-s4zzq\" (UID: \"48c3c3a9-66e3-4aab-b200-71340090b4b8\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-s4zzq" Sep 11 09:35:12.523602 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:35:12.523581 2575 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-offline-tls: secret "feast-simple-feast-setup-offline-tls" not found Sep 11 09:35:12.523856 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:35:12.523652 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48c3c3a9-66e3-4aab-b200-71340090b4b8-offline-tls podName:48c3c3a9-66e3-4aab-b200-71340090b4b8 nodeName:}" failed. No retries permitted until 2026-09-11 09:35:13.023631903 +0000 UTC m=+1329.364242087 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/48c3c3a9-66e3-4aab-b200-71340090b4b8-offline-tls") pod "feast-simple-feast-setup-57576b8fcf-s4zzq" (UID: "48c3c3a9-66e3-4aab-b200-71340090b4b8") : secret "feast-simple-feast-setup-offline-tls" not found Sep 11 09:35:12.524211 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:35:12.524183 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/48c3c3a9-66e3-4aab-b200-71340090b4b8-feast-data\") pod \"feast-simple-feast-setup-57576b8fcf-s4zzq\" (UID: \"48c3c3a9-66e3-4aab-b200-71340090b4b8\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-s4zzq" Sep 11 09:35:12.525945 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:35:12.525923 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/48c3c3a9-66e3-4aab-b200-71340090b4b8-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-s4zzq\" (UID: \"48c3c3a9-66e3-4aab-b200-71340090b4b8\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-s4zzq" Sep 11 09:35:12.526076 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:35:12.526055 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/48c3c3a9-66e3-4aab-b200-71340090b4b8-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-s4zzq\" (UID: \"48c3c3a9-66e3-4aab-b200-71340090b4b8\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-s4zzq" Sep 11 09:35:12.526202 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:35:12.526181 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/48c3c3a9-66e3-4aab-b200-71340090b4b8-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-s4zzq\" (UID: \"48c3c3a9-66e3-4aab-b200-71340090b4b8\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-s4zzq" Sep 11 09:35:12.536312 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:35:12.536278 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w94gb\" (UniqueName: \"kubernetes.io/projected/48c3c3a9-66e3-4aab-b200-71340090b4b8-kube-api-access-w94gb\") pod \"feast-simple-feast-setup-57576b8fcf-s4zzq\" (UID: \"48c3c3a9-66e3-4aab-b200-71340090b4b8\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-s4zzq" Sep 11 09:35:13.028681 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:35:13.028651 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/48c3c3a9-66e3-4aab-b200-71340090b4b8-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-s4zzq\" (UID: \"48c3c3a9-66e3-4aab-b200-71340090b4b8\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-s4zzq" Sep 11 09:35:13.029040 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:35:13.028783 2575 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-offline-tls: secret "feast-simple-feast-setup-offline-tls" not found Sep 11 09:35:13.029040 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:35:13.028840 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48c3c3a9-66e3-4aab-b200-71340090b4b8-offline-tls podName:48c3c3a9-66e3-4aab-b200-71340090b4b8 nodeName:}" failed. No retries permitted until 2026-09-11 09:35:14.028824369 +0000 UTC m=+1330.369434553 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/48c3c3a9-66e3-4aab-b200-71340090b4b8-offline-tls") pod "feast-simple-feast-setup-57576b8fcf-s4zzq" (UID: "48c3c3a9-66e3-4aab-b200-71340090b4b8") : secret "feast-simple-feast-setup-offline-tls" not found Sep 11 09:35:13.068197 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:35:13.068161 2575 generic.go:356] "Generic (PLEG): container finished" podID="f3b318f8-393b-4936-abca-27b73ea7c3bd" containerID="e09dea3354358b0a7426347c9ddd56f1bb26e9a101842da96640f1b581d6d138" exitCode=0 Sep 11 09:35:13.068197 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:35:13.068188 2575 generic.go:356] "Generic (PLEG): container finished" podID="f3b318f8-393b-4936-abca-27b73ea7c3bd" containerID="499c96b9e6799b23ca397ba1bccc1ac4142c0f73f778124910cba5eef676b0a9" exitCode=0 Sep 11 09:35:13.068398 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:35:13.068232 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6kxkr" event={"ID":"f3b318f8-393b-4936-abca-27b73ea7c3bd","Type":"ContainerDied","Data":"e09dea3354358b0a7426347c9ddd56f1bb26e9a101842da96640f1b581d6d138"} Sep 11 09:35:13.068398 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:35:13.068273 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6kxkr" event={"ID":"f3b318f8-393b-4936-abca-27b73ea7c3bd","Type":"ContainerDied","Data":"499c96b9e6799b23ca397ba1bccc1ac4142c0f73f778124910cba5eef676b0a9"} Sep 11 09:35:14.035413 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:35:14.035363 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/48c3c3a9-66e3-4aab-b200-71340090b4b8-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-s4zzq\" (UID: \"48c3c3a9-66e3-4aab-b200-71340090b4b8\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-s4zzq" Sep 11 09:35:14.037865 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:35:14.037826 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/48c3c3a9-66e3-4aab-b200-71340090b4b8-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-s4zzq\" (UID: \"48c3c3a9-66e3-4aab-b200-71340090b4b8\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-s4zzq" Sep 11 09:35:14.151040 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:35:14.150989 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-s4zzq" Sep 11 09:35:14.432945 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:35:14.432915 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-s4zzq"] Sep 11 09:35:14.433812 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:35:14.433782 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod48c3c3a9_66e3_4aab_b200_71340090b4b8.slice/crio-6d49aada29243b4b36719596c742cc4fcf2528974d954a297d688cfaa6d344c4 WatchSource:0}: Error finding container 6d49aada29243b4b36719596c742cc4fcf2528974d954a297d688cfaa6d344c4: Status 404 returned error can't find the container with id 6d49aada29243b4b36719596c742cc4fcf2528974d954a297d688cfaa6d344c4 Sep 11 09:35:15.078861 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:35:15.078826 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-s4zzq" event={"ID":"48c3c3a9-66e3-4aab-b200-71340090b4b8","Type":"ContainerStarted","Data":"4c9a874903e8ebf64de8abd43f164ab4f890a48727013220bf6c31b87c2fe21e"} Sep 11 09:35:15.079246 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:35:15.078870 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-s4zzq" event={"ID":"48c3c3a9-66e3-4aab-b200-71340090b4b8","Type":"ContainerStarted","Data":"6d49aada29243b4b36719596c742cc4fcf2528974d954a297d688cfaa6d344c4"} Sep 11 09:35:15.794095 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:35:15.794051 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6kxkr" podUID="f3b318f8-393b-4936-abca-27b73ea7c3bd" containerName="online" probeResult="failure" output="Get \"https://10.134.0.28:6567/health\": dial tcp 10.134.0.28:6567: connect: connection refused" Sep 11 09:35:18.806062 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:35:18.806005 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6kxkr" podUID="f3b318f8-393b-4936-abca-27b73ea7c3bd" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.28:8443: connect: connection refused" Sep 11 09:35:20.101871 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:35:20.101837 2575 generic.go:356] "Generic (PLEG): container finished" podID="48c3c3a9-66e3-4aab-b200-71340090b4b8" containerID="4c9a874903e8ebf64de8abd43f164ab4f890a48727013220bf6c31b87c2fe21e" exitCode=0 Sep 11 09:35:20.102260 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:35:20.101910 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-s4zzq" event={"ID":"48c3c3a9-66e3-4aab-b200-71340090b4b8","Type":"ContainerDied","Data":"4c9a874903e8ebf64de8abd43f164ab4f890a48727013220bf6c31b87c2fe21e"} Sep 11 09:35:21.108202 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:35:21.108163 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-s4zzq" event={"ID":"48c3c3a9-66e3-4aab-b200-71340090b4b8","Type":"ContainerStarted","Data":"770ad57418f3ef2befce556b6b2c6092f4a429554a03105257042d2ac6bbda1b"} Sep 11 09:35:25.793763 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:35:25.793714 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6kxkr" podUID="f3b318f8-393b-4936-abca-27b73ea7c3bd" containerName="online" probeResult="failure" output="Get \"https://10.134.0.28:6567/health\": dial tcp 10.134.0.28:6567: connect: connection refused" Sep 11 09:35:26.132743 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:35:26.132647 2575 generic.go:356] "Generic (PLEG): container finished" podID="48c3c3a9-66e3-4aab-b200-71340090b4b8" containerID="770ad57418f3ef2befce556b6b2c6092f4a429554a03105257042d2ac6bbda1b" exitCode=0 Sep 11 09:35:26.132743 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:35:26.132703 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-s4zzq" event={"ID":"48c3c3a9-66e3-4aab-b200-71340090b4b8","Type":"ContainerDied","Data":"770ad57418f3ef2befce556b6b2c6092f4a429554a03105257042d2ac6bbda1b"} Sep 11 09:35:27.143250 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:35:27.143205 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-s4zzq" event={"ID":"48c3c3a9-66e3-4aab-b200-71340090b4b8","Type":"ContainerStarted","Data":"48da851e4310d4c67d72e8b77ef2c414dcc4a229a60991d42914902ff355a86d"} Sep 11 09:35:27.143754 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:35:27.143272 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-s4zzq" event={"ID":"48c3c3a9-66e3-4aab-b200-71340090b4b8","Type":"ContainerStarted","Data":"9925a7a002dd1568283efc4501734e4eb97b7817a2e8de5ca766ab244f3de61f"} Sep 11 09:35:27.143754 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:35:27.143294 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-s4zzq" event={"ID":"48c3c3a9-66e3-4aab-b200-71340090b4b8","Type":"ContainerStarted","Data":"410510e71d23571cf5d6776763d6a50c98d09fb2baa9eab4a44d13564b3074c4"} Sep 11 09:35:27.143754 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:35:27.143313 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-s4zzq" event={"ID":"48c3c3a9-66e3-4aab-b200-71340090b4b8","Type":"ContainerStarted","Data":"8e75e4fb65cd7a3005f1a6d109d2e859ad49d82adf3b8f3e86920640bccedde7"} Sep 11 09:35:27.173353 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:35:27.173285 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-s4zzq" podStartSLOduration=15.173261856 podStartE2EDuration="15.173261856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 09:35:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 09:35:27.170026325 +0000 UTC m=+1343.510636544" watchObservedRunningTime="2026-09-11 09:35:27.173261856 +0000 UTC m=+1343.513872064" Sep 11 09:35:28.806672 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:35:28.806615 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6kxkr" podUID="f3b318f8-393b-4936-abca-27b73ea7c3bd" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.28:8443: connect: connection refused" Sep 11 09:35:29.151775 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:35:29.151678 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-s4zzq" Sep 11 09:35:29.151775 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:35:29.151739 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-s4zzq" Sep 11 09:35:29.151775 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:35:29.151758 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-s4zzq" Sep 11 09:35:29.151775 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:35:29.151777 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-s4zzq" Sep 11 09:35:29.153985 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:35:29.153936 2575 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-s4zzq" podUID="48c3c3a9-66e3-4aab-b200-71340090b4b8" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.29:8443: connect: connection refused" Sep 11 09:35:29.153985 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:35:29.153975 2575 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-s4zzq" podUID="48c3c3a9-66e3-4aab-b200-71340090b4b8" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.29:8816: connect: connection refused" Sep 11 09:35:29.154191 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:35:29.154017 2575 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-s4zzq" podUID="48c3c3a9-66e3-4aab-b200-71340090b4b8" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.29:6571: connect: connection refused" Sep 11 09:35:29.154191 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:35:29.153974 2575 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-s4zzq" podUID="48c3c3a9-66e3-4aab-b200-71340090b4b8" containerName="online" probeResult="failure" output="Get \"https://10.134.0.29:6567/health\": dial tcp 10.134.0.29:6567: connect: connection refused" Sep 11 09:35:32.152327 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:35:32.152276 2575 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-s4zzq" podUID="48c3c3a9-66e3-4aab-b200-71340090b4b8" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.29:8816: connect: connection refused" Sep 11 09:35:32.152851 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:35:32.152333 2575 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-s4zzq" podUID="48c3c3a9-66e3-4aab-b200-71340090b4b8" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.29:8443: connect: connection refused" Sep 11 09:35:32.152851 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:35:32.152287 2575 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-s4zzq" podUID="48c3c3a9-66e3-4aab-b200-71340090b4b8" containerName="online" probeResult="failure" output="Get \"https://10.134.0.29:6567/health\": dial tcp 10.134.0.29:6567: connect: connection refused" Sep 11 09:35:32.152851 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:35:32.152276 2575 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-s4zzq" podUID="48c3c3a9-66e3-4aab-b200-71340090b4b8" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.29:6571: connect: connection refused" Sep 11 09:35:34.151361 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:35:34.151319 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-s4zzq" Sep 11 09:35:34.151902 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:35:34.151376 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-s4zzq" Sep 11 09:35:34.151902 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:35:34.151392 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-s4zzq" Sep 11 09:35:34.151902 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:35:34.151404 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-s4zzq" Sep 11 09:35:35.152700 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:35:35.152664 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-s4zzq" Sep 11 09:35:35.153157 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:35:35.152860 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-s4zzq" Sep 11 09:35:35.153221 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:35:35.153164 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-s4zzq" Sep 11 09:35:35.153659 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:35:35.153631 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-s4zzq" Sep 11 09:35:35.153779 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:35:35.153709 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-s4zzq" Sep 11 09:35:35.153907 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:35:35.153886 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-s4zzq" Sep 11 09:35:35.158288 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:35:35.158271 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-s4zzq" Sep 11 09:35:35.200240 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:35:35.200216 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-s4zzq" Sep 11 09:35:35.794069 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:35:35.794026 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6kxkr" podUID="f3b318f8-393b-4936-abca-27b73ea7c3bd" containerName="online" probeResult="failure" output="Get \"https://10.134.0.28:6567/health\": dial tcp 10.134.0.28:6567: connect: connection refused" Sep 11 09:35:35.794277 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:35:35.794167 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6kxkr" Sep 11 09:35:38.806384 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:35:38.806341 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6kxkr" podUID="f3b318f8-393b-4936-abca-27b73ea7c3bd" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.28:8443: connect: connection refused" Sep 11 09:35:38.806785 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:35:38.806461 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6kxkr" Sep 11 09:35:42.229070 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:35:42.228988 2575 generic.go:356] "Generic (PLEG): container finished" podID="f3b318f8-393b-4936-abca-27b73ea7c3bd" containerID="cabc571b0914a05887a2721ebc48b73ac1faeec8c27001a503e93f06dbd8be5e" exitCode=137 Sep 11 09:35:42.229070 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:35:42.229011 2575 generic.go:356] "Generic (PLEG): container finished" podID="f3b318f8-393b-4936-abca-27b73ea7c3bd" containerID="7cb5016ae8b178f1c7eb31e740932c8b65ecfcebdbb73a70a265b819efe8e91f" exitCode=137 Sep 11 09:35:42.229070 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:35:42.229056 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6kxkr" event={"ID":"f3b318f8-393b-4936-abca-27b73ea7c3bd","Type":"ContainerDied","Data":"cabc571b0914a05887a2721ebc48b73ac1faeec8c27001a503e93f06dbd8be5e"} Sep 11 09:35:42.229499 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:35:42.229096 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6kxkr" event={"ID":"f3b318f8-393b-4936-abca-27b73ea7c3bd","Type":"ContainerDied","Data":"7cb5016ae8b178f1c7eb31e740932c8b65ecfcebdbb73a70a265b819efe8e91f"} Sep 11 09:35:42.583563 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:35:42.583535 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6kxkr" Sep 11 09:35:42.690529 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:35:42.690490 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nf5lx\" (UniqueName: \"kubernetes.io/projected/f3b318f8-393b-4936-abca-27b73ea7c3bd-kube-api-access-nf5lx\") pod \"f3b318f8-393b-4936-abca-27b73ea7c3bd\" (UID: \"f3b318f8-393b-4936-abca-27b73ea7c3bd\") " Sep 11 09:35:42.690729 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:35:42.690548 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/f3b318f8-393b-4936-abca-27b73ea7c3bd-ui-tls\") pod \"f3b318f8-393b-4936-abca-27b73ea7c3bd\" (UID: \"f3b318f8-393b-4936-abca-27b73ea7c3bd\") " Sep 11 09:35:42.690729 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:35:42.690584 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/f3b318f8-393b-4936-abca-27b73ea7c3bd-registry-tls\") pod \"f3b318f8-393b-4936-abca-27b73ea7c3bd\" (UID: \"f3b318f8-393b-4936-abca-27b73ea7c3bd\") " Sep 11 09:35:42.690729 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:35:42.690619 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/f3b318f8-393b-4936-abca-27b73ea7c3bd-online-tls\") pod \"f3b318f8-393b-4936-abca-27b73ea7c3bd\" (UID: \"f3b318f8-393b-4936-abca-27b73ea7c3bd\") " Sep 11 09:35:42.690729 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:35:42.690644 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/f3b318f8-393b-4936-abca-27b73ea7c3bd-feast-data\") pod \"f3b318f8-393b-4936-abca-27b73ea7c3bd\" (UID: \"f3b318f8-393b-4936-abca-27b73ea7c3bd\") " Sep 11 09:35:42.690729 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:35:42.690699 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/f3b318f8-393b-4936-abca-27b73ea7c3bd-offline-tls\") pod \"f3b318f8-393b-4936-abca-27b73ea7c3bd\" (UID: \"f3b318f8-393b-4936-abca-27b73ea7c3bd\") " Sep 11 09:35:42.691341 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:35:42.691226 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3b318f8-393b-4936-abca-27b73ea7c3bd-feast-data" (OuterVolumeSpecName: "feast-data") pod "f3b318f8-393b-4936-abca-27b73ea7c3bd" (UID: "f3b318f8-393b-4936-abca-27b73ea7c3bd"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 09:35:42.692982 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:35:42.692946 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3b318f8-393b-4936-abca-27b73ea7c3bd-online-tls" (OuterVolumeSpecName: "online-tls") pod "f3b318f8-393b-4936-abca-27b73ea7c3bd" (UID: "f3b318f8-393b-4936-abca-27b73ea7c3bd"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 09:35:42.693100 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:35:42.692993 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3b318f8-393b-4936-abca-27b73ea7c3bd-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "f3b318f8-393b-4936-abca-27b73ea7c3bd" (UID: "f3b318f8-393b-4936-abca-27b73ea7c3bd"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 09:35:42.693100 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:35:42.693009 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3b318f8-393b-4936-abca-27b73ea7c3bd-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "f3b318f8-393b-4936-abca-27b73ea7c3bd" (UID: "f3b318f8-393b-4936-abca-27b73ea7c3bd"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 09:35:42.693213 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:35:42.693141 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3b318f8-393b-4936-abca-27b73ea7c3bd-kube-api-access-nf5lx" (OuterVolumeSpecName: "kube-api-access-nf5lx") pod "f3b318f8-393b-4936-abca-27b73ea7c3bd" (UID: "f3b318f8-393b-4936-abca-27b73ea7c3bd"). InnerVolumeSpecName "kube-api-access-nf5lx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 09:35:42.693213 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:35:42.693150 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3b318f8-393b-4936-abca-27b73ea7c3bd-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "f3b318f8-393b-4936-abca-27b73ea7c3bd" (UID: "f3b318f8-393b-4936-abca-27b73ea7c3bd"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 09:35:42.792243 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:35:42.792210 2575 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/f3b318f8-393b-4936-abca-27b73ea7c3bd-registry-tls\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\"" Sep 11 09:35:42.792243 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:35:42.792237 2575 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/f3b318f8-393b-4936-abca-27b73ea7c3bd-online-tls\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\"" Sep 11 09:35:42.792243 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:35:42.792246 2575 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/f3b318f8-393b-4936-abca-27b73ea7c3bd-feast-data\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\"" Sep 11 09:35:42.792243 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:35:42.792253 2575 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/f3b318f8-393b-4936-abca-27b73ea7c3bd-offline-tls\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\"" Sep 11 09:35:42.792539 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:35:42.792262 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nf5lx\" (UniqueName: \"kubernetes.io/projected/f3b318f8-393b-4936-abca-27b73ea7c3bd-kube-api-access-nf5lx\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\"" Sep 11 09:35:42.792539 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:35:42.792271 2575 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/f3b318f8-393b-4936-abca-27b73ea7c3bd-ui-tls\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\"" Sep 11 09:35:43.235790 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:35:43.235713 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6kxkr" event={"ID":"f3b318f8-393b-4936-abca-27b73ea7c3bd","Type":"ContainerDied","Data":"78b194f2c3906652f2db98147ac4c986fc2a6256f3308229bb1847ffaf57099f"} Sep 11 09:35:43.235790 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:35:43.235767 2575 scope.go:117] "RemoveContainer" containerID="e09dea3354358b0a7426347c9ddd56f1bb26e9a101842da96640f1b581d6d138" Sep 11 09:35:43.236342 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:35:43.235726 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-6kxkr" Sep 11 09:35:43.244652 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:35:43.244636 2575 scope.go:117] "RemoveContainer" containerID="cabc571b0914a05887a2721ebc48b73ac1faeec8c27001a503e93f06dbd8be5e" Sep 11 09:35:43.252649 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:35:43.252632 2575 scope.go:117] "RemoveContainer" containerID="499c96b9e6799b23ca397ba1bccc1ac4142c0f73f778124910cba5eef676b0a9" Sep 11 09:35:43.259058 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:35:43.259034 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-6kxkr"] Sep 11 09:35:43.260622 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:35:43.260603 2575 scope.go:117] "RemoveContainer" containerID="7cb5016ae8b178f1c7eb31e740932c8b65ecfcebdbb73a70a265b819efe8e91f" Sep 11 09:35:43.264457 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:35:43.264436 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-6kxkr"] Sep 11 09:35:43.268136 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:35:43.268120 2575 scope.go:117] "RemoveContainer" containerID="e82320b633535a438cb89ee3fe77af15dad77fb387d94275a145728475cf7bd6" Sep 11 09:35:43.275277 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:35:43.275260 2575 scope.go:117] "RemoveContainer" containerID="0272785e0b9a5077e04698c65d3f732ee2714d64fcbc6e6730e4e6afd6a2fe6e" Sep 11 09:35:44.114071 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:35:44.114034 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3b318f8-393b-4936-abca-27b73ea7c3bd" path="/var/lib/kubelet/pods/f3b318f8-393b-4936-abca-27b73ea7c3bd/volumes" Sep 11 09:36:37.433240 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:36:37.433202 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7w7vs"] Sep 11 09:36:37.433773 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:36:37.433714 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3b318f8-393b-4936-abca-27b73ea7c3bd" containerName="ui" Sep 11 09:36:37.433773 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:36:37.433734 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3b318f8-393b-4936-abca-27b73ea7c3bd" containerName="ui" Sep 11 09:36:37.433773 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:36:37.433758 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3b318f8-393b-4936-abca-27b73ea7c3bd" containerName="online" Sep 11 09:36:37.433773 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:36:37.433768 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3b318f8-393b-4936-abca-27b73ea7c3bd" containerName="online" Sep 11 09:36:37.433906 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:36:37.433785 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3b318f8-393b-4936-abca-27b73ea7c3bd" containerName="offline" Sep 11 09:36:37.433906 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:36:37.433794 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3b318f8-393b-4936-abca-27b73ea7c3bd" containerName="offline" Sep 11 09:36:37.433906 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:36:37.433808 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3b318f8-393b-4936-abca-27b73ea7c3bd" containerName="feast-init" Sep 11 09:36:37.433906 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:36:37.433817 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3b318f8-393b-4936-abca-27b73ea7c3bd" containerName="feast-init" Sep 11 09:36:37.433906 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:36:37.433835 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3b318f8-393b-4936-abca-27b73ea7c3bd" containerName="registry" Sep 11 09:36:37.433906 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:36:37.433843 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3b318f8-393b-4936-abca-27b73ea7c3bd" containerName="registry" Sep 11 09:36:37.433906 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:36:37.433860 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3b318f8-393b-4936-abca-27b73ea7c3bd" containerName="feast-apply" Sep 11 09:36:37.433906 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:36:37.433871 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3b318f8-393b-4936-abca-27b73ea7c3bd" containerName="feast-apply" Sep 11 09:36:37.434143 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:36:37.433958 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3b318f8-393b-4936-abca-27b73ea7c3bd" containerName="ui" Sep 11 09:36:37.434143 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:36:37.433971 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3b318f8-393b-4936-abca-27b73ea7c3bd" containerName="registry" Sep 11 09:36:37.434143 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:36:37.433984 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3b318f8-393b-4936-abca-27b73ea7c3bd" containerName="online" Sep 11 09:36:37.434143 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:36:37.433994 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3b318f8-393b-4936-abca-27b73ea7c3bd" containerName="offline" Sep 11 09:36:37.437289 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:36:37.437272 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7w7vs" Sep 11 09:36:37.440682 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:36:37.440650 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-offline-tls\"" Sep 11 09:36:37.441298 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:36:37.441269 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-dockercfg-gf2p9\"" Sep 11 09:36:37.441521 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:36:37.441269 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-client-ca\"" Sep 11 09:36:37.441606 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:36:37.441502 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"openshift-service-ca.crt\"" Sep 11 09:36:37.441834 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:36:37.441813 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"kube-root-ca.crt\"" Sep 11 09:36:37.441911 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:36:37.441851 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-online-tls\"" Sep 11 09:36:37.446486 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:36:37.442890 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-ui-tls\"" Sep 11 09:36:37.457537 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:36:37.457514 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7w7vs"] Sep 11 09:36:37.567199 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:36:37.567167 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/acfd9c8e-51bf-426a-bbd2-f6f93257ce38-ui-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-7w7vs\" (UID: \"acfd9c8e-51bf-426a-bbd2-f6f93257ce38\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7w7vs" Sep 11 09:36:37.567375 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:36:37.567217 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/acfd9c8e-51bf-426a-bbd2-f6f93257ce38-feast-data\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-7w7vs\" (UID: \"acfd9c8e-51bf-426a-bbd2-f6f93257ce38\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7w7vs" Sep 11 09:36:37.567375 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:36:37.567272 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/acfd9c8e-51bf-426a-bbd2-f6f93257ce38-registry-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-7w7vs\" (UID: \"acfd9c8e-51bf-426a-bbd2-f6f93257ce38\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7w7vs" Sep 11 09:36:37.567375 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:36:37.567300 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/acfd9c8e-51bf-426a-bbd2-f6f93257ce38-offline-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-7w7vs\" (UID: \"acfd9c8e-51bf-426a-bbd2-f6f93257ce38\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7w7vs" Sep 11 09:36:37.567375 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:36:37.567343 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/acfd9c8e-51bf-426a-bbd2-f6f93257ce38-online-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-7w7vs\" (UID: \"acfd9c8e-51bf-426a-bbd2-f6f93257ce38\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7w7vs" Sep 11 09:36:37.567375 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:36:37.567358 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59m5s\" (UniqueName: \"kubernetes.io/projected/acfd9c8e-51bf-426a-bbd2-f6f93257ce38-kube-api-access-59m5s\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-7w7vs\" (UID: \"acfd9c8e-51bf-426a-bbd2-f6f93257ce38\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7w7vs" Sep 11 09:36:37.668790 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:36:37.668756 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/acfd9c8e-51bf-426a-bbd2-f6f93257ce38-offline-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-7w7vs\" (UID: \"acfd9c8e-51bf-426a-bbd2-f6f93257ce38\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7w7vs" Sep 11 09:36:37.668969 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:36:37.668805 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/acfd9c8e-51bf-426a-bbd2-f6f93257ce38-online-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-7w7vs\" (UID: \"acfd9c8e-51bf-426a-bbd2-f6f93257ce38\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7w7vs" Sep 11 09:36:37.668969 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:36:37.668833 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59m5s\" (UniqueName: \"kubernetes.io/projected/acfd9c8e-51bf-426a-bbd2-f6f93257ce38-kube-api-access-59m5s\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-7w7vs\" (UID: \"acfd9c8e-51bf-426a-bbd2-f6f93257ce38\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7w7vs" Sep 11 09:36:37.668969 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:36:37.668862 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/acfd9c8e-51bf-426a-bbd2-f6f93257ce38-ui-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-7w7vs\" (UID: \"acfd9c8e-51bf-426a-bbd2-f6f93257ce38\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7w7vs" Sep 11 09:36:37.668969 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:36:37.668909 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/acfd9c8e-51bf-426a-bbd2-f6f93257ce38-feast-data\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-7w7vs\" (UID: \"acfd9c8e-51bf-426a-bbd2-f6f93257ce38\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7w7vs" Sep 11 09:36:37.668969 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:36:37.668960 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/acfd9c8e-51bf-426a-bbd2-f6f93257ce38-registry-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-7w7vs\" (UID: \"acfd9c8e-51bf-426a-bbd2-f6f93257ce38\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7w7vs" Sep 11 09:36:37.669416 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:36:37.669385 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/acfd9c8e-51bf-426a-bbd2-f6f93257ce38-feast-data\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-7w7vs\" (UID: \"acfd9c8e-51bf-426a-bbd2-f6f93257ce38\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7w7vs" Sep 11 09:36:37.669742 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:36:37.669717 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/acfd9c8e-51bf-426a-bbd2-f6f93257ce38-registry-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-7w7vs\" (UID: \"acfd9c8e-51bf-426a-bbd2-f6f93257ce38\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7w7vs" Sep 11 09:36:37.671344 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:36:37.671318 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/acfd9c8e-51bf-426a-bbd2-f6f93257ce38-online-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-7w7vs\" (UID: \"acfd9c8e-51bf-426a-bbd2-f6f93257ce38\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7w7vs" Sep 11 09:36:37.671344 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:36:37.671326 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/acfd9c8e-51bf-426a-bbd2-f6f93257ce38-offline-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-7w7vs\" (UID: \"acfd9c8e-51bf-426a-bbd2-f6f93257ce38\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7w7vs" Sep 11 09:36:37.671510 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:36:37.671414 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/acfd9c8e-51bf-426a-bbd2-f6f93257ce38-ui-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-7w7vs\" (UID: \"acfd9c8e-51bf-426a-bbd2-f6f93257ce38\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7w7vs" Sep 11 09:36:37.678045 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:36:37.678017 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-59m5s\" (UniqueName: \"kubernetes.io/projected/acfd9c8e-51bf-426a-bbd2-f6f93257ce38-kube-api-access-59m5s\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-7w7vs\" (UID: \"acfd9c8e-51bf-426a-bbd2-f6f93257ce38\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7w7vs" Sep 11 09:36:37.751656 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:36:37.751617 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7w7vs" Sep 11 09:36:38.039397 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:36:38.039369 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7w7vs"] Sep 11 09:36:38.041905 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:36:38.041876 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podacfd9c8e_51bf_426a_bbd2_f6f93257ce38.slice/crio-1027a94eb1e053f12450799476b0ff297bae58e9b162914534f372da54919a4f WatchSource:0}: Error finding container 1027a94eb1e053f12450799476b0ff297bae58e9b162914534f372da54919a4f: Status 404 returned error can't find the container with id 1027a94eb1e053f12450799476b0ff297bae58e9b162914534f372da54919a4f Sep 11 09:36:38.466067 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:36:38.465960 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7w7vs" event={"ID":"acfd9c8e-51bf-426a-bbd2-f6f93257ce38","Type":"ContainerStarted","Data":"13b54aeb6b9c17e875d47adbfef8e6e0cda0508a75ced7fcc880e58de1b02df4"} Sep 11 09:36:38.466067 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:36:38.466011 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7w7vs" event={"ID":"acfd9c8e-51bf-426a-bbd2-f6f93257ce38","Type":"ContainerStarted","Data":"1027a94eb1e053f12450799476b0ff297bae58e9b162914534f372da54919a4f"} Sep 11 09:36:43.486915 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:36:43.486818 2575 generic.go:356] "Generic (PLEG): container finished" podID="acfd9c8e-51bf-426a-bbd2-f6f93257ce38" containerID="13b54aeb6b9c17e875d47adbfef8e6e0cda0508a75ced7fcc880e58de1b02df4" exitCode=0 Sep 11 09:36:43.486915 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:36:43.486894 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7w7vs" event={"ID":"acfd9c8e-51bf-426a-bbd2-f6f93257ce38","Type":"ContainerDied","Data":"13b54aeb6b9c17e875d47adbfef8e6e0cda0508a75ced7fcc880e58de1b02df4"} Sep 11 09:36:44.493208 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:36:44.493174 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7w7vs" event={"ID":"acfd9c8e-51bf-426a-bbd2-f6f93257ce38","Type":"ContainerStarted","Data":"18889110de74017fab89dc29926d87872f2714a5edcc0cf19bf4a33baec85a6d"} Sep 11 09:37:07.590675 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:37:07.590639 2575 generic.go:356] "Generic (PLEG): container finished" podID="acfd9c8e-51bf-426a-bbd2-f6f93257ce38" containerID="18889110de74017fab89dc29926d87872f2714a5edcc0cf19bf4a33baec85a6d" exitCode=0 Sep 11 09:37:07.591072 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:37:07.590714 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7w7vs" event={"ID":"acfd9c8e-51bf-426a-bbd2-f6f93257ce38","Type":"ContainerDied","Data":"18889110de74017fab89dc29926d87872f2714a5edcc0cf19bf4a33baec85a6d"} Sep 11 09:37:08.598855 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:37:08.598816 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7w7vs" event={"ID":"acfd9c8e-51bf-426a-bbd2-f6f93257ce38","Type":"ContainerStarted","Data":"09379930e017e026e210df4ed42fb7c7e3f6f02627fc66772c3d3c9eff8a6870"} Sep 11 09:37:08.599240 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:37:08.598878 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7w7vs" event={"ID":"acfd9c8e-51bf-426a-bbd2-f6f93257ce38","Type":"ContainerStarted","Data":"7777b1090e56b5016e3148b736aa7ac9b79fb2ec89a90fe193a2a84e652a1d0d"} Sep 11 09:37:08.599240 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:37:08.598899 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7w7vs" event={"ID":"acfd9c8e-51bf-426a-bbd2-f6f93257ce38","Type":"ContainerStarted","Data":"b80114db28d31a81518b9c672df1c0a667a8be67102d9241b9e290885909ce91"} Sep 11 09:37:08.625416 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:37:08.625346 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7w7vs" podStartSLOduration=31.625330263 podStartE2EDuration="31.625330263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 09:36:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 09:37:08.624210775 +0000 UTC m=+1444.964820993" watchObservedRunningTime="2026-09-11 09:37:08.625330263 +0000 UTC m=+1444.965940469" Sep 11 09:37:10.752180 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:37:10.752135 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7w7vs" Sep 11 09:37:10.752700 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:37:10.752202 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7w7vs" Sep 11 09:37:10.752700 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:37:10.752218 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7w7vs" Sep 11 09:37:10.753886 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:37:10.753848 2575 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7w7vs" podUID="acfd9c8e-51bf-426a-bbd2-f6f93257ce38" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.30:8443: connect: connection refused" Sep 11 09:37:10.754031 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:37:10.753874 2575 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7w7vs" podUID="acfd9c8e-51bf-426a-bbd2-f6f93257ce38" containerName="online" probeResult="failure" output="Get \"https://10.134.0.30:6567/health\": dial tcp 10.134.0.30:6567: connect: connection refused" Sep 11 09:37:10.754031 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:37:10.753856 2575 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7w7vs" podUID="acfd9c8e-51bf-426a-bbd2-f6f93257ce38" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.30:8816: connect: connection refused" Sep 11 09:37:13.756720 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:37:13.756662 2575 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7w7vs" podUID="acfd9c8e-51bf-426a-bbd2-f6f93257ce38" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.30:8816: connect: connection refused" Sep 11 09:37:13.757210 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:37:13.756662 2575 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7w7vs" podUID="acfd9c8e-51bf-426a-bbd2-f6f93257ce38" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.30:8443: connect: connection refused" Sep 11 09:37:13.757415 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:37:13.757382 2575 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7w7vs" podUID="acfd9c8e-51bf-426a-bbd2-f6f93257ce38" containerName="online" probeResult="failure" output="Get \"https://10.134.0.30:6567/health\": dial tcp 10.134.0.30:6567: connect: connection refused" Sep 11 09:37:16.753095 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:37:16.753065 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7w7vs" Sep 11 09:37:16.753754 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:37:16.753182 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7w7vs" Sep 11 09:37:16.753754 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:37:16.753512 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7w7vs" Sep 11 09:37:16.753754 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:37:16.753556 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7w7vs" Sep 11 09:37:16.753906 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:37:16.753806 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7w7vs" Sep 11 09:37:16.753990 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:37:16.753969 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7w7vs" Sep 11 09:37:17.751727 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:37:17.751691 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7w7vs" Sep 11 09:37:17.756597 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:37:17.756561 2575 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7w7vs" podUID="acfd9c8e-51bf-426a-bbd2-f6f93257ce38" containerName="online" probeResult="failure" output="Get \"https://10.134.0.30:6567/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Sep 11 09:37:20.461687 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:37:20.461649 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7w7vs" Sep 11 09:37:21.340482 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:37:21.340437 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7w7vs" Sep 11 09:38:18.756617 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:38:18.756569 2575 prober.go:120] "Probe failed" probeType="Liveness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7w7vs" podUID="acfd9c8e-51bf-426a-bbd2-f6f93257ce38" containerName="online" probeResult="failure" output="Get \"https://10.134.0.30:6567/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 11 09:38:23.796409 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:38:23.796372 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7w7vs"] Sep 11 09:38:23.796856 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:38:23.796625 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7w7vs" podUID="acfd9c8e-51bf-426a-bbd2-f6f93257ce38" containerName="online" containerID="cri-o://b80114db28d31a81518b9c672df1c0a667a8be67102d9241b9e290885909ce91" gracePeriod=30 Sep 11 09:38:23.796856 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:38:23.796684 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7w7vs" podUID="acfd9c8e-51bf-426a-bbd2-f6f93257ce38" containerName="offline" containerID="cri-o://7777b1090e56b5016e3148b736aa7ac9b79fb2ec89a90fe193a2a84e652a1d0d" gracePeriod=30 Sep 11 09:38:23.796856 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:38:23.796684 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7w7vs" podUID="acfd9c8e-51bf-426a-bbd2-f6f93257ce38" containerName="ui" containerID="cri-o://09379930e017e026e210df4ed42fb7c7e3f6f02627fc66772c3d3c9eff8a6870" gracePeriod=30 Sep 11 09:38:23.926870 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:38:23.926829 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-s4zzq"] Sep 11 09:38:23.927194 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:38:23.927135 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-s4zzq" podUID="48c3c3a9-66e3-4aab-b200-71340090b4b8" containerName="registry" containerID="cri-o://8e75e4fb65cd7a3005f1a6d109d2e859ad49d82adf3b8f3e86920640bccedde7" gracePeriod=30 Sep 11 09:38:23.927346 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:38:23.927325 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-s4zzq" podUID="48c3c3a9-66e3-4aab-b200-71340090b4b8" containerName="ui" containerID="cri-o://48da851e4310d4c67d72e8b77ef2c414dcc4a229a60991d42914902ff355a86d" gracePeriod=30 Sep 11 09:38:23.927436 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:38:23.927420 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-s4zzq" podUID="48c3c3a9-66e3-4aab-b200-71340090b4b8" containerName="offline" containerID="cri-o://9925a7a002dd1568283efc4501734e4eb97b7817a2e8de5ca766ab244f3de61f" gracePeriod=30 Sep 11 09:38:23.927529 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:38:23.927503 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-s4zzq" podUID="48c3c3a9-66e3-4aab-b200-71340090b4b8" containerName="online" containerID="cri-o://410510e71d23571cf5d6776763d6a50c98d09fb2baa9eab4a44d13564b3074c4" gracePeriod=30 Sep 11 09:38:24.936861 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:38:24.936780 2575 generic.go:356] "Generic (PLEG): container finished" podID="48c3c3a9-66e3-4aab-b200-71340090b4b8" containerID="48da851e4310d4c67d72e8b77ef2c414dcc4a229a60991d42914902ff355a86d" exitCode=0 Sep 11 09:38:24.936861 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:38:24.936804 2575 generic.go:356] "Generic (PLEG): container finished" podID="48c3c3a9-66e3-4aab-b200-71340090b4b8" containerID="410510e71d23571cf5d6776763d6a50c98d09fb2baa9eab4a44d13564b3074c4" exitCode=0 Sep 11 09:38:24.936861 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:38:24.936854 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-s4zzq" event={"ID":"48c3c3a9-66e3-4aab-b200-71340090b4b8","Type":"ContainerDied","Data":"48da851e4310d4c67d72e8b77ef2c414dcc4a229a60991d42914902ff355a86d"} Sep 11 09:38:24.937323 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:38:24.936899 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-s4zzq" event={"ID":"48c3c3a9-66e3-4aab-b200-71340090b4b8","Type":"ContainerDied","Data":"410510e71d23571cf5d6776763d6a50c98d09fb2baa9eab4a44d13564b3074c4"} Sep 11 09:38:24.939679 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:38:24.939653 2575 generic.go:356] "Generic (PLEG): container finished" podID="acfd9c8e-51bf-426a-bbd2-f6f93257ce38" containerID="09379930e017e026e210df4ed42fb7c7e3f6f02627fc66772c3d3c9eff8a6870" exitCode=0 Sep 11 09:38:24.939679 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:38:24.939673 2575 generic.go:356] "Generic (PLEG): container finished" podID="acfd9c8e-51bf-426a-bbd2-f6f93257ce38" containerID="b80114db28d31a81518b9c672df1c0a667a8be67102d9241b9e290885909ce91" exitCode=0 Sep 11 09:38:24.939835 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:38:24.939721 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7w7vs" event={"ID":"acfd9c8e-51bf-426a-bbd2-f6f93257ce38","Type":"ContainerDied","Data":"09379930e017e026e210df4ed42fb7c7e3f6f02627fc66772c3d3c9eff8a6870"} Sep 11 09:38:24.939835 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:38:24.939753 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7w7vs" event={"ID":"acfd9c8e-51bf-426a-bbd2-f6f93257ce38","Type":"ContainerDied","Data":"b80114db28d31a81518b9c672df1c0a667a8be67102d9241b9e290885909ce91"} Sep 11 09:38:25.154290 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:38:25.154242 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-s4zzq" podUID="48c3c3a9-66e3-4aab-b200-71340090b4b8" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.29:8443: connect: connection refused" Sep 11 09:38:25.197696 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:38:25.197603 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-s4zzq" podUID="48c3c3a9-66e3-4aab-b200-71340090b4b8" containerName="online" probeResult="failure" output="Get \"https://10.134.0.29:6567/health\": dial tcp 10.134.0.29:6567: connect: connection refused" Sep 11 09:38:26.754589 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:38:26.754462 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7w7vs" podUID="acfd9c8e-51bf-426a-bbd2-f6f93257ce38" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.30:8443: connect: connection refused" Sep 11 09:38:30.656743 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:38:30.656702 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7w7vs" podUID="acfd9c8e-51bf-426a-bbd2-f6f93257ce38" containerName="online" probeResult="failure" output="Get \"https://10.134.0.30:6567/health\": dial tcp 10.134.0.30:6567: connect: connection refused" Sep 11 09:38:35.153410 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:38:35.153363 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-s4zzq" podUID="48c3c3a9-66e3-4aab-b200-71340090b4b8" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.29:8443: connect: connection refused" Sep 11 09:38:35.197723 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:38:35.197678 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-s4zzq" podUID="48c3c3a9-66e3-4aab-b200-71340090b4b8" containerName="online" probeResult="failure" output="Get \"https://10.134.0.29:6567/health\": dial tcp 10.134.0.29:6567: connect: connection refused" Sep 11 09:38:36.753738 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:38:36.753688 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7w7vs" podUID="acfd9c8e-51bf-426a-bbd2-f6f93257ce38" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.30:8443: connect: connection refused" Sep 11 09:38:40.656640 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:38:40.656590 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7w7vs" podUID="acfd9c8e-51bf-426a-bbd2-f6f93257ce38" containerName="online" probeResult="failure" output="Get \"https://10.134.0.30:6567/health\": dial tcp 10.134.0.30:6567: connect: connection refused" Sep 11 09:38:45.154430 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:38:45.154381 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-s4zzq" podUID="48c3c3a9-66e3-4aab-b200-71340090b4b8" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.29:8443: connect: connection refused" Sep 11 09:38:45.154922 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:38:45.154572 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-s4zzq" Sep 11 09:38:45.197670 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:38:45.197627 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-s4zzq" podUID="48c3c3a9-66e3-4aab-b200-71340090b4b8" containerName="online" probeResult="failure" output="Get \"https://10.134.0.29:6567/health\": dial tcp 10.134.0.29:6567: connect: connection refused" Sep 11 09:38:45.197877 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:38:45.197735 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-s4zzq" Sep 11 09:38:46.754182 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:38:46.754136 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7w7vs" podUID="acfd9c8e-51bf-426a-bbd2-f6f93257ce38" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.30:8443: connect: connection refused" Sep 11 09:38:46.754609 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:38:46.754275 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7w7vs" Sep 11 09:38:50.657054 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:38:50.657007 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7w7vs" podUID="acfd9c8e-51bf-426a-bbd2-f6f93257ce38" containerName="online" probeResult="failure" output="Get \"https://10.134.0.30:6567/health\": dial tcp 10.134.0.30:6567: connect: connection refused" Sep 11 09:38:50.657553 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:38:50.657115 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7w7vs" Sep 11 09:38:54.053721 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:38:54.053663 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7w7vs" Sep 11 09:38:54.063144 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:38:54.063119 2575 generic.go:356] "Generic (PLEG): container finished" podID="48c3c3a9-66e3-4aab-b200-71340090b4b8" containerID="9925a7a002dd1568283efc4501734e4eb97b7817a2e8de5ca766ab244f3de61f" exitCode=137 Sep 11 09:38:54.063144 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:38:54.063141 2575 generic.go:356] "Generic (PLEG): container finished" podID="48c3c3a9-66e3-4aab-b200-71340090b4b8" containerID="8e75e4fb65cd7a3005f1a6d109d2e859ad49d82adf3b8f3e86920640bccedde7" exitCode=137 Sep 11 09:38:54.063340 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:38:54.063195 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-s4zzq" event={"ID":"48c3c3a9-66e3-4aab-b200-71340090b4b8","Type":"ContainerDied","Data":"9925a7a002dd1568283efc4501734e4eb97b7817a2e8de5ca766ab244f3de61f"} Sep 11 09:38:54.063340 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:38:54.063241 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-s4zzq" event={"ID":"48c3c3a9-66e3-4aab-b200-71340090b4b8","Type":"ContainerDied","Data":"8e75e4fb65cd7a3005f1a6d109d2e859ad49d82adf3b8f3e86920640bccedde7"} Sep 11 09:38:54.065621 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:38:54.065598 2575 generic.go:356] "Generic (PLEG): container finished" podID="acfd9c8e-51bf-426a-bbd2-f6f93257ce38" containerID="7777b1090e56b5016e3148b736aa7ac9b79fb2ec89a90fe193a2a84e652a1d0d" exitCode=137 Sep 11 09:38:54.065712 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:38:54.065664 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7w7vs" Sep 11 09:38:54.065712 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:38:54.065673 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7w7vs" event={"ID":"acfd9c8e-51bf-426a-bbd2-f6f93257ce38","Type":"ContainerDied","Data":"7777b1090e56b5016e3148b736aa7ac9b79fb2ec89a90fe193a2a84e652a1d0d"} Sep 11 09:38:54.065783 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:38:54.065711 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7w7vs" event={"ID":"acfd9c8e-51bf-426a-bbd2-f6f93257ce38","Type":"ContainerDied","Data":"1027a94eb1e053f12450799476b0ff297bae58e9b162914534f372da54919a4f"} Sep 11 09:38:54.065783 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:38:54.065731 2575 scope.go:117] "RemoveContainer" containerID="09379930e017e026e210df4ed42fb7c7e3f6f02627fc66772c3d3c9eff8a6870" Sep 11 09:38:54.073746 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:38:54.073728 2575 scope.go:117] "RemoveContainer" containerID="7777b1090e56b5016e3148b736aa7ac9b79fb2ec89a90fe193a2a84e652a1d0d" Sep 11 09:38:54.083148 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:38:54.083119 2575 scope.go:117] "RemoveContainer" containerID="b80114db28d31a81518b9c672df1c0a667a8be67102d9241b9e290885909ce91" Sep 11 09:38:54.090953 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:38:54.090928 2575 scope.go:117] "RemoveContainer" containerID="18889110de74017fab89dc29926d87872f2714a5edcc0cf19bf4a33baec85a6d" Sep 11 09:38:54.098085 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:38:54.098069 2575 scope.go:117] "RemoveContainer" containerID="13b54aeb6b9c17e875d47adbfef8e6e0cda0508a75ced7fcc880e58de1b02df4" Sep 11 09:38:54.105188 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:38:54.105172 2575 scope.go:117] "RemoveContainer" containerID="09379930e017e026e210df4ed42fb7c7e3f6f02627fc66772c3d3c9eff8a6870" Sep 11 09:38:54.105442 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:38:54.105421 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09379930e017e026e210df4ed42fb7c7e3f6f02627fc66772c3d3c9eff8a6870\": container with ID starting with 09379930e017e026e210df4ed42fb7c7e3f6f02627fc66772c3d3c9eff8a6870 not found: ID does not exist" containerID="09379930e017e026e210df4ed42fb7c7e3f6f02627fc66772c3d3c9eff8a6870" Sep 11 09:38:54.105505 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:38:54.105455 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09379930e017e026e210df4ed42fb7c7e3f6f02627fc66772c3d3c9eff8a6870"} err="failed to get container status \"09379930e017e026e210df4ed42fb7c7e3f6f02627fc66772c3d3c9eff8a6870\": rpc error: code = NotFound desc = could not find container \"09379930e017e026e210df4ed42fb7c7e3f6f02627fc66772c3d3c9eff8a6870\": container with ID starting with 09379930e017e026e210df4ed42fb7c7e3f6f02627fc66772c3d3c9eff8a6870 not found: ID does not exist" Sep 11 09:38:54.105505 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:38:54.105494 2575 scope.go:117] "RemoveContainer" containerID="7777b1090e56b5016e3148b736aa7ac9b79fb2ec89a90fe193a2a84e652a1d0d" Sep 11 09:38:54.105762 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:38:54.105745 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7777b1090e56b5016e3148b736aa7ac9b79fb2ec89a90fe193a2a84e652a1d0d\": container with ID starting with 7777b1090e56b5016e3148b736aa7ac9b79fb2ec89a90fe193a2a84e652a1d0d not found: ID does not exist" containerID="7777b1090e56b5016e3148b736aa7ac9b79fb2ec89a90fe193a2a84e652a1d0d" Sep 11 09:38:54.105803 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:38:54.105769 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7777b1090e56b5016e3148b736aa7ac9b79fb2ec89a90fe193a2a84e652a1d0d"} err="failed to get container status \"7777b1090e56b5016e3148b736aa7ac9b79fb2ec89a90fe193a2a84e652a1d0d\": rpc error: code = NotFound desc = could not find container \"7777b1090e56b5016e3148b736aa7ac9b79fb2ec89a90fe193a2a84e652a1d0d\": container with ID starting with 7777b1090e56b5016e3148b736aa7ac9b79fb2ec89a90fe193a2a84e652a1d0d not found: ID does not exist" Sep 11 09:38:54.105803 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:38:54.105787 2575 scope.go:117] "RemoveContainer" containerID="b80114db28d31a81518b9c672df1c0a667a8be67102d9241b9e290885909ce91" Sep 11 09:38:54.106037 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:38:54.106020 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b80114db28d31a81518b9c672df1c0a667a8be67102d9241b9e290885909ce91\": container with ID starting with b80114db28d31a81518b9c672df1c0a667a8be67102d9241b9e290885909ce91 not found: ID does not exist" containerID="b80114db28d31a81518b9c672df1c0a667a8be67102d9241b9e290885909ce91" Sep 11 09:38:54.106096 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:38:54.106041 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b80114db28d31a81518b9c672df1c0a667a8be67102d9241b9e290885909ce91"} err="failed to get container status \"b80114db28d31a81518b9c672df1c0a667a8be67102d9241b9e290885909ce91\": rpc error: code = NotFound desc = could not find container \"b80114db28d31a81518b9c672df1c0a667a8be67102d9241b9e290885909ce91\": container with ID starting with b80114db28d31a81518b9c672df1c0a667a8be67102d9241b9e290885909ce91 not found: ID does not exist" Sep 11 09:38:54.106096 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:38:54.106057 2575 scope.go:117] "RemoveContainer" containerID="18889110de74017fab89dc29926d87872f2714a5edcc0cf19bf4a33baec85a6d" Sep 11 09:38:54.106282 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:38:54.106265 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18889110de74017fab89dc29926d87872f2714a5edcc0cf19bf4a33baec85a6d\": container with ID starting with 18889110de74017fab89dc29926d87872f2714a5edcc0cf19bf4a33baec85a6d not found: ID does not exist" containerID="18889110de74017fab89dc29926d87872f2714a5edcc0cf19bf4a33baec85a6d" Sep 11 09:38:54.106322 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:38:54.106288 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18889110de74017fab89dc29926d87872f2714a5edcc0cf19bf4a33baec85a6d"} err="failed to get container status \"18889110de74017fab89dc29926d87872f2714a5edcc0cf19bf4a33baec85a6d\": rpc error: code = NotFound desc = could not find container \"18889110de74017fab89dc29926d87872f2714a5edcc0cf19bf4a33baec85a6d\": container with ID starting with 18889110de74017fab89dc29926d87872f2714a5edcc0cf19bf4a33baec85a6d not found: ID does not exist" Sep 11 09:38:54.106322 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:38:54.106309 2575 scope.go:117] "RemoveContainer" containerID="13b54aeb6b9c17e875d47adbfef8e6e0cda0508a75ced7fcc880e58de1b02df4" Sep 11 09:38:54.106562 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:38:54.106547 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13b54aeb6b9c17e875d47adbfef8e6e0cda0508a75ced7fcc880e58de1b02df4\": container with ID starting with 13b54aeb6b9c17e875d47adbfef8e6e0cda0508a75ced7fcc880e58de1b02df4 not found: ID does not exist" containerID="13b54aeb6b9c17e875d47adbfef8e6e0cda0508a75ced7fcc880e58de1b02df4" Sep 11 09:38:54.106625 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:38:54.106565 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13b54aeb6b9c17e875d47adbfef8e6e0cda0508a75ced7fcc880e58de1b02df4"} err="failed to get container status \"13b54aeb6b9c17e875d47adbfef8e6e0cda0508a75ced7fcc880e58de1b02df4\": rpc error: code = NotFound desc = could not find container \"13b54aeb6b9c17e875d47adbfef8e6e0cda0508a75ced7fcc880e58de1b02df4\": container with ID starting with 13b54aeb6b9c17e875d47adbfef8e6e0cda0508a75ced7fcc880e58de1b02df4 not found: ID does not exist" Sep 11 09:38:54.224888 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:38:54.224853 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/acfd9c8e-51bf-426a-bbd2-f6f93257ce38-offline-tls\") pod \"acfd9c8e-51bf-426a-bbd2-f6f93257ce38\" (UID: \"acfd9c8e-51bf-426a-bbd2-f6f93257ce38\") " Sep 11 09:38:54.225083 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:38:54.224915 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/acfd9c8e-51bf-426a-bbd2-f6f93257ce38-online-tls\") pod \"acfd9c8e-51bf-426a-bbd2-f6f93257ce38\" (UID: \"acfd9c8e-51bf-426a-bbd2-f6f93257ce38\") " Sep 11 09:38:54.225160 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:38:54.225087 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/acfd9c8e-51bf-426a-bbd2-f6f93257ce38-feast-data\") pod \"acfd9c8e-51bf-426a-bbd2-f6f93257ce38\" (UID: \"acfd9c8e-51bf-426a-bbd2-f6f93257ce38\") " Sep 11 09:38:54.225160 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:38:54.225122 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-59m5s\" (UniqueName: \"kubernetes.io/projected/acfd9c8e-51bf-426a-bbd2-f6f93257ce38-kube-api-access-59m5s\") pod \"acfd9c8e-51bf-426a-bbd2-f6f93257ce38\" (UID: \"acfd9c8e-51bf-426a-bbd2-f6f93257ce38\") " Sep 11 09:38:54.225160 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:38:54.225156 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/acfd9c8e-51bf-426a-bbd2-f6f93257ce38-ui-tls\") pod \"acfd9c8e-51bf-426a-bbd2-f6f93257ce38\" (UID: \"acfd9c8e-51bf-426a-bbd2-f6f93257ce38\") " Sep 11 09:38:54.225328 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:38:54.225178 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/acfd9c8e-51bf-426a-bbd2-f6f93257ce38-registry-tls\") pod \"acfd9c8e-51bf-426a-bbd2-f6f93257ce38\" (UID: \"acfd9c8e-51bf-426a-bbd2-f6f93257ce38\") " Sep 11 09:38:54.225929 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:38:54.225762 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acfd9c8e-51bf-426a-bbd2-f6f93257ce38-feast-data" (OuterVolumeSpecName: "feast-data") pod "acfd9c8e-51bf-426a-bbd2-f6f93257ce38" (UID: "acfd9c8e-51bf-426a-bbd2-f6f93257ce38"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 09:38:54.225929 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:38:54.225872 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acfd9c8e-51bf-426a-bbd2-f6f93257ce38-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "acfd9c8e-51bf-426a-bbd2-f6f93257ce38" (UID: "acfd9c8e-51bf-426a-bbd2-f6f93257ce38"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 09:38:54.227327 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:38:54.227300 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/acfd9c8e-51bf-426a-bbd2-f6f93257ce38-online-tls" (OuterVolumeSpecName: "online-tls") pod "acfd9c8e-51bf-426a-bbd2-f6f93257ce38" (UID: "acfd9c8e-51bf-426a-bbd2-f6f93257ce38"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 09:38:54.227445 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:38:54.227354 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acfd9c8e-51bf-426a-bbd2-f6f93257ce38-kube-api-access-59m5s" (OuterVolumeSpecName: "kube-api-access-59m5s") pod "acfd9c8e-51bf-426a-bbd2-f6f93257ce38" (UID: "acfd9c8e-51bf-426a-bbd2-f6f93257ce38"). InnerVolumeSpecName "kube-api-access-59m5s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 09:38:54.227445 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:38:54.227434 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/acfd9c8e-51bf-426a-bbd2-f6f93257ce38-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "acfd9c8e-51bf-426a-bbd2-f6f93257ce38" (UID: "acfd9c8e-51bf-426a-bbd2-f6f93257ce38"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 09:38:54.227675 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:38:54.227658 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/acfd9c8e-51bf-426a-bbd2-f6f93257ce38-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "acfd9c8e-51bf-426a-bbd2-f6f93257ce38" (UID: "acfd9c8e-51bf-426a-bbd2-f6f93257ce38"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 09:38:54.326463 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:38:54.326383 2575 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/acfd9c8e-51bf-426a-bbd2-f6f93257ce38-offline-tls\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\"" Sep 11 09:38:54.326463 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:38:54.326413 2575 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/acfd9c8e-51bf-426a-bbd2-f6f93257ce38-online-tls\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\"" Sep 11 09:38:54.326463 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:38:54.326422 2575 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/acfd9c8e-51bf-426a-bbd2-f6f93257ce38-feast-data\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\"" Sep 11 09:38:54.326463 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:38:54.326430 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59m5s\" (UniqueName: \"kubernetes.io/projected/acfd9c8e-51bf-426a-bbd2-f6f93257ce38-kube-api-access-59m5s\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\"" Sep 11 09:38:54.326463 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:38:54.326442 2575 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/acfd9c8e-51bf-426a-bbd2-f6f93257ce38-ui-tls\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\"" Sep 11 09:38:54.326463 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:38:54.326451 2575 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/acfd9c8e-51bf-426a-bbd2-f6f93257ce38-registry-tls\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\"" Sep 11 09:38:54.390585 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:38:54.390554 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7w7vs"] Sep 11 09:38:54.393723 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:38:54.393700 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-7w7vs"] Sep 11 09:38:54.575509 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:38:54.575439 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-s4zzq" Sep 11 09:38:54.731214 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:38:54.731127 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/48c3c3a9-66e3-4aab-b200-71340090b4b8-feast-data\") pod \"48c3c3a9-66e3-4aab-b200-71340090b4b8\" (UID: \"48c3c3a9-66e3-4aab-b200-71340090b4b8\") " Sep 11 09:38:54.731214 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:38:54.731171 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/48c3c3a9-66e3-4aab-b200-71340090b4b8-offline-tls\") pod \"48c3c3a9-66e3-4aab-b200-71340090b4b8\" (UID: \"48c3c3a9-66e3-4aab-b200-71340090b4b8\") " Sep 11 09:38:54.731214 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:38:54.731210 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w94gb\" (UniqueName: \"kubernetes.io/projected/48c3c3a9-66e3-4aab-b200-71340090b4b8-kube-api-access-w94gb\") pod \"48c3c3a9-66e3-4aab-b200-71340090b4b8\" (UID: \"48c3c3a9-66e3-4aab-b200-71340090b4b8\") " Sep 11 09:38:54.731446 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:38:54.731226 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/48c3c3a9-66e3-4aab-b200-71340090b4b8-ui-tls\") pod \"48c3c3a9-66e3-4aab-b200-71340090b4b8\" (UID: \"48c3c3a9-66e3-4aab-b200-71340090b4b8\") " Sep 11 09:38:54.731446 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:38:54.731243 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/48c3c3a9-66e3-4aab-b200-71340090b4b8-registry-tls\") pod \"48c3c3a9-66e3-4aab-b200-71340090b4b8\" (UID: \"48c3c3a9-66e3-4aab-b200-71340090b4b8\") " Sep 11 09:38:54.731446 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:38:54.731331 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/48c3c3a9-66e3-4aab-b200-71340090b4b8-online-tls\") pod \"48c3c3a9-66e3-4aab-b200-71340090b4b8\" (UID: \"48c3c3a9-66e3-4aab-b200-71340090b4b8\") " Sep 11 09:38:54.731755 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:38:54.731726 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48c3c3a9-66e3-4aab-b200-71340090b4b8-feast-data" (OuterVolumeSpecName: "feast-data") pod "48c3c3a9-66e3-4aab-b200-71340090b4b8" (UID: "48c3c3a9-66e3-4aab-b200-71340090b4b8"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 09:38:54.733456 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:38:54.733431 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48c3c3a9-66e3-4aab-b200-71340090b4b8-online-tls" (OuterVolumeSpecName: "online-tls") pod "48c3c3a9-66e3-4aab-b200-71340090b4b8" (UID: "48c3c3a9-66e3-4aab-b200-71340090b4b8"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 09:38:54.733913 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:38:54.733889 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48c3c3a9-66e3-4aab-b200-71340090b4b8-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "48c3c3a9-66e3-4aab-b200-71340090b4b8" (UID: "48c3c3a9-66e3-4aab-b200-71340090b4b8"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 09:38:54.733913 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:38:54.733899 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48c3c3a9-66e3-4aab-b200-71340090b4b8-kube-api-access-w94gb" (OuterVolumeSpecName: "kube-api-access-w94gb") pod "48c3c3a9-66e3-4aab-b200-71340090b4b8" (UID: "48c3c3a9-66e3-4aab-b200-71340090b4b8"). InnerVolumeSpecName "kube-api-access-w94gb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 09:38:54.734022 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:38:54.733903 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48c3c3a9-66e3-4aab-b200-71340090b4b8-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "48c3c3a9-66e3-4aab-b200-71340090b4b8" (UID: "48c3c3a9-66e3-4aab-b200-71340090b4b8"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 09:38:54.734022 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:38:54.733919 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48c3c3a9-66e3-4aab-b200-71340090b4b8-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "48c3c3a9-66e3-4aab-b200-71340090b4b8" (UID: "48c3c3a9-66e3-4aab-b200-71340090b4b8"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 09:38:54.831960 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:38:54.831930 2575 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/48c3c3a9-66e3-4aab-b200-71340090b4b8-online-tls\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\"" Sep 11 09:38:54.831960 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:38:54.831959 2575 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/48c3c3a9-66e3-4aab-b200-71340090b4b8-feast-data\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\"" Sep 11 09:38:54.832099 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:38:54.831969 2575 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/48c3c3a9-66e3-4aab-b200-71340090b4b8-offline-tls\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\"" Sep 11 09:38:54.832099 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:38:54.831977 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w94gb\" (UniqueName: \"kubernetes.io/projected/48c3c3a9-66e3-4aab-b200-71340090b4b8-kube-api-access-w94gb\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\"" Sep 11 09:38:54.832099 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:38:54.831986 2575 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/48c3c3a9-66e3-4aab-b200-71340090b4b8-ui-tls\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\"" Sep 11 09:38:54.832099 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:38:54.831993 2575 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/48c3c3a9-66e3-4aab-b200-71340090b4b8-registry-tls\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\"" Sep 11 09:38:55.072093 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:38:55.072061 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-s4zzq" Sep 11 09:38:55.072093 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:38:55.072071 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-s4zzq" event={"ID":"48c3c3a9-66e3-4aab-b200-71340090b4b8","Type":"ContainerDied","Data":"6d49aada29243b4b36719596c742cc4fcf2528974d954a297d688cfaa6d344c4"} Sep 11 09:38:55.072651 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:38:55.072130 2575 scope.go:117] "RemoveContainer" containerID="48da851e4310d4c67d72e8b77ef2c414dcc4a229a60991d42914902ff355a86d" Sep 11 09:38:55.080898 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:38:55.080879 2575 scope.go:117] "RemoveContainer" containerID="9925a7a002dd1568283efc4501734e4eb97b7817a2e8de5ca766ab244f3de61f" Sep 11 09:38:55.088821 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:38:55.088802 2575 scope.go:117] "RemoveContainer" containerID="410510e71d23571cf5d6776763d6a50c98d09fb2baa9eab4a44d13564b3074c4" Sep 11 09:38:55.097789 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:38:55.097768 2575 scope.go:117] "RemoveContainer" containerID="8e75e4fb65cd7a3005f1a6d109d2e859ad49d82adf3b8f3e86920640bccedde7" Sep 11 09:38:55.099329 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:38:55.099296 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-s4zzq"] Sep 11 09:38:55.101523 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:38:55.101502 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-s4zzq"] Sep 11 09:38:55.106678 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:38:55.106663 2575 scope.go:117] "RemoveContainer" containerID="770ad57418f3ef2befce556b6b2c6092f4a429554a03105257042d2ac6bbda1b" Sep 11 09:38:55.113878 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:38:55.113861 2575 scope.go:117] "RemoveContainer" containerID="4c9a874903e8ebf64de8abd43f164ab4f890a48727013220bf6c31b87c2fe21e" Sep 11 09:38:56.113932 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:38:56.113900 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="48c3c3a9-66e3-4aab-b200-71340090b4b8" path="/var/lib/kubelet/pods/48c3c3a9-66e3-4aab-b200-71340090b4b8/volumes" Sep 11 09:38:56.114580 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:38:56.114563 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="acfd9c8e-51bf-426a-bbd2-f6f93257ce38" path="/var/lib/kubelet/pods/acfd9c8e-51bf-426a-bbd2-f6f93257ce38/volumes" Sep 11 09:39:35.435386 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:39:35.435347 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-psfvw"] Sep 11 09:39:35.436011 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:39:35.435692 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="acfd9c8e-51bf-426a-bbd2-f6f93257ce38" containerName="feast-apply" Sep 11 09:39:35.436011 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:39:35.435710 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="acfd9c8e-51bf-426a-bbd2-f6f93257ce38" containerName="feast-apply" Sep 11 09:39:35.436011 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:39:35.435733 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="acfd9c8e-51bf-426a-bbd2-f6f93257ce38" containerName="online" Sep 11 09:39:35.436011 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:39:35.435742 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="acfd9c8e-51bf-426a-bbd2-f6f93257ce38" containerName="online" Sep 11 09:39:35.436011 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:39:35.435758 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48c3c3a9-66e3-4aab-b200-71340090b4b8" containerName="online" Sep 11 09:39:35.436011 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:39:35.435767 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="48c3c3a9-66e3-4aab-b200-71340090b4b8" containerName="online" Sep 11 09:39:35.436011 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:39:35.435776 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48c3c3a9-66e3-4aab-b200-71340090b4b8" containerName="registry" Sep 11 09:39:35.436011 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:39:35.435782 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="48c3c3a9-66e3-4aab-b200-71340090b4b8" containerName="registry" Sep 11 09:39:35.436011 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:39:35.435788 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48c3c3a9-66e3-4aab-b200-71340090b4b8" containerName="ui" Sep 11 09:39:35.436011 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:39:35.435793 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="48c3c3a9-66e3-4aab-b200-71340090b4b8" containerName="ui" Sep 11 09:39:35.436011 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:39:35.435799 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="acfd9c8e-51bf-426a-bbd2-f6f93257ce38" containerName="feast-init" Sep 11 09:39:35.436011 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:39:35.435804 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="acfd9c8e-51bf-426a-bbd2-f6f93257ce38" containerName="feast-init" Sep 11 09:39:35.436011 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:39:35.435809 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48c3c3a9-66e3-4aab-b200-71340090b4b8" containerName="feast-init" Sep 11 09:39:35.436011 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:39:35.435813 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="48c3c3a9-66e3-4aab-b200-71340090b4b8" containerName="feast-init" Sep 11 09:39:35.436011 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:39:35.435821 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="acfd9c8e-51bf-426a-bbd2-f6f93257ce38" containerName="offline" Sep 11 09:39:35.436011 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:39:35.435829 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="acfd9c8e-51bf-426a-bbd2-f6f93257ce38" containerName="offline" Sep 11 09:39:35.436011 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:39:35.435838 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="acfd9c8e-51bf-426a-bbd2-f6f93257ce38" containerName="ui" Sep 11 09:39:35.436011 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:39:35.435847 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="acfd9c8e-51bf-426a-bbd2-f6f93257ce38" containerName="ui" Sep 11 09:39:35.436011 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:39:35.435857 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48c3c3a9-66e3-4aab-b200-71340090b4b8" containerName="feast-apply" Sep 11 09:39:35.436011 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:39:35.435862 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="48c3c3a9-66e3-4aab-b200-71340090b4b8" containerName="feast-apply" Sep 11 09:39:35.436011 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:39:35.435870 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48c3c3a9-66e3-4aab-b200-71340090b4b8" containerName="offline" Sep 11 09:39:35.436011 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:39:35.435875 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="48c3c3a9-66e3-4aab-b200-71340090b4b8" containerName="offline" Sep 11 09:39:35.436011 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:39:35.435942 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="48c3c3a9-66e3-4aab-b200-71340090b4b8" containerName="registry" Sep 11 09:39:35.436011 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:39:35.435950 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="48c3c3a9-66e3-4aab-b200-71340090b4b8" containerName="online" Sep 11 09:39:35.436011 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:39:35.435956 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="48c3c3a9-66e3-4aab-b200-71340090b4b8" containerName="offline" Sep 11 09:39:35.436011 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:39:35.435961 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="48c3c3a9-66e3-4aab-b200-71340090b4b8" containerName="ui" Sep 11 09:39:35.436011 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:39:35.435968 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="acfd9c8e-51bf-426a-bbd2-f6f93257ce38" containerName="ui" Sep 11 09:39:35.436011 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:39:35.435974 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="acfd9c8e-51bf-426a-bbd2-f6f93257ce38" containerName="online" Sep 11 09:39:35.436011 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:39:35.435981 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="acfd9c8e-51bf-426a-bbd2-f6f93257ce38" containerName="offline" Sep 11 09:39:35.439330 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:39:35.439312 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-psfvw" Sep 11 09:39:35.441820 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:39:35.441797 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-ui-tls\"" Sep 11 09:39:35.441942 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:39:35.441799 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-registry-tls\"" Sep 11 09:39:35.441942 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:39:35.441800 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-online-tls\"" Sep 11 09:39:35.441942 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:39:35.441799 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-offline-tls\"" Sep 11 09:39:35.442321 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:39:35.442304 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-lsn9k\"" Sep 11 09:39:35.442365 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:39:35.442320 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\"" Sep 11 09:39:35.442365 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:39:35.442350 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\"" Sep 11 09:39:35.454329 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:39:35.454309 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-psfvw"] Sep 11 09:39:35.568394 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:39:35.568361 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/9a3750ad-46fe-4bfd-ac42-fc9703e16e99-feast-data\") pod \"feast-simple-feast-setup-57576b8fcf-psfvw\" (UID: \"9a3750ad-46fe-4bfd-ac42-fc9703e16e99\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-psfvw" Sep 11 09:39:35.568600 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:39:35.568410 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/9a3750ad-46fe-4bfd-ac42-fc9703e16e99-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-psfvw\" (UID: \"9a3750ad-46fe-4bfd-ac42-fc9703e16e99\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-psfvw" Sep 11 09:39:35.568600 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:39:35.568431 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/9a3750ad-46fe-4bfd-ac42-fc9703e16e99-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-psfvw\" (UID: \"9a3750ad-46fe-4bfd-ac42-fc9703e16e99\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-psfvw" Sep 11 09:39:35.568600 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:39:35.568450 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/9a3750ad-46fe-4bfd-ac42-fc9703e16e99-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-psfvw\" (UID: \"9a3750ad-46fe-4bfd-ac42-fc9703e16e99\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-psfvw" Sep 11 09:39:35.568600 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:39:35.568525 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j84zd\" (UniqueName: \"kubernetes.io/projected/9a3750ad-46fe-4bfd-ac42-fc9703e16e99-kube-api-access-j84zd\") pod \"feast-simple-feast-setup-57576b8fcf-psfvw\" (UID: \"9a3750ad-46fe-4bfd-ac42-fc9703e16e99\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-psfvw" Sep 11 09:39:35.568600 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:39:35.568577 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/9a3750ad-46fe-4bfd-ac42-fc9703e16e99-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-psfvw\" (UID: \"9a3750ad-46fe-4bfd-ac42-fc9703e16e99\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-psfvw" Sep 11 09:39:35.669401 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:39:35.669367 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/9a3750ad-46fe-4bfd-ac42-fc9703e16e99-feast-data\") pod \"feast-simple-feast-setup-57576b8fcf-psfvw\" (UID: \"9a3750ad-46fe-4bfd-ac42-fc9703e16e99\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-psfvw" Sep 11 09:39:35.669637 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:39:35.669416 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/9a3750ad-46fe-4bfd-ac42-fc9703e16e99-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-psfvw\" (UID: \"9a3750ad-46fe-4bfd-ac42-fc9703e16e99\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-psfvw" Sep 11 09:39:35.669637 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:39:35.669435 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/9a3750ad-46fe-4bfd-ac42-fc9703e16e99-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-psfvw\" (UID: \"9a3750ad-46fe-4bfd-ac42-fc9703e16e99\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-psfvw" Sep 11 09:39:35.669637 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:39:35.669522 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/9a3750ad-46fe-4bfd-ac42-fc9703e16e99-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-psfvw\" (UID: \"9a3750ad-46fe-4bfd-ac42-fc9703e16e99\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-psfvw" Sep 11 09:39:35.669637 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:39:35.669574 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j84zd\" (UniqueName: \"kubernetes.io/projected/9a3750ad-46fe-4bfd-ac42-fc9703e16e99-kube-api-access-j84zd\") pod \"feast-simple-feast-setup-57576b8fcf-psfvw\" (UID: \"9a3750ad-46fe-4bfd-ac42-fc9703e16e99\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-psfvw" Sep 11 09:39:35.669637 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:39:35.669537 2575 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-online-tls: secret "feast-simple-feast-setup-online-tls" not found Sep 11 09:39:35.669637 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:39:35.669619 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/9a3750ad-46fe-4bfd-ac42-fc9703e16e99-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-psfvw\" (UID: \"9a3750ad-46fe-4bfd-ac42-fc9703e16e99\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-psfvw" Sep 11 09:39:35.669967 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:39:35.669650 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a3750ad-46fe-4bfd-ac42-fc9703e16e99-online-tls podName:9a3750ad-46fe-4bfd-ac42-fc9703e16e99 nodeName:}" failed. No retries permitted until 2026-09-11 09:39:36.169629071 +0000 UTC m=+1592.510239277 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/9a3750ad-46fe-4bfd-ac42-fc9703e16e99-online-tls") pod "feast-simple-feast-setup-57576b8fcf-psfvw" (UID: "9a3750ad-46fe-4bfd-ac42-fc9703e16e99") : secret "feast-simple-feast-setup-online-tls" not found Sep 11 09:39:35.669967 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:39:35.669582 2575 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-offline-tls: secret "feast-simple-feast-setup-offline-tls" not found Sep 11 09:39:35.669967 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:39:35.669681 2575 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-registry-tls: secret "feast-simple-feast-setup-registry-tls" not found Sep 11 09:39:35.669967 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:39:35.669745 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a3750ad-46fe-4bfd-ac42-fc9703e16e99-registry-tls podName:9a3750ad-46fe-4bfd-ac42-fc9703e16e99 nodeName:}" failed. No retries permitted until 2026-09-11 09:39:36.169728305 +0000 UTC m=+1592.510338492 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/9a3750ad-46fe-4bfd-ac42-fc9703e16e99-registry-tls") pod "feast-simple-feast-setup-57576b8fcf-psfvw" (UID: "9a3750ad-46fe-4bfd-ac42-fc9703e16e99") : secret "feast-simple-feast-setup-registry-tls" not found Sep 11 09:39:35.669967 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:39:35.669761 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a3750ad-46fe-4bfd-ac42-fc9703e16e99-offline-tls podName:9a3750ad-46fe-4bfd-ac42-fc9703e16e99 nodeName:}" failed. No retries permitted until 2026-09-11 09:39:36.169752874 +0000 UTC m=+1592.510363059 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/9a3750ad-46fe-4bfd-ac42-fc9703e16e99-offline-tls") pod "feast-simple-feast-setup-57576b8fcf-psfvw" (UID: "9a3750ad-46fe-4bfd-ac42-fc9703e16e99") : secret "feast-simple-feast-setup-offline-tls" not found Sep 11 09:39:35.669967 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:39:35.669861 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/9a3750ad-46fe-4bfd-ac42-fc9703e16e99-feast-data\") pod \"feast-simple-feast-setup-57576b8fcf-psfvw\" (UID: \"9a3750ad-46fe-4bfd-ac42-fc9703e16e99\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-psfvw" Sep 11 09:39:35.671900 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:39:35.671873 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/9a3750ad-46fe-4bfd-ac42-fc9703e16e99-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-psfvw\" (UID: \"9a3750ad-46fe-4bfd-ac42-fc9703e16e99\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-psfvw" Sep 11 09:39:35.680322 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:39:35.680289 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j84zd\" (UniqueName: \"kubernetes.io/projected/9a3750ad-46fe-4bfd-ac42-fc9703e16e99-kube-api-access-j84zd\") pod \"feast-simple-feast-setup-57576b8fcf-psfvw\" (UID: \"9a3750ad-46fe-4bfd-ac42-fc9703e16e99\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-psfvw" Sep 11 09:39:36.173821 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:39:36.173776 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/9a3750ad-46fe-4bfd-ac42-fc9703e16e99-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-psfvw\" (UID: \"9a3750ad-46fe-4bfd-ac42-fc9703e16e99\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-psfvw" Sep 11 09:39:36.173821 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:39:36.173822 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/9a3750ad-46fe-4bfd-ac42-fc9703e16e99-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-psfvw\" (UID: \"9a3750ad-46fe-4bfd-ac42-fc9703e16e99\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-psfvw" Sep 11 09:39:36.174056 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:39:36.173879 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/9a3750ad-46fe-4bfd-ac42-fc9703e16e99-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-psfvw\" (UID: \"9a3750ad-46fe-4bfd-ac42-fc9703e16e99\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-psfvw" Sep 11 09:39:36.174056 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:39:36.173935 2575 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-online-tls: secret "feast-simple-feast-setup-online-tls" not found Sep 11 09:39:36.174056 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:39:36.174038 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a3750ad-46fe-4bfd-ac42-fc9703e16e99-online-tls podName:9a3750ad-46fe-4bfd-ac42-fc9703e16e99 nodeName:}" failed. No retries permitted until 2026-09-11 09:39:37.174019068 +0000 UTC m=+1593.514629269 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/9a3750ad-46fe-4bfd-ac42-fc9703e16e99-online-tls") pod "feast-simple-feast-setup-57576b8fcf-psfvw" (UID: "9a3750ad-46fe-4bfd-ac42-fc9703e16e99") : secret "feast-simple-feast-setup-online-tls" not found Sep 11 09:39:36.176406 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:39:36.176366 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/9a3750ad-46fe-4bfd-ac42-fc9703e16e99-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-psfvw\" (UID: \"9a3750ad-46fe-4bfd-ac42-fc9703e16e99\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-psfvw" Sep 11 09:39:36.176565 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:39:36.176543 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/9a3750ad-46fe-4bfd-ac42-fc9703e16e99-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-psfvw\" (UID: \"9a3750ad-46fe-4bfd-ac42-fc9703e16e99\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-psfvw" Sep 11 09:39:37.183645 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:39:37.183603 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/9a3750ad-46fe-4bfd-ac42-fc9703e16e99-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-psfvw\" (UID: \"9a3750ad-46fe-4bfd-ac42-fc9703e16e99\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-psfvw" Sep 11 09:39:37.185942 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:39:37.185919 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/9a3750ad-46fe-4bfd-ac42-fc9703e16e99-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-psfvw\" (UID: \"9a3750ad-46fe-4bfd-ac42-fc9703e16e99\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-psfvw" Sep 11 09:39:37.257973 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:39:37.257937 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-psfvw" Sep 11 09:39:37.534260 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:39:37.534224 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-psfvw"] Sep 11 09:39:37.537560 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:39:37.537530 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9a3750ad_46fe_4bfd_ac42_fc9703e16e99.slice/crio-855d1aa3db6aaa6932185d51c442d45f8c075e1f5b3350b024ff198a11301bc8 WatchSource:0}: Error finding container 855d1aa3db6aaa6932185d51c442d45f8c075e1f5b3350b024ff198a11301bc8: Status 404 returned error can't find the container with id 855d1aa3db6aaa6932185d51c442d45f8c075e1f5b3350b024ff198a11301bc8 Sep 11 09:39:38.251589 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:39:38.251551 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-psfvw" event={"ID":"9a3750ad-46fe-4bfd-ac42-fc9703e16e99","Type":"ContainerStarted","Data":"8837ae1c3364eaa89ec4f73241c84f7001b807d444f1ee9d8a77198563654939"} Sep 11 09:39:38.251589 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:39:38.251596 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-psfvw" event={"ID":"9a3750ad-46fe-4bfd-ac42-fc9703e16e99","Type":"ContainerStarted","Data":"855d1aa3db6aaa6932185d51c442d45f8c075e1f5b3350b024ff198a11301bc8"} Sep 11 09:39:43.274450 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:39:43.274417 2575 generic.go:356] "Generic (PLEG): container finished" podID="9a3750ad-46fe-4bfd-ac42-fc9703e16e99" containerID="8837ae1c3364eaa89ec4f73241c84f7001b807d444f1ee9d8a77198563654939" exitCode=0 Sep 11 09:39:43.274859 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:39:43.274458 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-psfvw" event={"ID":"9a3750ad-46fe-4bfd-ac42-fc9703e16e99","Type":"ContainerDied","Data":"8837ae1c3364eaa89ec4f73241c84f7001b807d444f1ee9d8a77198563654939"} Sep 11 09:39:44.279930 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:39:44.279895 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-psfvw" event={"ID":"9a3750ad-46fe-4bfd-ac42-fc9703e16e99","Type":"ContainerStarted","Data":"e0e71ec4335367e2380ddee7f31e3d435f585c48b925349bb46ef811da11e4e4"} Sep 11 09:39:50.308553 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:39:50.308517 2575 generic.go:356] "Generic (PLEG): container finished" podID="9a3750ad-46fe-4bfd-ac42-fc9703e16e99" containerID="e0e71ec4335367e2380ddee7f31e3d435f585c48b925349bb46ef811da11e4e4" exitCode=0 Sep 11 09:39:50.308935 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:39:50.308595 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-psfvw" event={"ID":"9a3750ad-46fe-4bfd-ac42-fc9703e16e99","Type":"ContainerDied","Data":"e0e71ec4335367e2380ddee7f31e3d435f585c48b925349bb46ef811da11e4e4"} Sep 11 09:39:51.319445 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:39:51.319399 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-psfvw" event={"ID":"9a3750ad-46fe-4bfd-ac42-fc9703e16e99","Type":"ContainerStarted","Data":"3e178e8454366429afb01297e99b6cf396b966d360260a825dc987507ec230c6"} Sep 11 09:39:51.319929 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:39:51.319485 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-psfvw" event={"ID":"9a3750ad-46fe-4bfd-ac42-fc9703e16e99","Type":"ContainerStarted","Data":"8131d9ebbc93b7a42930c1a4c6f3203c3f2e75dcf41cd9bd6f9a666b3c2ce3e3"} Sep 11 09:39:51.319929 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:39:51.319511 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-psfvw" event={"ID":"9a3750ad-46fe-4bfd-ac42-fc9703e16e99","Type":"ContainerStarted","Data":"2508a74706105bbb587a20b6dcfb2ce28774acea4af8a1bd7951be411fdedcac"} Sep 11 09:39:51.319929 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:39:51.319531 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-psfvw" event={"ID":"9a3750ad-46fe-4bfd-ac42-fc9703e16e99","Type":"ContainerStarted","Data":"8e707d417814b94837b641644c65f9b3438a2f67aca8ce324ffdbd3200663eca"} Sep 11 09:39:51.348172 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:39:51.348104 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-psfvw" podStartSLOduration=16.348081116 podStartE2EDuration="16.348081116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 09:39:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 09:39:51.34481696 +0000 UTC m=+1607.685427167" watchObservedRunningTime="2026-09-11 09:39:51.348081116 +0000 UTC m=+1607.688691324" Sep 11 09:39:52.258794 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:39:52.258698 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-psfvw" Sep 11 09:39:52.258794 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:39:52.258787 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-psfvw" Sep 11 09:39:52.259089 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:39:52.258810 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-psfvw" Sep 11 09:39:52.259089 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:39:52.258830 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-psfvw" Sep 11 09:39:52.261174 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:39:52.261123 2575 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-psfvw" podUID="9a3750ad-46fe-4bfd-ac42-fc9703e16e99" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.31:8443: connect: connection refused" Sep 11 09:39:52.261174 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:39:52.261144 2575 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-psfvw" podUID="9a3750ad-46fe-4bfd-ac42-fc9703e16e99" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.31:6571: connect: connection refused" Sep 11 09:39:52.261382 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:39:52.261122 2575 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-psfvw" podUID="9a3750ad-46fe-4bfd-ac42-fc9703e16e99" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.31:8816: connect: connection refused" Sep 11 09:39:52.261382 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:39:52.261129 2575 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-psfvw" podUID="9a3750ad-46fe-4bfd-ac42-fc9703e16e99" containerName="online" probeResult="failure" output="Get \"https://10.134.0.31:6567/health\": dial tcp 10.134.0.31:6567: connect: connection refused" Sep 11 09:39:55.258672 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:39:55.258434 2575 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-psfvw" podUID="9a3750ad-46fe-4bfd-ac42-fc9703e16e99" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.31:8443: connect: connection refused" Sep 11 09:39:55.258672 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:39:55.258517 2575 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-psfvw" podUID="9a3750ad-46fe-4bfd-ac42-fc9703e16e99" containerName="online" probeResult="failure" output="Get \"https://10.134.0.31:6567/health\": dial tcp 10.134.0.31:6567: connect: connection refused" Sep 11 09:39:55.258672 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:39:55.258435 2575 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-psfvw" podUID="9a3750ad-46fe-4bfd-ac42-fc9703e16e99" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.31:6571: connect: connection refused" Sep 11 09:39:55.258672 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:39:55.258620 2575 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-psfvw" podUID="9a3750ad-46fe-4bfd-ac42-fc9703e16e99" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.31:8816: connect: connection refused" Sep 11 09:39:57.258565 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:39:57.258448 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-psfvw" Sep 11 09:39:57.258565 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:39:57.258544 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-psfvw" Sep 11 09:39:57.258565 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:39:57.258571 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-psfvw" Sep 11 09:39:57.259168 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:39:57.258590 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-psfvw" Sep 11 09:39:58.258702 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:39:58.258575 2575 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-psfvw" podUID="9a3750ad-46fe-4bfd-ac42-fc9703e16e99" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.31:8443: connect: connection refused" Sep 11 09:39:58.258702 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:39:58.258616 2575 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-psfvw" podUID="9a3750ad-46fe-4bfd-ac42-fc9703e16e99" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.31:6571: connect: connection refused" Sep 11 09:39:58.258702 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:39:58.258627 2575 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-psfvw" podUID="9a3750ad-46fe-4bfd-ac42-fc9703e16e99" containerName="online" probeResult="failure" output="Get \"https://10.134.0.31:6567/health\": dial tcp 10.134.0.31:6567: connect: connection refused" Sep 11 09:39:58.259769 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:39:58.259745 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-psfvw" Sep 11 09:39:58.260690 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:39:58.260671 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-psfvw" Sep 11 09:40:01.259702 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:40:01.259672 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-psfvw" Sep 11 09:40:01.260084 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:40:01.259795 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-psfvw" Sep 11 09:40:01.260556 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:40:01.260533 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-psfvw" Sep 11 09:40:01.260673 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:40:01.260587 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-psfvw" Sep 11 09:40:01.264797 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:40:01.264781 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-psfvw" Sep 11 09:40:01.377954 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:40:01.377923 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-psfvw" Sep 11 09:41:03.616151 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:41:03.616117 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-psfvw"] Sep 11 09:41:03.618982 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:41:03.616387 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-psfvw" podUID="9a3750ad-46fe-4bfd-ac42-fc9703e16e99" containerName="registry" containerID="cri-o://8e707d417814b94837b641644c65f9b3438a2f67aca8ce324ffdbd3200663eca" gracePeriod=30 Sep 11 09:41:03.618982 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:41:03.616424 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-psfvw" podUID="9a3750ad-46fe-4bfd-ac42-fc9703e16e99" containerName="ui" containerID="cri-o://3e178e8454366429afb01297e99b6cf396b966d360260a825dc987507ec230c6" gracePeriod=30 Sep 11 09:41:03.618982 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:41:03.616436 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-psfvw" podUID="9a3750ad-46fe-4bfd-ac42-fc9703e16e99" containerName="offline" containerID="cri-o://8131d9ebbc93b7a42930c1a4c6f3203c3f2e75dcf41cd9bd6f9a666b3c2ce3e3" gracePeriod=30 Sep 11 09:41:03.618982 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:41:03.616447 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-psfvw" podUID="9a3750ad-46fe-4bfd-ac42-fc9703e16e99" containerName="online" containerID="cri-o://2508a74706105bbb587a20b6dcfb2ce28774acea4af8a1bd7951be411fdedcac" gracePeriod=30 Sep 11 09:41:04.068298 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:41:04.068255 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-snd7f"] Sep 11 09:41:04.072431 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:41:04.072405 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-snd7f" Sep 11 09:41:04.078339 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:41:04.078315 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-4lcv7\"" Sep 11 09:41:04.087086 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:41:04.087059 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-snd7f"] Sep 11 09:41:04.144598 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:41:04.144557 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/e167c209-6613-44ab-a772-2075234032a4-feast-data\") pod \"feast-simple-feast-setup-57576b8fcf-snd7f\" (UID: \"e167c209-6613-44ab-a772-2075234032a4\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-snd7f" Sep 11 09:41:04.144789 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:41:04.144647 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbkdh\" (UniqueName: \"kubernetes.io/projected/e167c209-6613-44ab-a772-2075234032a4-kube-api-access-tbkdh\") pod \"feast-simple-feast-setup-57576b8fcf-snd7f\" (UID: \"e167c209-6613-44ab-a772-2075234032a4\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-snd7f" Sep 11 09:41:04.144789 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:41:04.144676 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/e167c209-6613-44ab-a772-2075234032a4-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-snd7f\" (UID: \"e167c209-6613-44ab-a772-2075234032a4\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-snd7f" Sep 11 09:41:04.144789 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:41:04.144705 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/e167c209-6613-44ab-a772-2075234032a4-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-snd7f\" (UID: \"e167c209-6613-44ab-a772-2075234032a4\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-snd7f" Sep 11 09:41:04.144789 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:41:04.144763 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/e167c209-6613-44ab-a772-2075234032a4-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-snd7f\" (UID: \"e167c209-6613-44ab-a772-2075234032a4\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-snd7f" Sep 11 09:41:04.144947 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:41:04.144816 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/e167c209-6613-44ab-a772-2075234032a4-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-snd7f\" (UID: \"e167c209-6613-44ab-a772-2075234032a4\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-snd7f" Sep 11 09:41:04.246196 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:41:04.246156 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/e167c209-6613-44ab-a772-2075234032a4-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-snd7f\" (UID: \"e167c209-6613-44ab-a772-2075234032a4\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-snd7f" Sep 11 09:41:04.246196 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:41:04.246214 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/e167c209-6613-44ab-a772-2075234032a4-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-snd7f\" (UID: \"e167c209-6613-44ab-a772-2075234032a4\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-snd7f" Sep 11 09:41:04.246493 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:41:04.246236 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/e167c209-6613-44ab-a772-2075234032a4-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-snd7f\" (UID: \"e167c209-6613-44ab-a772-2075234032a4\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-snd7f" Sep 11 09:41:04.246493 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:41:04.246269 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/e167c209-6613-44ab-a772-2075234032a4-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-snd7f\" (UID: \"e167c209-6613-44ab-a772-2075234032a4\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-snd7f" Sep 11 09:41:04.246493 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:41:04.246311 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/e167c209-6613-44ab-a772-2075234032a4-feast-data\") pod \"feast-simple-feast-setup-57576b8fcf-snd7f\" (UID: \"e167c209-6613-44ab-a772-2075234032a4\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-snd7f" Sep 11 09:41:04.246493 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:41:04.246392 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbkdh\" (UniqueName: \"kubernetes.io/projected/e167c209-6613-44ab-a772-2075234032a4-kube-api-access-tbkdh\") pod \"feast-simple-feast-setup-57576b8fcf-snd7f\" (UID: \"e167c209-6613-44ab-a772-2075234032a4\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-snd7f" Sep 11 09:41:04.246493 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:41:04.246399 2575 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-offline-tls: secret "feast-simple-feast-setup-offline-tls" not found Sep 11 09:41:04.246768 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:41:04.246507 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e167c209-6613-44ab-a772-2075234032a4-offline-tls podName:e167c209-6613-44ab-a772-2075234032a4 nodeName:}" failed. No retries permitted until 2026-09-11 09:41:04.746460652 +0000 UTC m=+1681.087070842 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/e167c209-6613-44ab-a772-2075234032a4-offline-tls") pod "feast-simple-feast-setup-57576b8fcf-snd7f" (UID: "e167c209-6613-44ab-a772-2075234032a4") : secret "feast-simple-feast-setup-offline-tls" not found Sep 11 09:41:04.246940 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:41:04.246920 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/e167c209-6613-44ab-a772-2075234032a4-feast-data\") pod \"feast-simple-feast-setup-57576b8fcf-snd7f\" (UID: \"e167c209-6613-44ab-a772-2075234032a4\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-snd7f" Sep 11 09:41:04.248955 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:41:04.248923 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/e167c209-6613-44ab-a772-2075234032a4-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-snd7f\" (UID: \"e167c209-6613-44ab-a772-2075234032a4\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-snd7f" Sep 11 09:41:04.249384 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:41:04.249361 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/e167c209-6613-44ab-a772-2075234032a4-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-snd7f\" (UID: \"e167c209-6613-44ab-a772-2075234032a4\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-snd7f" Sep 11 09:41:04.249480 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:41:04.249450 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/e167c209-6613-44ab-a772-2075234032a4-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-snd7f\" (UID: \"e167c209-6613-44ab-a772-2075234032a4\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-snd7f" Sep 11 09:41:04.264436 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:41:04.264405 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbkdh\" (UniqueName: \"kubernetes.io/projected/e167c209-6613-44ab-a772-2075234032a4-kube-api-access-tbkdh\") pod \"feast-simple-feast-setup-57576b8fcf-snd7f\" (UID: \"e167c209-6613-44ab-a772-2075234032a4\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-snd7f" Sep 11 09:41:04.635378 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:41:04.635291 2575 generic.go:356] "Generic (PLEG): container finished" podID="9a3750ad-46fe-4bfd-ac42-fc9703e16e99" containerID="3e178e8454366429afb01297e99b6cf396b966d360260a825dc987507ec230c6" exitCode=0 Sep 11 09:41:04.635378 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:41:04.635318 2575 generic.go:356] "Generic (PLEG): container finished" podID="9a3750ad-46fe-4bfd-ac42-fc9703e16e99" containerID="2508a74706105bbb587a20b6dcfb2ce28774acea4af8a1bd7951be411fdedcac" exitCode=0 Sep 11 09:41:04.635378 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:41:04.635360 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-psfvw" event={"ID":"9a3750ad-46fe-4bfd-ac42-fc9703e16e99","Type":"ContainerDied","Data":"3e178e8454366429afb01297e99b6cf396b966d360260a825dc987507ec230c6"} Sep 11 09:41:04.635816 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:41:04.635404 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-psfvw" event={"ID":"9a3750ad-46fe-4bfd-ac42-fc9703e16e99","Type":"ContainerDied","Data":"2508a74706105bbb587a20b6dcfb2ce28774acea4af8a1bd7951be411fdedcac"} Sep 11 09:41:04.751579 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:41:04.751539 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/e167c209-6613-44ab-a772-2075234032a4-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-snd7f\" (UID: \"e167c209-6613-44ab-a772-2075234032a4\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-snd7f" Sep 11 09:41:04.753902 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:41:04.753883 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/e167c209-6613-44ab-a772-2075234032a4-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-snd7f\" (UID: \"e167c209-6613-44ab-a772-2075234032a4\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-snd7f" Sep 11 09:41:04.984052 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:41:04.983957 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-snd7f" Sep 11 09:41:05.263452 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:41:05.263421 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-snd7f"] Sep 11 09:41:05.264854 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:41:05.264828 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode167c209_6613_44ab_a772_2075234032a4.slice/crio-bab6d0c66819454625211fde17737eb5a86dd36002c3175ac678e941ac3a72e0 WatchSource:0}: Error finding container bab6d0c66819454625211fde17737eb5a86dd36002c3175ac678e941ac3a72e0: Status 404 returned error can't find the container with id bab6d0c66819454625211fde17737eb5a86dd36002c3175ac678e941ac3a72e0 Sep 11 09:41:05.643230 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:41:05.643133 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-snd7f" event={"ID":"e167c209-6613-44ab-a772-2075234032a4","Type":"ContainerStarted","Data":"e74d4baa051df34d265d5bc639bacbbf7f859139d726b0529fdcb18d71d3da30"} Sep 11 09:41:05.643230 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:41:05.643188 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-snd7f" event={"ID":"e167c209-6613-44ab-a772-2075234032a4","Type":"ContainerStarted","Data":"bab6d0c66819454625211fde17737eb5a86dd36002c3175ac678e941ac3a72e0"} Sep 11 09:41:10.668624 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:41:10.668533 2575 generic.go:356] "Generic (PLEG): container finished" podID="e167c209-6613-44ab-a772-2075234032a4" containerID="e74d4baa051df34d265d5bc639bacbbf7f859139d726b0529fdcb18d71d3da30" exitCode=0 Sep 11 09:41:10.668624 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:41:10.668607 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-snd7f" event={"ID":"e167c209-6613-44ab-a772-2075234032a4","Type":"ContainerDied","Data":"e74d4baa051df34d265d5bc639bacbbf7f859139d726b0529fdcb18d71d3da30"} Sep 11 09:41:11.260708 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:41:11.260667 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-psfvw" podUID="9a3750ad-46fe-4bfd-ac42-fc9703e16e99" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.31:8443: connect: connection refused" Sep 11 09:41:11.374987 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:41:11.374934 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-psfvw" podUID="9a3750ad-46fe-4bfd-ac42-fc9703e16e99" containerName="online" probeResult="failure" output="Get \"https://10.134.0.31:6567/health\": dial tcp 10.134.0.31:6567: connect: connection refused" Sep 11 09:41:11.674812 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:41:11.674723 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-snd7f" event={"ID":"e167c209-6613-44ab-a772-2075234032a4","Type":"ContainerStarted","Data":"e62c681910350f277d3ae03dc982fe42f267d02c4e1af771497e3c59bc08ea2c"} Sep 11 09:41:17.701998 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:41:17.701957 2575 generic.go:356] "Generic (PLEG): container finished" podID="e167c209-6613-44ab-a772-2075234032a4" containerID="e62c681910350f277d3ae03dc982fe42f267d02c4e1af771497e3c59bc08ea2c" exitCode=0 Sep 11 09:41:17.702357 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:41:17.702010 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-snd7f" event={"ID":"e167c209-6613-44ab-a772-2075234032a4","Type":"ContainerDied","Data":"e62c681910350f277d3ae03dc982fe42f267d02c4e1af771497e3c59bc08ea2c"} Sep 11 09:41:18.713355 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:41:18.713313 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-snd7f" event={"ID":"e167c209-6613-44ab-a772-2075234032a4","Type":"ContainerStarted","Data":"674b65b9086ca22a6ceefaad5661d2a2fafb60182ee657829451706274355305"} Sep 11 09:41:18.713855 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:41:18.713372 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-snd7f" event={"ID":"e167c209-6613-44ab-a772-2075234032a4","Type":"ContainerStarted","Data":"59e82930fe54ef9b3bc8852f783dd950b5ceb2117185364d7772fa06c078ae79"} Sep 11 09:41:18.713855 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:41:18.713392 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-snd7f" event={"ID":"e167c209-6613-44ab-a772-2075234032a4","Type":"ContainerStarted","Data":"32c827dee115e25189c933d55a5b1e22bf606a94cde0bd7141f258af7a9af708"} Sep 11 09:41:18.713855 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:41:18.713417 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-snd7f" event={"ID":"e167c209-6613-44ab-a772-2075234032a4","Type":"ContainerStarted","Data":"9ab9dc0a744b82afe8040bcbdfe91bda08838c672400d02961da0c156901f83f"} Sep 11 09:41:18.752511 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:41:18.752428 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-snd7f" podStartSLOduration=14.752405697 podStartE2EDuration="14.752405697s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 09:41:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 09:41:18.750685847 +0000 UTC m=+1695.091296054" watchObservedRunningTime="2026-09-11 09:41:18.752405697 +0000 UTC m=+1695.093015905" Sep 11 09:41:19.984559 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:41:19.984456 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-snd7f" Sep 11 09:41:19.984559 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:41:19.984546 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-snd7f" Sep 11 09:41:19.984559 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:41:19.984568 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-snd7f" Sep 11 09:41:19.985169 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:41:19.984590 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-snd7f" Sep 11 09:41:19.986667 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:41:19.986630 2575 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-snd7f" podUID="e167c209-6613-44ab-a772-2075234032a4" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.32:8816: connect: connection refused" Sep 11 09:41:19.986939 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:41:19.986661 2575 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-snd7f" podUID="e167c209-6613-44ab-a772-2075234032a4" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.32:6571: connect: connection refused" Sep 11 09:41:19.987065 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:41:19.986678 2575 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-snd7f" podUID="e167c209-6613-44ab-a772-2075234032a4" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.32:8443: connect: connection refused" Sep 11 09:41:19.987065 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:41:19.986739 2575 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-snd7f" podUID="e167c209-6613-44ab-a772-2075234032a4" containerName="online" probeResult="failure" output="Get \"https://10.134.0.32:6567/health\": dial tcp 10.134.0.32:6567: connect: connection refused" Sep 11 09:41:21.260954 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:41:21.260902 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-psfvw" podUID="9a3750ad-46fe-4bfd-ac42-fc9703e16e99" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.31:8443: connect: connection refused" Sep 11 09:41:21.374844 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:41:21.374789 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-psfvw" podUID="9a3750ad-46fe-4bfd-ac42-fc9703e16e99" containerName="online" probeResult="failure" output="Get \"https://10.134.0.31:6567/health\": dial tcp 10.134.0.31:6567: connect: connection refused" Sep 11 09:41:22.984952 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:41:22.984902 2575 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-snd7f" podUID="e167c209-6613-44ab-a772-2075234032a4" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.32:6571: connect: connection refused" Sep 11 09:41:22.985511 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:41:22.984903 2575 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-snd7f" podUID="e167c209-6613-44ab-a772-2075234032a4" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.32:8443: connect: connection refused" Sep 11 09:41:22.985640 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:41:22.984937 2575 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-snd7f" podUID="e167c209-6613-44ab-a772-2075234032a4" containerName="online" probeResult="failure" output="Get \"https://10.134.0.32:6567/health\": dial tcp 10.134.0.32:6567: connect: connection refused" Sep 11 09:41:22.985640 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:41:22.984950 2575 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-snd7f" podUID="e167c209-6613-44ab-a772-2075234032a4" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.32:8816: connect: connection refused" Sep 11 09:41:24.984690 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:41:24.984648 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-snd7f" Sep 11 09:41:24.984690 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:41:24.984711 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-snd7f" Sep 11 09:41:24.985252 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:41:24.984731 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-snd7f" Sep 11 09:41:24.985252 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:41:24.984750 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-snd7f" Sep 11 09:41:25.984592 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:41:25.984320 2575 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-snd7f" podUID="e167c209-6613-44ab-a772-2075234032a4" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.32:8443: connect: connection refused" Sep 11 09:41:25.985062 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:41:25.985035 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-snd7f" Sep 11 09:41:25.985701 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:41:25.985680 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-snd7f" Sep 11 09:41:25.986294 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:41:25.986271 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-snd7f" Sep 11 09:41:25.986537 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:41:25.986521 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-snd7f" Sep 11 09:41:25.991060 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:41:25.991036 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-snd7f" Sep 11 09:41:26.769951 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:41:26.769921 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-snd7f" Sep 11 09:41:28.984784 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:41:28.984750 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-snd7f" Sep 11 09:41:29.780813 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:41:29.780786 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-snd7f" Sep 11 09:41:31.260361 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:41:31.260308 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-psfvw" podUID="9a3750ad-46fe-4bfd-ac42-fc9703e16e99" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.31:8443: connect: connection refused" Sep 11 09:41:31.260877 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:41:31.260514 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-psfvw" Sep 11 09:41:31.375341 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:41:31.375288 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-psfvw" podUID="9a3750ad-46fe-4bfd-ac42-fc9703e16e99" containerName="online" probeResult="failure" output="Get \"https://10.134.0.31:6567/health\": dial tcp 10.134.0.31:6567: connect: connection refused" Sep 11 09:41:31.375518 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:41:31.375435 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-psfvw" Sep 11 09:41:33.800483 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:41:33.800437 2575 generic.go:356] "Generic (PLEG): container finished" podID="9a3750ad-46fe-4bfd-ac42-fc9703e16e99" containerID="8131d9ebbc93b7a42930c1a4c6f3203c3f2e75dcf41cd9bd6f9a666b3c2ce3e3" exitCode=137 Sep 11 09:41:33.800483 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:41:33.800462 2575 generic.go:356] "Generic (PLEG): container finished" podID="9a3750ad-46fe-4bfd-ac42-fc9703e16e99" containerID="8e707d417814b94837b641644c65f9b3438a2f67aca8ce324ffdbd3200663eca" exitCode=137 Sep 11 09:41:33.800894 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:41:33.800506 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-psfvw" event={"ID":"9a3750ad-46fe-4bfd-ac42-fc9703e16e99","Type":"ContainerDied","Data":"8131d9ebbc93b7a42930c1a4c6f3203c3f2e75dcf41cd9bd6f9a666b3c2ce3e3"} Sep 11 09:41:33.800894 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:41:33.800558 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-psfvw" event={"ID":"9a3750ad-46fe-4bfd-ac42-fc9703e16e99","Type":"ContainerDied","Data":"8e707d417814b94837b641644c65f9b3438a2f67aca8ce324ffdbd3200663eca"} Sep 11 09:41:34.270431 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:41:34.270406 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-psfvw" Sep 11 09:41:34.341231 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:41:34.341195 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j84zd\" (UniqueName: \"kubernetes.io/projected/9a3750ad-46fe-4bfd-ac42-fc9703e16e99-kube-api-access-j84zd\") pod \"9a3750ad-46fe-4bfd-ac42-fc9703e16e99\" (UID: \"9a3750ad-46fe-4bfd-ac42-fc9703e16e99\") " Sep 11 09:41:34.341396 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:41:34.341263 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/9a3750ad-46fe-4bfd-ac42-fc9703e16e99-online-tls\") pod \"9a3750ad-46fe-4bfd-ac42-fc9703e16e99\" (UID: \"9a3750ad-46fe-4bfd-ac42-fc9703e16e99\") " Sep 11 09:41:34.341396 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:41:34.341294 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/9a3750ad-46fe-4bfd-ac42-fc9703e16e99-offline-tls\") pod \"9a3750ad-46fe-4bfd-ac42-fc9703e16e99\" (UID: \"9a3750ad-46fe-4bfd-ac42-fc9703e16e99\") " Sep 11 09:41:34.341396 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:41:34.341355 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/9a3750ad-46fe-4bfd-ac42-fc9703e16e99-registry-tls\") pod \"9a3750ad-46fe-4bfd-ac42-fc9703e16e99\" (UID: \"9a3750ad-46fe-4bfd-ac42-fc9703e16e99\") " Sep 11 09:41:34.341396 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:41:34.341391 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/9a3750ad-46fe-4bfd-ac42-fc9703e16e99-feast-data\") pod \"9a3750ad-46fe-4bfd-ac42-fc9703e16e99\" (UID: \"9a3750ad-46fe-4bfd-ac42-fc9703e16e99\") " Sep 11 09:41:34.341604 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:41:34.341432 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/9a3750ad-46fe-4bfd-ac42-fc9703e16e99-ui-tls\") pod \"9a3750ad-46fe-4bfd-ac42-fc9703e16e99\" (UID: \"9a3750ad-46fe-4bfd-ac42-fc9703e16e99\") " Sep 11 09:41:34.342079 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:41:34.342002 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a3750ad-46fe-4bfd-ac42-fc9703e16e99-feast-data" (OuterVolumeSpecName: "feast-data") pod "9a3750ad-46fe-4bfd-ac42-fc9703e16e99" (UID: "9a3750ad-46fe-4bfd-ac42-fc9703e16e99"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 09:41:34.343554 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:41:34.343498 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a3750ad-46fe-4bfd-ac42-fc9703e16e99-kube-api-access-j84zd" (OuterVolumeSpecName: "kube-api-access-j84zd") pod "9a3750ad-46fe-4bfd-ac42-fc9703e16e99" (UID: "9a3750ad-46fe-4bfd-ac42-fc9703e16e99"). InnerVolumeSpecName "kube-api-access-j84zd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 09:41:34.343669 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:41:34.343597 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a3750ad-46fe-4bfd-ac42-fc9703e16e99-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "9a3750ad-46fe-4bfd-ac42-fc9703e16e99" (UID: "9a3750ad-46fe-4bfd-ac42-fc9703e16e99"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 09:41:34.343669 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:41:34.343613 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a3750ad-46fe-4bfd-ac42-fc9703e16e99-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "9a3750ad-46fe-4bfd-ac42-fc9703e16e99" (UID: "9a3750ad-46fe-4bfd-ac42-fc9703e16e99"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 09:41:34.343795 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:41:34.343773 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a3750ad-46fe-4bfd-ac42-fc9703e16e99-online-tls" (OuterVolumeSpecName: "online-tls") pod "9a3750ad-46fe-4bfd-ac42-fc9703e16e99" (UID: "9a3750ad-46fe-4bfd-ac42-fc9703e16e99"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 09:41:34.343858 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:41:34.343796 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a3750ad-46fe-4bfd-ac42-fc9703e16e99-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "9a3750ad-46fe-4bfd-ac42-fc9703e16e99" (UID: "9a3750ad-46fe-4bfd-ac42-fc9703e16e99"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 09:41:34.443057 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:41:34.442972 2575 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/9a3750ad-46fe-4bfd-ac42-fc9703e16e99-registry-tls\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\"" Sep 11 09:41:34.443057 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:41:34.443000 2575 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/9a3750ad-46fe-4bfd-ac42-fc9703e16e99-feast-data\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\"" Sep 11 09:41:34.443057 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:41:34.443008 2575 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/9a3750ad-46fe-4bfd-ac42-fc9703e16e99-ui-tls\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\"" Sep 11 09:41:34.443057 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:41:34.443017 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j84zd\" (UniqueName: \"kubernetes.io/projected/9a3750ad-46fe-4bfd-ac42-fc9703e16e99-kube-api-access-j84zd\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\"" Sep 11 09:41:34.443057 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:41:34.443026 2575 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/9a3750ad-46fe-4bfd-ac42-fc9703e16e99-online-tls\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\"" Sep 11 09:41:34.443057 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:41:34.443034 2575 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/9a3750ad-46fe-4bfd-ac42-fc9703e16e99-offline-tls\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\"" Sep 11 09:41:34.808118 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:41:34.808075 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-psfvw" event={"ID":"9a3750ad-46fe-4bfd-ac42-fc9703e16e99","Type":"ContainerDied","Data":"855d1aa3db6aaa6932185d51c442d45f8c075e1f5b3350b024ff198a11301bc8"} Sep 11 09:41:34.808656 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:41:34.808141 2575 scope.go:117] "RemoveContainer" containerID="3e178e8454366429afb01297e99b6cf396b966d360260a825dc987507ec230c6" Sep 11 09:41:34.808656 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:41:34.808098 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-psfvw" Sep 11 09:41:34.817040 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:41:34.817020 2575 scope.go:117] "RemoveContainer" containerID="8131d9ebbc93b7a42930c1a4c6f3203c3f2e75dcf41cd9bd6f9a666b3c2ce3e3" Sep 11 09:41:34.824869 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:41:34.824851 2575 scope.go:117] "RemoveContainer" containerID="2508a74706105bbb587a20b6dcfb2ce28774acea4af8a1bd7951be411fdedcac" Sep 11 09:41:34.830566 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:41:34.830542 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-psfvw"] Sep 11 09:41:34.833520 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:41:34.833497 2575 scope.go:117] "RemoveContainer" containerID="8e707d417814b94837b641644c65f9b3438a2f67aca8ce324ffdbd3200663eca" Sep 11 09:41:34.834623 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:41:34.834605 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-psfvw"] Sep 11 09:41:34.841344 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:41:34.841328 2575 scope.go:117] "RemoveContainer" containerID="e0e71ec4335367e2380ddee7f31e3d435f585c48b925349bb46ef811da11e4e4" Sep 11 09:41:34.848924 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:41:34.848905 2575 scope.go:117] "RemoveContainer" containerID="8837ae1c3364eaa89ec4f73241c84f7001b807d444f1ee9d8a77198563654939" Sep 11 09:41:36.114273 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:41:36.114233 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9a3750ad-46fe-4bfd-ac42-fc9703e16e99" path="/var/lib/kubelet/pods/9a3750ad-46fe-4bfd-ac42-fc9703e16e99/volumes" Sep 11 09:42:32.851848 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:42:32.851807 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-sssrl"] Sep 11 09:42:32.852276 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:42:32.852181 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a3750ad-46fe-4bfd-ac42-fc9703e16e99" containerName="feast-init" Sep 11 09:42:32.852276 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:42:32.852193 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a3750ad-46fe-4bfd-ac42-fc9703e16e99" containerName="feast-init" Sep 11 09:42:32.852276 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:42:32.852203 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a3750ad-46fe-4bfd-ac42-fc9703e16e99" containerName="online" Sep 11 09:42:32.852276 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:42:32.852209 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a3750ad-46fe-4bfd-ac42-fc9703e16e99" containerName="online" Sep 11 09:42:32.852276 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:42:32.852219 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a3750ad-46fe-4bfd-ac42-fc9703e16e99" containerName="offline" Sep 11 09:42:32.852276 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:42:32.852224 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a3750ad-46fe-4bfd-ac42-fc9703e16e99" containerName="offline" Sep 11 09:42:32.852276 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:42:32.852230 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a3750ad-46fe-4bfd-ac42-fc9703e16e99" containerName="ui" Sep 11 09:42:32.852276 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:42:32.852235 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a3750ad-46fe-4bfd-ac42-fc9703e16e99" containerName="ui" Sep 11 09:42:32.852276 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:42:32.852256 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a3750ad-46fe-4bfd-ac42-fc9703e16e99" containerName="feast-apply" Sep 11 09:42:32.852276 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:42:32.852261 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a3750ad-46fe-4bfd-ac42-fc9703e16e99" containerName="feast-apply" Sep 11 09:42:32.852276 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:42:32.852266 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a3750ad-46fe-4bfd-ac42-fc9703e16e99" containerName="registry" Sep 11 09:42:32.852276 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:42:32.852271 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a3750ad-46fe-4bfd-ac42-fc9703e16e99" containerName="registry" Sep 11 09:42:32.852675 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:42:32.852319 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a3750ad-46fe-4bfd-ac42-fc9703e16e99" containerName="online" Sep 11 09:42:32.852675 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:42:32.852326 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a3750ad-46fe-4bfd-ac42-fc9703e16e99" containerName="ui" Sep 11 09:42:32.852675 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:42:32.852333 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a3750ad-46fe-4bfd-ac42-fc9703e16e99" containerName="offline" Sep 11 09:42:32.852675 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:42:32.852340 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a3750ad-46fe-4bfd-ac42-fc9703e16e99" containerName="registry" Sep 11 09:42:32.855909 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:42:32.855888 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-sssrl" Sep 11 09:42:32.861521 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:42:32.861498 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"openshift-service-ca.crt\"" Sep 11 09:42:32.861734 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:42:32.861713 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-online-tls\"" Sep 11 09:42:32.861821 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:42:32.861751 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-client-ca\"" Sep 11 09:42:32.861928 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:42:32.861911 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-ui-tls\"" Sep 11 09:42:32.862182 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:42:32.862165 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-offline-tls\"" Sep 11 09:42:32.862258 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:42:32.862169 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-dockercfg-czxq2\"" Sep 11 09:42:32.862740 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:42:32.862726 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"kube-root-ca.crt\"" Sep 11 09:42:32.880600 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:42:32.880571 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-sssrl"] Sep 11 09:42:32.956081 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:42:32.956036 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/176e1214-6eeb-4c70-b64c-59ad98b1b2ed-feast-data\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-sssrl\" (UID: \"176e1214-6eeb-4c70-b64c-59ad98b1b2ed\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-sssrl" Sep 11 09:42:32.956265 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:42:32.956085 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/176e1214-6eeb-4c70-b64c-59ad98b1b2ed-registry-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-sssrl\" (UID: \"176e1214-6eeb-4c70-b64c-59ad98b1b2ed\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-sssrl" Sep 11 09:42:32.956265 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:42:32.956171 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/176e1214-6eeb-4c70-b64c-59ad98b1b2ed-offline-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-sssrl\" (UID: \"176e1214-6eeb-4c70-b64c-59ad98b1b2ed\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-sssrl" Sep 11 09:42:32.956265 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:42:32.956209 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdxch\" (UniqueName: \"kubernetes.io/projected/176e1214-6eeb-4c70-b64c-59ad98b1b2ed-kube-api-access-fdxch\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-sssrl\" (UID: \"176e1214-6eeb-4c70-b64c-59ad98b1b2ed\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-sssrl" Sep 11 09:42:32.956371 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:42:32.956282 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/176e1214-6eeb-4c70-b64c-59ad98b1b2ed-ui-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-sssrl\" (UID: \"176e1214-6eeb-4c70-b64c-59ad98b1b2ed\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-sssrl" Sep 11 09:42:32.956371 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:42:32.956320 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/176e1214-6eeb-4c70-b64c-59ad98b1b2ed-online-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-sssrl\" (UID: \"176e1214-6eeb-4c70-b64c-59ad98b1b2ed\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-sssrl" Sep 11 09:42:33.056886 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:42:33.056852 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/176e1214-6eeb-4c70-b64c-59ad98b1b2ed-online-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-sssrl\" (UID: \"176e1214-6eeb-4c70-b64c-59ad98b1b2ed\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-sssrl" Sep 11 09:42:33.057067 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:42:33.056902 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/176e1214-6eeb-4c70-b64c-59ad98b1b2ed-feast-data\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-sssrl\" (UID: \"176e1214-6eeb-4c70-b64c-59ad98b1b2ed\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-sssrl" Sep 11 09:42:33.057067 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:42:33.056926 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/176e1214-6eeb-4c70-b64c-59ad98b1b2ed-registry-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-sssrl\" (UID: \"176e1214-6eeb-4c70-b64c-59ad98b1b2ed\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-sssrl" Sep 11 09:42:33.057067 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:42:33.056948 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/176e1214-6eeb-4c70-b64c-59ad98b1b2ed-offline-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-sssrl\" (UID: \"176e1214-6eeb-4c70-b64c-59ad98b1b2ed\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-sssrl" Sep 11 09:42:33.057067 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:42:33.056977 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdxch\" (UniqueName: \"kubernetes.io/projected/176e1214-6eeb-4c70-b64c-59ad98b1b2ed-kube-api-access-fdxch\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-sssrl\" (UID: \"176e1214-6eeb-4c70-b64c-59ad98b1b2ed\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-sssrl" Sep 11 09:42:33.057293 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:42:33.057117 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/176e1214-6eeb-4c70-b64c-59ad98b1b2ed-ui-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-sssrl\" (UID: \"176e1214-6eeb-4c70-b64c-59ad98b1b2ed\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-sssrl" Sep 11 09:42:33.057449 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:42:33.057418 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/176e1214-6eeb-4c70-b64c-59ad98b1b2ed-feast-data\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-sssrl\" (UID: \"176e1214-6eeb-4c70-b64c-59ad98b1b2ed\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-sssrl" Sep 11 09:42:33.057661 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:42:33.057641 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/176e1214-6eeb-4c70-b64c-59ad98b1b2ed-registry-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-sssrl\" (UID: \"176e1214-6eeb-4c70-b64c-59ad98b1b2ed\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-sssrl" Sep 11 09:42:33.059371 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:42:33.059351 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/176e1214-6eeb-4c70-b64c-59ad98b1b2ed-offline-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-sssrl\" (UID: \"176e1214-6eeb-4c70-b64c-59ad98b1b2ed\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-sssrl" Sep 11 09:42:33.059490 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:42:33.059443 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/176e1214-6eeb-4c70-b64c-59ad98b1b2ed-ui-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-sssrl\" (UID: \"176e1214-6eeb-4c70-b64c-59ad98b1b2ed\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-sssrl" Sep 11 09:42:33.059538 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:42:33.059521 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/176e1214-6eeb-4c70-b64c-59ad98b1b2ed-online-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-sssrl\" (UID: \"176e1214-6eeb-4c70-b64c-59ad98b1b2ed\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-sssrl" Sep 11 09:42:33.067601 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:42:33.067580 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdxch\" (UniqueName: \"kubernetes.io/projected/176e1214-6eeb-4c70-b64c-59ad98b1b2ed-kube-api-access-fdxch\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-sssrl\" (UID: \"176e1214-6eeb-4c70-b64c-59ad98b1b2ed\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-sssrl" Sep 11 09:42:33.165027 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:42:33.164919 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-sssrl" Sep 11 09:42:33.450036 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:42:33.449916 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-sssrl"] Sep 11 09:42:33.452220 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:42:33.452192 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod176e1214_6eeb_4c70_b64c_59ad98b1b2ed.slice/crio-cfaed88b1db5991bd1cfba2226914276dd168cec442ccce9b4f93d30021e5932 WatchSource:0}: Error finding container cfaed88b1db5991bd1cfba2226914276dd168cec442ccce9b4f93d30021e5932: Status 404 returned error can't find the container with id cfaed88b1db5991bd1cfba2226914276dd168cec442ccce9b4f93d30021e5932 Sep 11 09:42:34.059747 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:42:34.059707 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-sssrl" event={"ID":"176e1214-6eeb-4c70-b64c-59ad98b1b2ed","Type":"ContainerStarted","Data":"61edce64d382ea697721e020c8d8ced05ce2e8ca457871eb0e2bc91bbaa846b9"} Sep 11 09:42:34.060230 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:42:34.059759 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-sssrl" event={"ID":"176e1214-6eeb-4c70-b64c-59ad98b1b2ed","Type":"ContainerStarted","Data":"cfaed88b1db5991bd1cfba2226914276dd168cec442ccce9b4f93d30021e5932"} Sep 11 09:42:39.082651 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:42:39.082613 2575 generic.go:356] "Generic (PLEG): container finished" podID="176e1214-6eeb-4c70-b64c-59ad98b1b2ed" containerID="61edce64d382ea697721e020c8d8ced05ce2e8ca457871eb0e2bc91bbaa846b9" exitCode=0 Sep 11 09:42:39.083025 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:42:39.082674 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-sssrl" event={"ID":"176e1214-6eeb-4c70-b64c-59ad98b1b2ed","Type":"ContainerDied","Data":"61edce64d382ea697721e020c8d8ced05ce2e8ca457871eb0e2bc91bbaa846b9"} Sep 11 09:42:40.088753 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:42:40.088716 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-sssrl" event={"ID":"176e1214-6eeb-4c70-b64c-59ad98b1b2ed","Type":"ContainerStarted","Data":"64339e2b2e13cb3074c5cf85ce5d20e318fec3ddfa432240142b27938cdb6731"} Sep 11 09:43:03.192769 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:43:03.192737 2575 generic.go:356] "Generic (PLEG): container finished" podID="176e1214-6eeb-4c70-b64c-59ad98b1b2ed" containerID="64339e2b2e13cb3074c5cf85ce5d20e318fec3ddfa432240142b27938cdb6731" exitCode=0 Sep 11 09:43:03.193177 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:43:03.192810 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-sssrl" event={"ID":"176e1214-6eeb-4c70-b64c-59ad98b1b2ed","Type":"ContainerDied","Data":"64339e2b2e13cb3074c5cf85ce5d20e318fec3ddfa432240142b27938cdb6731"} Sep 11 09:43:04.202147 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:43:04.202108 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-sssrl" event={"ID":"176e1214-6eeb-4c70-b64c-59ad98b1b2ed","Type":"ContainerStarted","Data":"5f64ef8a426d81cd95e865f00c84a4224918018d6ee8efd3afa8a550e0248c84"} Sep 11 09:43:04.202571 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:43:04.202172 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-sssrl" event={"ID":"176e1214-6eeb-4c70-b64c-59ad98b1b2ed","Type":"ContainerStarted","Data":"1cbb4dcf3bce6df70de2e51291071124628b8174694da1806ac83caac796dd38"} Sep 11 09:43:04.202571 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:43:04.202192 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-sssrl" event={"ID":"176e1214-6eeb-4c70-b64c-59ad98b1b2ed","Type":"ContainerStarted","Data":"c26010b6e0610f7e4355cd5957c204555a85c09603f02d19cfc3458950724a3b"} Sep 11 09:43:04.237229 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:43:04.237140 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-sssrl" podStartSLOduration=32.237118742 podStartE2EDuration="32.237118742s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 09:42:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 09:43:04.235533977 +0000 UTC m=+1800.576144185" watchObservedRunningTime="2026-09-11 09:43:04.237118742 +0000 UTC m=+1800.577728949" Sep 11 09:43:06.165554 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:43:06.165515 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-sssrl" Sep 11 09:43:06.166044 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:43:06.165659 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-sssrl" Sep 11 09:43:06.166044 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:43:06.165682 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-sssrl" Sep 11 09:43:06.167553 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:43:06.167520 2575 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-sssrl" podUID="176e1214-6eeb-4c70-b64c-59ad98b1b2ed" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.33:8443: connect: connection refused" Sep 11 09:43:06.167685 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:43:06.167566 2575 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-sssrl" podUID="176e1214-6eeb-4c70-b64c-59ad98b1b2ed" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.33:8816: connect: connection refused" Sep 11 09:43:06.167685 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:43:06.167607 2575 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-sssrl" podUID="176e1214-6eeb-4c70-b64c-59ad98b1b2ed" containerName="online" probeResult="failure" output="Get \"https://10.134.0.33:6567/health\": dial tcp 10.134.0.33:6567: connect: connection refused" Sep 11 09:43:09.165276 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:43:09.165223 2575 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-sssrl" podUID="176e1214-6eeb-4c70-b64c-59ad98b1b2ed" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.33:8816: connect: connection refused" Sep 11 09:43:09.165276 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:43:09.165260 2575 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-sssrl" podUID="176e1214-6eeb-4c70-b64c-59ad98b1b2ed" containerName="online" probeResult="failure" output="Get \"https://10.134.0.33:6567/health\": dial tcp 10.134.0.33:6567: connect: connection refused" Sep 11 09:43:09.165744 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:43:09.165223 2575 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-sssrl" podUID="176e1214-6eeb-4c70-b64c-59ad98b1b2ed" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.33:8443: connect: connection refused" Sep 11 09:43:12.166023 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:43:12.165973 2575 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-sssrl" podUID="176e1214-6eeb-4c70-b64c-59ad98b1b2ed" containerName="online" probeResult="failure" output="Get \"https://10.134.0.33:6567/health\": dial tcp 10.134.0.33:6567: connect: connection refused" Sep 11 09:43:12.166726 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:43:12.166700 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-sssrl" Sep 11 09:43:12.166958 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:43:12.166938 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-sssrl" Sep 11 09:43:12.167287 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:43:12.167270 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-sssrl" Sep 11 09:43:12.167357 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:43:12.167303 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-sssrl" Sep 11 09:43:12.167462 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:43:12.167444 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-sssrl" Sep 11 09:43:12.167531 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:43:12.167520 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-sssrl" Sep 11 09:43:13.165773 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:43:13.165734 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-sssrl" Sep 11 09:43:16.115159 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:43:16.115122 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-sssrl" Sep 11 09:43:16.966136 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:43:16.966102 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-sssrl" Sep 11 09:44:19.042842 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:44:19.042801 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-sssrl"] Sep 11 09:44:19.043424 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:44:19.043031 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-sssrl" podUID="176e1214-6eeb-4c70-b64c-59ad98b1b2ed" containerName="online" containerID="cri-o://c26010b6e0610f7e4355cd5957c204555a85c09603f02d19cfc3458950724a3b" gracePeriod=30 Sep 11 09:44:19.043424 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:44:19.043100 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-sssrl" podUID="176e1214-6eeb-4c70-b64c-59ad98b1b2ed" containerName="offline" containerID="cri-o://1cbb4dcf3bce6df70de2e51291071124628b8174694da1806ac83caac796dd38" gracePeriod=30 Sep 11 09:44:19.043424 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:44:19.043100 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-sssrl" podUID="176e1214-6eeb-4c70-b64c-59ad98b1b2ed" containerName="ui" containerID="cri-o://5f64ef8a426d81cd95e865f00c84a4224918018d6ee8efd3afa8a550e0248c84" gracePeriod=30 Sep 11 09:44:19.216407 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:44:19.216367 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-snd7f"] Sep 11 09:44:19.216646 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:44:19.216614 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-snd7f" podUID="e167c209-6613-44ab-a772-2075234032a4" containerName="registry" containerID="cri-o://9ab9dc0a744b82afe8040bcbdfe91bda08838c672400d02961da0c156901f83f" gracePeriod=30 Sep 11 09:44:19.216738 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:44:19.216678 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-snd7f" podUID="e167c209-6613-44ab-a772-2075234032a4" containerName="ui" containerID="cri-o://674b65b9086ca22a6ceefaad5661d2a2fafb60182ee657829451706274355305" gracePeriod=30 Sep 11 09:44:19.216738 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:44:19.216702 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-snd7f" podUID="e167c209-6613-44ab-a772-2075234032a4" containerName="online" containerID="cri-o://32c827dee115e25189c933d55a5b1e22bf606a94cde0bd7141f258af7a9af708" gracePeriod=30 Sep 11 09:44:19.216827 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:44:19.216678 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-snd7f" podUID="e167c209-6613-44ab-a772-2075234032a4" containerName="offline" containerID="cri-o://59e82930fe54ef9b3bc8852f783dd950b5ceb2117185364d7772fa06c078ae79" gracePeriod=30 Sep 11 09:44:19.781149 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:44:19.781096 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-snd7f" podUID="e167c209-6613-44ab-a772-2075234032a4" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.32:8443: connect: connection refused" Sep 11 09:44:20.542785 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:44:20.542749 2575 generic.go:356] "Generic (PLEG): container finished" podID="e167c209-6613-44ab-a772-2075234032a4" containerID="674b65b9086ca22a6ceefaad5661d2a2fafb60182ee657829451706274355305" exitCode=0 Sep 11 09:44:20.542785 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:44:20.542775 2575 generic.go:356] "Generic (PLEG): container finished" podID="e167c209-6613-44ab-a772-2075234032a4" containerID="32c827dee115e25189c933d55a5b1e22bf606a94cde0bd7141f258af7a9af708" exitCode=0 Sep 11 09:44:20.543271 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:44:20.542789 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-snd7f" event={"ID":"e167c209-6613-44ab-a772-2075234032a4","Type":"ContainerDied","Data":"674b65b9086ca22a6ceefaad5661d2a2fafb60182ee657829451706274355305"} Sep 11 09:44:20.543271 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:44:20.542844 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-snd7f" event={"ID":"e167c209-6613-44ab-a772-2075234032a4","Type":"ContainerDied","Data":"32c827dee115e25189c933d55a5b1e22bf606a94cde0bd7141f258af7a9af708"} Sep 11 09:44:20.545723 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:44:20.545702 2575 generic.go:356] "Generic (PLEG): container finished" podID="176e1214-6eeb-4c70-b64c-59ad98b1b2ed" containerID="5f64ef8a426d81cd95e865f00c84a4224918018d6ee8efd3afa8a550e0248c84" exitCode=0 Sep 11 09:44:20.545723 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:44:20.545719 2575 generic.go:356] "Generic (PLEG): container finished" podID="176e1214-6eeb-4c70-b64c-59ad98b1b2ed" containerID="c26010b6e0610f7e4355cd5957c204555a85c09603f02d19cfc3458950724a3b" exitCode=0 Sep 11 09:44:20.545901 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:44:20.545749 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-sssrl" event={"ID":"176e1214-6eeb-4c70-b64c-59ad98b1b2ed","Type":"ContainerDied","Data":"5f64ef8a426d81cd95e865f00c84a4224918018d6ee8efd3afa8a550e0248c84"} Sep 11 09:44:20.545901 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:44:20.545794 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-sssrl" event={"ID":"176e1214-6eeb-4c70-b64c-59ad98b1b2ed","Type":"ContainerDied","Data":"c26010b6e0610f7e4355cd5957c204555a85c09603f02d19cfc3458950724a3b"} Sep 11 09:44:22.167725 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:44:22.167680 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-sssrl" podUID="176e1214-6eeb-4c70-b64c-59ad98b1b2ed" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.33:8443: connect: connection refused" Sep 11 09:44:26.116509 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:44:26.116454 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-sssrl" podUID="176e1214-6eeb-4c70-b64c-59ad98b1b2ed" containerName="online" probeResult="failure" output="Get \"https://10.134.0.33:6567/health\": dial tcp 10.134.0.33:6567: connect: connection refused" Sep 11 09:44:26.767592 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:44:26.767547 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-snd7f" podUID="e167c209-6613-44ab-a772-2075234032a4" containerName="online" probeResult="failure" output="Get \"https://10.134.0.32:6567/health\": dial tcp 10.134.0.32:6567: connect: connection refused" Sep 11 09:44:29.780681 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:44:29.780630 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-snd7f" podUID="e167c209-6613-44ab-a772-2075234032a4" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.32:8443: connect: connection refused" Sep 11 09:44:32.167961 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:44:32.167909 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-sssrl" podUID="176e1214-6eeb-4c70-b64c-59ad98b1b2ed" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.33:8443: connect: connection refused" Sep 11 09:44:36.116705 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:44:36.116664 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-sssrl" podUID="176e1214-6eeb-4c70-b64c-59ad98b1b2ed" containerName="online" probeResult="failure" output="Get \"https://10.134.0.33:6567/health\": dial tcp 10.134.0.33:6567: connect: connection refused" Sep 11 09:44:36.767301 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:44:36.767254 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-snd7f" podUID="e167c209-6613-44ab-a772-2075234032a4" containerName="online" probeResult="failure" output="Get \"https://10.134.0.32:6567/health\": dial tcp 10.134.0.32:6567: connect: connection refused" Sep 11 09:44:39.780547 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:44:39.780502 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-snd7f" podUID="e167c209-6613-44ab-a772-2075234032a4" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.32:8443: connect: connection refused" Sep 11 09:44:39.780939 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:44:39.780640 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-snd7f" Sep 11 09:44:42.168214 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:44:42.168160 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-sssrl" podUID="176e1214-6eeb-4c70-b64c-59ad98b1b2ed" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.33:8443: connect: connection refused" Sep 11 09:44:42.168657 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:44:42.168292 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-sssrl" Sep 11 09:44:46.115853 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:44:46.115815 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-sssrl" podUID="176e1214-6eeb-4c70-b64c-59ad98b1b2ed" containerName="online" probeResult="failure" output="Get \"https://10.134.0.33:6567/health\": dial tcp 10.134.0.33:6567: connect: connection refused" Sep 11 09:44:46.116257 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:44:46.115938 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-sssrl" Sep 11 09:44:46.767377 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:44:46.767334 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-snd7f" podUID="e167c209-6613-44ab-a772-2075234032a4" containerName="online" probeResult="failure" output="Get \"https://10.134.0.32:6567/health\": dial tcp 10.134.0.32:6567: connect: connection refused" Sep 11 09:44:46.767585 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:44:46.767462 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-snd7f" Sep 11 09:44:49.666810 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:44:49.666763 2575 generic.go:356] "Generic (PLEG): container finished" podID="e167c209-6613-44ab-a772-2075234032a4" containerID="59e82930fe54ef9b3bc8852f783dd950b5ceb2117185364d7772fa06c078ae79" exitCode=137 Sep 11 09:44:49.666810 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:44:49.666792 2575 generic.go:356] "Generic (PLEG): container finished" podID="e167c209-6613-44ab-a772-2075234032a4" containerID="9ab9dc0a744b82afe8040bcbdfe91bda08838c672400d02961da0c156901f83f" exitCode=137 Sep 11 09:44:49.667284 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:44:49.666835 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-snd7f" event={"ID":"e167c209-6613-44ab-a772-2075234032a4","Type":"ContainerDied","Data":"59e82930fe54ef9b3bc8852f783dd950b5ceb2117185364d7772fa06c078ae79"} Sep 11 09:44:49.667284 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:44:49.666891 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-snd7f" event={"ID":"e167c209-6613-44ab-a772-2075234032a4","Type":"ContainerDied","Data":"9ab9dc0a744b82afe8040bcbdfe91bda08838c672400d02961da0c156901f83f"} Sep 11 09:44:49.669962 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:44:49.669936 2575 generic.go:356] "Generic (PLEG): container finished" podID="176e1214-6eeb-4c70-b64c-59ad98b1b2ed" containerID="1cbb4dcf3bce6df70de2e51291071124628b8174694da1806ac83caac796dd38" exitCode=137 Sep 11 09:44:49.670094 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:44:49.670007 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-sssrl" event={"ID":"176e1214-6eeb-4c70-b64c-59ad98b1b2ed","Type":"ContainerDied","Data":"1cbb4dcf3bce6df70de2e51291071124628b8174694da1806ac83caac796dd38"} Sep 11 09:44:49.688806 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:44:49.688778 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-sssrl" Sep 11 09:44:49.780431 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:44:49.780394 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-snd7f" podUID="e167c209-6613-44ab-a772-2075234032a4" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.32:8443: connect: connection refused" Sep 11 09:44:49.817456 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:44:49.817362 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/176e1214-6eeb-4c70-b64c-59ad98b1b2ed-feast-data\") pod \"176e1214-6eeb-4c70-b64c-59ad98b1b2ed\" (UID: \"176e1214-6eeb-4c70-b64c-59ad98b1b2ed\") " Sep 11 09:44:49.817456 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:44:49.817406 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/176e1214-6eeb-4c70-b64c-59ad98b1b2ed-offline-tls\") pod \"176e1214-6eeb-4c70-b64c-59ad98b1b2ed\" (UID: \"176e1214-6eeb-4c70-b64c-59ad98b1b2ed\") " Sep 11 09:44:49.817704 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:44:49.817481 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/176e1214-6eeb-4c70-b64c-59ad98b1b2ed-ui-tls\") pod \"176e1214-6eeb-4c70-b64c-59ad98b1b2ed\" (UID: \"176e1214-6eeb-4c70-b64c-59ad98b1b2ed\") " Sep 11 09:44:49.817704 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:44:49.817597 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fdxch\" (UniqueName: \"kubernetes.io/projected/176e1214-6eeb-4c70-b64c-59ad98b1b2ed-kube-api-access-fdxch\") pod \"176e1214-6eeb-4c70-b64c-59ad98b1b2ed\" (UID: \"176e1214-6eeb-4c70-b64c-59ad98b1b2ed\") " Sep 11 09:44:49.817704 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:44:49.817651 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/176e1214-6eeb-4c70-b64c-59ad98b1b2ed-online-tls\") pod \"176e1214-6eeb-4c70-b64c-59ad98b1b2ed\" (UID: \"176e1214-6eeb-4c70-b64c-59ad98b1b2ed\") " Sep 11 09:44:49.817704 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:44:49.817689 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/176e1214-6eeb-4c70-b64c-59ad98b1b2ed-registry-tls\") pod \"176e1214-6eeb-4c70-b64c-59ad98b1b2ed\" (UID: \"176e1214-6eeb-4c70-b64c-59ad98b1b2ed\") " Sep 11 09:44:49.817979 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:44:49.817946 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/176e1214-6eeb-4c70-b64c-59ad98b1b2ed-feast-data" (OuterVolumeSpecName: "feast-data") pod "176e1214-6eeb-4c70-b64c-59ad98b1b2ed" (UID: "176e1214-6eeb-4c70-b64c-59ad98b1b2ed"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 09:44:49.818240 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:44:49.818201 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/176e1214-6eeb-4c70-b64c-59ad98b1b2ed-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "176e1214-6eeb-4c70-b64c-59ad98b1b2ed" (UID: "176e1214-6eeb-4c70-b64c-59ad98b1b2ed"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 09:44:49.819809 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:44:49.819780 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/176e1214-6eeb-4c70-b64c-59ad98b1b2ed-kube-api-access-fdxch" (OuterVolumeSpecName: "kube-api-access-fdxch") pod "176e1214-6eeb-4c70-b64c-59ad98b1b2ed" (UID: "176e1214-6eeb-4c70-b64c-59ad98b1b2ed"). InnerVolumeSpecName "kube-api-access-fdxch". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 09:44:49.819903 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:44:49.819832 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/176e1214-6eeb-4c70-b64c-59ad98b1b2ed-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "176e1214-6eeb-4c70-b64c-59ad98b1b2ed" (UID: "176e1214-6eeb-4c70-b64c-59ad98b1b2ed"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 09:44:49.819903 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:44:49.819848 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/176e1214-6eeb-4c70-b64c-59ad98b1b2ed-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "176e1214-6eeb-4c70-b64c-59ad98b1b2ed" (UID: "176e1214-6eeb-4c70-b64c-59ad98b1b2ed"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 09:44:49.820107 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:44:49.820089 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/176e1214-6eeb-4c70-b64c-59ad98b1b2ed-online-tls" (OuterVolumeSpecName: "online-tls") pod "176e1214-6eeb-4c70-b64c-59ad98b1b2ed" (UID: "176e1214-6eeb-4c70-b64c-59ad98b1b2ed"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 09:44:49.862569 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:44:49.862544 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-snd7f" Sep 11 09:44:49.918555 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:44:49.918521 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/e167c209-6613-44ab-a772-2075234032a4-ui-tls\") pod \"e167c209-6613-44ab-a772-2075234032a4\" (UID: \"e167c209-6613-44ab-a772-2075234032a4\") " Sep 11 09:44:49.918555 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:44:49.918563 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/e167c209-6613-44ab-a772-2075234032a4-registry-tls\") pod \"e167c209-6613-44ab-a772-2075234032a4\" (UID: \"e167c209-6613-44ab-a772-2075234032a4\") " Sep 11 09:44:49.918824 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:44:49.918613 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/e167c209-6613-44ab-a772-2075234032a4-offline-tls\") pod \"e167c209-6613-44ab-a772-2075234032a4\" (UID: \"e167c209-6613-44ab-a772-2075234032a4\") " Sep 11 09:44:49.918824 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:44:49.918662 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/e167c209-6613-44ab-a772-2075234032a4-online-tls\") pod \"e167c209-6613-44ab-a772-2075234032a4\" (UID: \"e167c209-6613-44ab-a772-2075234032a4\") " Sep 11 09:44:49.918824 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:44:49.918691 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tbkdh\" (UniqueName: \"kubernetes.io/projected/e167c209-6613-44ab-a772-2075234032a4-kube-api-access-tbkdh\") pod \"e167c209-6613-44ab-a772-2075234032a4\" (UID: \"e167c209-6613-44ab-a772-2075234032a4\") " Sep 11 09:44:49.918824 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:44:49.918721 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/e167c209-6613-44ab-a772-2075234032a4-feast-data\") pod \"e167c209-6613-44ab-a772-2075234032a4\" (UID: \"e167c209-6613-44ab-a772-2075234032a4\") " Sep 11 09:44:49.919035 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:44:49.918964 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdxch\" (UniqueName: \"kubernetes.io/projected/176e1214-6eeb-4c70-b64c-59ad98b1b2ed-kube-api-access-fdxch\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\"" Sep 11 09:44:49.919035 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:44:49.919014 2575 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/176e1214-6eeb-4c70-b64c-59ad98b1b2ed-online-tls\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\"" Sep 11 09:44:49.919035 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:44:49.919027 2575 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/176e1214-6eeb-4c70-b64c-59ad98b1b2ed-registry-tls\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\"" Sep 11 09:44:49.919240 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:44:49.919039 2575 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/176e1214-6eeb-4c70-b64c-59ad98b1b2ed-feast-data\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\"" Sep 11 09:44:49.919240 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:44:49.919050 2575 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/176e1214-6eeb-4c70-b64c-59ad98b1b2ed-offline-tls\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\"" Sep 11 09:44:49.919240 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:44:49.919073 2575 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/176e1214-6eeb-4c70-b64c-59ad98b1b2ed-ui-tls\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\"" Sep 11 09:44:49.919497 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:44:49.919449 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e167c209-6613-44ab-a772-2075234032a4-feast-data" (OuterVolumeSpecName: "feast-data") pod "e167c209-6613-44ab-a772-2075234032a4" (UID: "e167c209-6613-44ab-a772-2075234032a4"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 09:44:49.920871 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:44:49.920840 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e167c209-6613-44ab-a772-2075234032a4-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "e167c209-6613-44ab-a772-2075234032a4" (UID: "e167c209-6613-44ab-a772-2075234032a4"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 09:44:49.920871 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:44:49.920860 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e167c209-6613-44ab-a772-2075234032a4-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "e167c209-6613-44ab-a772-2075234032a4" (UID: "e167c209-6613-44ab-a772-2075234032a4"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 09:44:49.921043 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:44:49.920894 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e167c209-6613-44ab-a772-2075234032a4-online-tls" (OuterVolumeSpecName: "online-tls") pod "e167c209-6613-44ab-a772-2075234032a4" (UID: "e167c209-6613-44ab-a772-2075234032a4"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 09:44:49.921198 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:44:49.921168 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e167c209-6613-44ab-a772-2075234032a4-kube-api-access-tbkdh" (OuterVolumeSpecName: "kube-api-access-tbkdh") pod "e167c209-6613-44ab-a772-2075234032a4" (UID: "e167c209-6613-44ab-a772-2075234032a4"). InnerVolumeSpecName "kube-api-access-tbkdh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 09:44:49.921333 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:44:49.921320 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e167c209-6613-44ab-a772-2075234032a4-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "e167c209-6613-44ab-a772-2075234032a4" (UID: "e167c209-6613-44ab-a772-2075234032a4"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 09:44:50.019934 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:44:50.019893 2575 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/e167c209-6613-44ab-a772-2075234032a4-online-tls\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\"" Sep 11 09:44:50.019934 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:44:50.019927 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbkdh\" (UniqueName: \"kubernetes.io/projected/e167c209-6613-44ab-a772-2075234032a4-kube-api-access-tbkdh\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\"" Sep 11 09:44:50.019934 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:44:50.019939 2575 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/e167c209-6613-44ab-a772-2075234032a4-feast-data\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\"" Sep 11 09:44:50.020175 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:44:50.019948 2575 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/e167c209-6613-44ab-a772-2075234032a4-ui-tls\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\"" Sep 11 09:44:50.020175 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:44:50.019956 2575 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/e167c209-6613-44ab-a772-2075234032a4-registry-tls\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\"" Sep 11 09:44:50.020175 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:44:50.019964 2575 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/e167c209-6613-44ab-a772-2075234032a4-offline-tls\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\"" Sep 11 09:44:50.677602 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:44:50.677572 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-snd7f" Sep 11 09:44:50.678071 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:44:50.677573 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-snd7f" event={"ID":"e167c209-6613-44ab-a772-2075234032a4","Type":"ContainerDied","Data":"bab6d0c66819454625211fde17737eb5a86dd36002c3175ac678e941ac3a72e0"} Sep 11 09:44:50.678071 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:44:50.677737 2575 scope.go:117] "RemoveContainer" containerID="674b65b9086ca22a6ceefaad5661d2a2fafb60182ee657829451706274355305" Sep 11 09:44:50.681128 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:44:50.681092 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-sssrl" event={"ID":"176e1214-6eeb-4c70-b64c-59ad98b1b2ed","Type":"ContainerDied","Data":"cfaed88b1db5991bd1cfba2226914276dd168cec442ccce9b4f93d30021e5932"} Sep 11 09:44:50.681128 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:44:50.681121 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-sssrl" Sep 11 09:44:50.687871 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:44:50.687854 2575 scope.go:117] "RemoveContainer" containerID="59e82930fe54ef9b3bc8852f783dd950b5ceb2117185364d7772fa06c078ae79" Sep 11 09:44:50.697559 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:44:50.697536 2575 scope.go:117] "RemoveContainer" containerID="32c827dee115e25189c933d55a5b1e22bf606a94cde0bd7141f258af7a9af708" Sep 11 09:44:50.700699 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:44:50.700674 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-snd7f"] Sep 11 09:44:50.703717 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:44:50.703697 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-snd7f"] Sep 11 09:44:50.706288 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:44:50.706270 2575 scope.go:117] "RemoveContainer" containerID="9ab9dc0a744b82afe8040bcbdfe91bda08838c672400d02961da0c156901f83f" Sep 11 09:44:50.714161 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:44:50.714138 2575 scope.go:117] "RemoveContainer" containerID="e62c681910350f277d3ae03dc982fe42f267d02c4e1af771497e3c59bc08ea2c" Sep 11 09:44:50.716287 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:44:50.716262 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-sssrl"] Sep 11 09:44:50.719479 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:44:50.719443 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-sssrl"] Sep 11 09:44:50.722529 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:44:50.722510 2575 scope.go:117] "RemoveContainer" containerID="e74d4baa051df34d265d5bc639bacbbf7f859139d726b0529fdcb18d71d3da30" Sep 11 09:44:50.730361 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:44:50.730341 2575 scope.go:117] "RemoveContainer" containerID="5f64ef8a426d81cd95e865f00c84a4224918018d6ee8efd3afa8a550e0248c84" Sep 11 09:44:50.737698 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:44:50.737680 2575 scope.go:117] "RemoveContainer" containerID="1cbb4dcf3bce6df70de2e51291071124628b8174694da1806ac83caac796dd38" Sep 11 09:44:50.745326 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:44:50.745306 2575 scope.go:117] "RemoveContainer" containerID="c26010b6e0610f7e4355cd5957c204555a85c09603f02d19cfc3458950724a3b" Sep 11 09:44:50.752487 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:44:50.752450 2575 scope.go:117] "RemoveContainer" containerID="64339e2b2e13cb3074c5cf85ce5d20e318fec3ddfa432240142b27938cdb6731" Sep 11 09:44:50.759622 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:44:50.759603 2575 scope.go:117] "RemoveContainer" containerID="61edce64d382ea697721e020c8d8ced05ce2e8ca457871eb0e2bc91bbaa846b9" Sep 11 09:44:52.114306 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:44:52.114271 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="176e1214-6eeb-4c70-b64c-59ad98b1b2ed" path="/var/lib/kubelet/pods/176e1214-6eeb-4c70-b64c-59ad98b1b2ed/volumes" Sep 11 09:44:52.114857 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:44:52.114842 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e167c209-6613-44ab-a772-2075234032a4" path="/var/lib/kubelet/pods/e167c209-6613-44ab-a772-2075234032a4/volumes" Sep 11 09:45:08.493886 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:08.493849 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-4fwjt/must-gather-v6l9w"] Sep 11 09:45:08.494399 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:08.494345 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="176e1214-6eeb-4c70-b64c-59ad98b1b2ed" containerName="online" Sep 11 09:45:08.494399 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:08.494365 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="176e1214-6eeb-4c70-b64c-59ad98b1b2ed" containerName="online" Sep 11 09:45:08.494399 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:08.494380 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e167c209-6613-44ab-a772-2075234032a4" containerName="feast-init" Sep 11 09:45:08.494399 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:08.494389 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e167c209-6613-44ab-a772-2075234032a4" containerName="feast-init" Sep 11 09:45:08.494399 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:08.494402 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e167c209-6613-44ab-a772-2075234032a4" containerName="offline" Sep 11 09:45:08.494709 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:08.494410 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e167c209-6613-44ab-a772-2075234032a4" containerName="offline" Sep 11 09:45:08.494709 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:08.494425 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="176e1214-6eeb-4c70-b64c-59ad98b1b2ed" containerName="offline" Sep 11 09:45:08.494709 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:08.494434 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="176e1214-6eeb-4c70-b64c-59ad98b1b2ed" containerName="offline" Sep 11 09:45:08.494709 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:08.494456 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="176e1214-6eeb-4c70-b64c-59ad98b1b2ed" containerName="ui" Sep 11 09:45:08.494709 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:08.494482 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="176e1214-6eeb-4c70-b64c-59ad98b1b2ed" containerName="ui" Sep 11 09:45:08.494709 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:08.494495 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e167c209-6613-44ab-a772-2075234032a4" containerName="feast-apply" Sep 11 09:45:08.494709 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:08.494505 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e167c209-6613-44ab-a772-2075234032a4" containerName="feast-apply" Sep 11 09:45:08.494709 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:08.494516 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e167c209-6613-44ab-a772-2075234032a4" containerName="online" Sep 11 09:45:08.494709 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:08.494524 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e167c209-6613-44ab-a772-2075234032a4" containerName="online" Sep 11 09:45:08.494709 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:08.494535 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e167c209-6613-44ab-a772-2075234032a4" containerName="registry" Sep 11 09:45:08.494709 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:08.494544 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e167c209-6613-44ab-a772-2075234032a4" containerName="registry" Sep 11 09:45:08.494709 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:08.494553 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="176e1214-6eeb-4c70-b64c-59ad98b1b2ed" containerName="feast-init" Sep 11 09:45:08.494709 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:08.494561 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="176e1214-6eeb-4c70-b64c-59ad98b1b2ed" containerName="feast-init" Sep 11 09:45:08.494709 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:08.494578 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e167c209-6613-44ab-a772-2075234032a4" containerName="ui" Sep 11 09:45:08.494709 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:08.494587 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e167c209-6613-44ab-a772-2075234032a4" containerName="ui" Sep 11 09:45:08.494709 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:08.494598 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="176e1214-6eeb-4c70-b64c-59ad98b1b2ed" containerName="feast-apply" Sep 11 09:45:08.494709 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:08.494606 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="176e1214-6eeb-4c70-b64c-59ad98b1b2ed" containerName="feast-apply" Sep 11 09:45:08.494709 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:08.494685 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="e167c209-6613-44ab-a772-2075234032a4" containerName="registry" Sep 11 09:45:08.494709 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:08.494700 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="e167c209-6613-44ab-a772-2075234032a4" containerName="offline" Sep 11 09:45:08.494709 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:08.494708 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="176e1214-6eeb-4c70-b64c-59ad98b1b2ed" containerName="ui" Sep 11 09:45:08.494709 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:08.494717 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="176e1214-6eeb-4c70-b64c-59ad98b1b2ed" containerName="offline" Sep 11 09:45:08.495751 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:08.494727 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="e167c209-6613-44ab-a772-2075234032a4" containerName="ui" Sep 11 09:45:08.495751 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:08.494736 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="e167c209-6613-44ab-a772-2075234032a4" containerName="online" Sep 11 09:45:08.495751 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:08.494745 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="176e1214-6eeb-4c70-b64c-59ad98b1b2ed" containerName="online" Sep 11 09:45:08.498036 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:08.498016 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4fwjt/must-gather-v6l9w" Sep 11 09:45:08.500600 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:08.500574 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4fwjt\"/\"kube-root-ca.crt\"" Sep 11 09:45:08.501551 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:08.501532 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-4fwjt\"/\"default-dockercfg-mtdpj\"" Sep 11 09:45:08.501627 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:08.501541 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4fwjt\"/\"openshift-service-ca.crt\"" Sep 11 09:45:08.505140 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:08.505119 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4fwjt/must-gather-v6l9w"] Sep 11 09:45:08.579832 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:08.579795 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/dff6a025-7b07-4f35-a5d0-a300818e4a4a-must-gather-output\") pod \"must-gather-v6l9w\" (UID: \"dff6a025-7b07-4f35-a5d0-a300818e4a4a\") " pod="openshift-must-gather-4fwjt/must-gather-v6l9w" Sep 11 09:45:08.579997 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:08.579861 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssdrl\" (UniqueName: \"kubernetes.io/projected/dff6a025-7b07-4f35-a5d0-a300818e4a4a-kube-api-access-ssdrl\") pod \"must-gather-v6l9w\" (UID: \"dff6a025-7b07-4f35-a5d0-a300818e4a4a\") " pod="openshift-must-gather-4fwjt/must-gather-v6l9w" Sep 11 09:45:08.680392 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:08.680349 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssdrl\" (UniqueName: \"kubernetes.io/projected/dff6a025-7b07-4f35-a5d0-a300818e4a4a-kube-api-access-ssdrl\") pod \"must-gather-v6l9w\" (UID: \"dff6a025-7b07-4f35-a5d0-a300818e4a4a\") " pod="openshift-must-gather-4fwjt/must-gather-v6l9w" Sep 11 09:45:08.680595 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:08.680432 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/dff6a025-7b07-4f35-a5d0-a300818e4a4a-must-gather-output\") pod \"must-gather-v6l9w\" (UID: \"dff6a025-7b07-4f35-a5d0-a300818e4a4a\") " pod="openshift-must-gather-4fwjt/must-gather-v6l9w" Sep 11 09:45:08.680783 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:08.680766 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/dff6a025-7b07-4f35-a5d0-a300818e4a4a-must-gather-output\") pod \"must-gather-v6l9w\" (UID: \"dff6a025-7b07-4f35-a5d0-a300818e4a4a\") " pod="openshift-must-gather-4fwjt/must-gather-v6l9w" Sep 11 09:45:08.689137 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:08.689102 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssdrl\" (UniqueName: \"kubernetes.io/projected/dff6a025-7b07-4f35-a5d0-a300818e4a4a-kube-api-access-ssdrl\") pod \"must-gather-v6l9w\" (UID: \"dff6a025-7b07-4f35-a5d0-a300818e4a4a\") " pod="openshift-must-gather-4fwjt/must-gather-v6l9w" Sep 11 09:45:08.819622 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:08.819586 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4fwjt/must-gather-v6l9w" Sep 11 09:45:09.091418 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:09.091367 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4fwjt/must-gather-v6l9w"] Sep 11 09:45:09.093624 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:45:09.093582 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddff6a025_7b07_4f35_a5d0_a300818e4a4a.slice/crio-c89a84b44de5018ea1e09095427b4bf3d331028547da023da6575e63f4c52267 WatchSource:0}: Error finding container c89a84b44de5018ea1e09095427b4bf3d331028547da023da6575e63f4c52267: Status 404 returned error can't find the container with id c89a84b44de5018ea1e09095427b4bf3d331028547da023da6575e63f4c52267 Sep 11 09:45:09.095366 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:09.095350 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 09:45:09.761327 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:09.761285 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4fwjt/must-gather-v6l9w" event={"ID":"dff6a025-7b07-4f35-a5d0-a300818e4a4a","Type":"ContainerStarted","Data":"c89a84b44de5018ea1e09095427b4bf3d331028547da023da6575e63f4c52267"} Sep 11 09:45:15.792085 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:15.792045 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4fwjt/must-gather-v6l9w" event={"ID":"dff6a025-7b07-4f35-a5d0-a300818e4a4a","Type":"ContainerStarted","Data":"65226f25e62b3dae1ee3066fe3ca60124c8d4c361faa1dcae5d4d30251527dad"} Sep 11 09:45:15.792527 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:15.792101 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4fwjt/must-gather-v6l9w" event={"ID":"dff6a025-7b07-4f35-a5d0-a300818e4a4a","Type":"ContainerStarted","Data":"9de348278e61ab2ee73fa73919e46aa07dbe6f4378b238e5ed2044191fc777a7"} Sep 11 09:45:15.810500 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:15.810420 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-4fwjt/must-gather-v6l9w" podStartSLOduration=1.756119015 podStartE2EDuration="7.810403053s" totalImagesPullingTime="6.054284038s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 09:45:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 09:45:15.808984653 +0000 UTC m=+1932.149594863" watchObservedRunningTime="2026-09-11 09:45:15.810403053 +0000 UTC m=+1932.151013262" Sep 11 09:45:25.833792 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:25.833755 2575 generic.go:356] "Generic (PLEG): container finished" podID="dff6a025-7b07-4f35-a5d0-a300818e4a4a" containerID="9de348278e61ab2ee73fa73919e46aa07dbe6f4378b238e5ed2044191fc777a7" exitCode=0 Sep 11 09:45:25.834235 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:25.833827 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4fwjt/must-gather-v6l9w" event={"ID":"dff6a025-7b07-4f35-a5d0-a300818e4a4a","Type":"ContainerDied","Data":"9de348278e61ab2ee73fa73919e46aa07dbe6f4378b238e5ed2044191fc777a7"} Sep 11 09:45:25.834235 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:25.834136 2575 scope.go:117] "RemoveContainer" containerID="9de348278e61ab2ee73fa73919e46aa07dbe6f4378b238e5ed2044191fc777a7" Sep 11 09:45:25.900273 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:25.900244 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-4fwjt_must-gather-v6l9w_dff6a025-7b07-4f35-a5d0-a300818e4a4a/gather/0.log" Sep 11 09:45:29.664382 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:29.664354 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-8ng6h_7eddd8ad-0ac3-421d-9649-95dc4fadfd7b/global-pull-secret-syncer/0.log" Sep 11 09:45:29.825255 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:29.825213 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-pq626_fc69178e-852e-4853-bb53-01b784cd8a3a/konnectivity-agent/0.log" Sep 11 09:45:29.845153 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:29.845106 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-119.ec2.internal_ce93b5afb1de6b9a80d64df7da81462d/haproxy/0.log" Sep 11 09:45:31.227240 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:31.227203 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-4fwjt/must-gather-v6l9w"] Sep 11 09:45:31.227708 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:31.227413 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-4fwjt/must-gather-v6l9w" podUID="dff6a025-7b07-4f35-a5d0-a300818e4a4a" containerName="copy" containerID="cri-o://65226f25e62b3dae1ee3066fe3ca60124c8d4c361faa1dcae5d4d30251527dad" gracePeriod=2 Sep 11 09:45:31.231111 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:31.231080 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-4fwjt/must-gather-v6l9w"] Sep 11 09:45:31.462678 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:31.462655 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-4fwjt_must-gather-v6l9w_dff6a025-7b07-4f35-a5d0-a300818e4a4a/copy/0.log" Sep 11 09:45:31.464534 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:31.464518 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4fwjt/must-gather-v6l9w" Sep 11 09:45:31.466708 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:45:31.466682 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-v6l9w\" is forbidden: User \"system:node:ip-10-0-135-119.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-4fwjt\": no relationship found between node 'ip-10-0-135-119.ec2.internal' and this object" podUID="dff6a025-7b07-4f35-a5d0-a300818e4a4a" pod="openshift-must-gather-4fwjt/must-gather-v6l9w" Sep 11 09:45:31.602360 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:31.602328 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ssdrl\" (UniqueName: \"kubernetes.io/projected/dff6a025-7b07-4f35-a5d0-a300818e4a4a-kube-api-access-ssdrl\") pod \"dff6a025-7b07-4f35-a5d0-a300818e4a4a\" (UID: \"dff6a025-7b07-4f35-a5d0-a300818e4a4a\") " Sep 11 09:45:31.602534 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:31.602424 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/dff6a025-7b07-4f35-a5d0-a300818e4a4a-must-gather-output\") pod \"dff6a025-7b07-4f35-a5d0-a300818e4a4a\" (UID: \"dff6a025-7b07-4f35-a5d0-a300818e4a4a\") " Sep 11 09:45:31.602808 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:31.602777 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dff6a025-7b07-4f35-a5d0-a300818e4a4a-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "dff6a025-7b07-4f35-a5d0-a300818e4a4a" (UID: "dff6a025-7b07-4f35-a5d0-a300818e4a4a"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 09:45:31.604554 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:31.604530 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dff6a025-7b07-4f35-a5d0-a300818e4a4a-kube-api-access-ssdrl" (OuterVolumeSpecName: "kube-api-access-ssdrl") pod "dff6a025-7b07-4f35-a5d0-a300818e4a4a" (UID: "dff6a025-7b07-4f35-a5d0-a300818e4a4a"). InnerVolumeSpecName "kube-api-access-ssdrl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 09:45:31.703050 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:31.703010 2575 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/dff6a025-7b07-4f35-a5d0-a300818e4a4a-must-gather-output\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\"" Sep 11 09:45:31.703050 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:31.703042 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ssdrl\" (UniqueName: \"kubernetes.io/projected/dff6a025-7b07-4f35-a5d0-a300818e4a4a-kube-api-access-ssdrl\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\"" Sep 11 09:45:31.857807 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:31.857730 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-4fwjt_must-gather-v6l9w_dff6a025-7b07-4f35-a5d0-a300818e4a4a/copy/0.log" Sep 11 09:45:31.860266 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:31.860182 2575 generic.go:356] "Generic (PLEG): container finished" podID="dff6a025-7b07-4f35-a5d0-a300818e4a4a" containerID="65226f25e62b3dae1ee3066fe3ca60124c8d4c361faa1dcae5d4d30251527dad" exitCode=143 Sep 11 09:45:31.860266 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:31.860240 2575 scope.go:117] "RemoveContainer" containerID="65226f25e62b3dae1ee3066fe3ca60124c8d4c361faa1dcae5d4d30251527dad" Sep 11 09:45:31.860482 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:31.860346 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4fwjt/must-gather-v6l9w" Sep 11 09:45:31.865416 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:45:31.865386 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-v6l9w\" is forbidden: User \"system:node:ip-10-0-135-119.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-4fwjt\": no relationship found between node 'ip-10-0-135-119.ec2.internal' and this object" podUID="dff6a025-7b07-4f35-a5d0-a300818e4a4a" pod="openshift-must-gather-4fwjt/must-gather-v6l9w" Sep 11 09:45:31.868247 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:31.868228 2575 scope.go:117] "RemoveContainer" containerID="9de348278e61ab2ee73fa73919e46aa07dbe6f4378b238e5ed2044191fc777a7" Sep 11 09:45:31.870562 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:45:31.870540 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-v6l9w\" is forbidden: User \"system:node:ip-10-0-135-119.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-4fwjt\": no relationship found between node 'ip-10-0-135-119.ec2.internal' and this object" podUID="dff6a025-7b07-4f35-a5d0-a300818e4a4a" pod="openshift-must-gather-4fwjt/must-gather-v6l9w" Sep 11 09:45:31.879158 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:31.879141 2575 scope.go:117] "RemoveContainer" containerID="65226f25e62b3dae1ee3066fe3ca60124c8d4c361faa1dcae5d4d30251527dad" Sep 11 09:45:31.879431 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:45:31.879413 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65226f25e62b3dae1ee3066fe3ca60124c8d4c361faa1dcae5d4d30251527dad\": container with ID starting with 65226f25e62b3dae1ee3066fe3ca60124c8d4c361faa1dcae5d4d30251527dad not found: ID does not exist" containerID="65226f25e62b3dae1ee3066fe3ca60124c8d4c361faa1dcae5d4d30251527dad" Sep 11 09:45:31.879494 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:31.879439 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65226f25e62b3dae1ee3066fe3ca60124c8d4c361faa1dcae5d4d30251527dad"} err="failed to get container status \"65226f25e62b3dae1ee3066fe3ca60124c8d4c361faa1dcae5d4d30251527dad\": rpc error: code = NotFound desc = could not find container \"65226f25e62b3dae1ee3066fe3ca60124c8d4c361faa1dcae5d4d30251527dad\": container with ID starting with 65226f25e62b3dae1ee3066fe3ca60124c8d4c361faa1dcae5d4d30251527dad not found: ID does not exist" Sep 11 09:45:31.879494 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:31.879455 2575 scope.go:117] "RemoveContainer" containerID="9de348278e61ab2ee73fa73919e46aa07dbe6f4378b238e5ed2044191fc777a7" Sep 11 09:45:31.879703 ip-10-0-135-119 kubenswrapper[2575]: E0911 09:45:31.879687 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9de348278e61ab2ee73fa73919e46aa07dbe6f4378b238e5ed2044191fc777a7\": container with ID starting with 9de348278e61ab2ee73fa73919e46aa07dbe6f4378b238e5ed2044191fc777a7 not found: ID does not exist" containerID="9de348278e61ab2ee73fa73919e46aa07dbe6f4378b238e5ed2044191fc777a7" Sep 11 09:45:31.879742 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:31.879708 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9de348278e61ab2ee73fa73919e46aa07dbe6f4378b238e5ed2044191fc777a7"} err="failed to get container status \"9de348278e61ab2ee73fa73919e46aa07dbe6f4378b238e5ed2044191fc777a7\": rpc error: code = NotFound desc = could not find container \"9de348278e61ab2ee73fa73919e46aa07dbe6f4378b238e5ed2044191fc777a7\": container with ID starting with 9de348278e61ab2ee73fa73919e46aa07dbe6f4378b238e5ed2044191fc777a7 not found: ID does not exist" Sep 11 09:45:32.114116 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:32.114034 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dff6a025-7b07-4f35-a5d0-a300818e4a4a" path="/var/lib/kubelet/pods/dff6a025-7b07-4f35-a5d0-a300818e4a4a/volumes" Sep 11 09:45:33.838225 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:33.838194 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_bbea5b4b-7d0d-4bbb-be9b-25fe1a714e3f/alertmanager/0.log" Sep 11 09:45:33.866995 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:33.866971 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_bbea5b4b-7d0d-4bbb-be9b-25fe1a714e3f/config-reloader/0.log" Sep 11 09:45:33.894175 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:33.894152 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_bbea5b4b-7d0d-4bbb-be9b-25fe1a714e3f/kube-rbac-proxy-web/0.log" Sep 11 09:45:33.923144 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:33.923121 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_bbea5b4b-7d0d-4bbb-be9b-25fe1a714e3f/kube-rbac-proxy/0.log" Sep 11 09:45:33.951439 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:33.951393 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_bbea5b4b-7d0d-4bbb-be9b-25fe1a714e3f/kube-rbac-proxy-metric/0.log" Sep 11 09:45:33.979863 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:33.979841 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_bbea5b4b-7d0d-4bbb-be9b-25fe1a714e3f/prom-label-proxy/0.log" Sep 11 09:45:34.011419 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:34.011377 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_bbea5b4b-7d0d-4bbb-be9b-25fe1a714e3f/init-config-reloader/0.log" Sep 11 09:45:34.095521 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:34.095401 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-pft5k_2c1e552c-df45-4843-8717-37ee9b7d500f/kube-state-metrics/0.log" Sep 11 09:45:34.120957 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:34.120925 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-pft5k_2c1e552c-df45-4843-8717-37ee9b7d500f/kube-rbac-proxy-main/0.log" Sep 11 09:45:34.149879 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:34.149844 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-pft5k_2c1e552c-df45-4843-8717-37ee9b7d500f/kube-rbac-proxy-self/0.log" Sep 11 09:45:34.181081 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:34.181055 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-7888d6449c-zdchh_1a18b6af-13e2-4090-8750-5a0a06ad2719/metrics-server/0.log" Sep 11 09:45:34.211803 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:34.211776 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5d85778b9d-wrc6v_9b068dbf-c4bd-4527-9025-7ce2b277ac0a/monitoring-plugin/0.log" Sep 11 09:45:34.247302 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:34.247278 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rbngs_fad3f414-87a1-4ae6-a3e4-62346dfa62e4/node-exporter/0.log" Sep 11 09:45:34.276047 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:34.276022 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rbngs_fad3f414-87a1-4ae6-a3e4-62346dfa62e4/kube-rbac-proxy/0.log" Sep 11 09:45:34.300514 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:34.300489 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rbngs_fad3f414-87a1-4ae6-a3e4-62346dfa62e4/init-textfile/0.log" Sep 11 09:45:34.501545 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:34.501517 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-4xdpg_58d6994e-3003-41c3-857f-734e806d6edd/kube-rbac-proxy-main/0.log" Sep 11 09:45:34.527516 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:34.527479 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-4xdpg_58d6994e-3003-41c3-857f-734e806d6edd/kube-rbac-proxy-self/0.log" Sep 11 09:45:34.555070 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:34.555040 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-4xdpg_58d6994e-3003-41c3-857f-734e806d6edd/openshift-state-metrics/0.log" Sep 11 09:45:34.595092 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:34.595071 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_86b6d5a2-9069-4296-ae71-cc14fe79dac8/prometheus/0.log" Sep 11 09:45:34.619762 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:34.619736 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_86b6d5a2-9069-4296-ae71-cc14fe79dac8/config-reloader/0.log" Sep 11 09:45:34.651877 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:34.651850 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_86b6d5a2-9069-4296-ae71-cc14fe79dac8/thanos-sidecar/0.log" Sep 11 09:45:34.730568 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:34.730543 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_86b6d5a2-9069-4296-ae71-cc14fe79dac8/kube-rbac-proxy-web/0.log" Sep 11 09:45:34.773129 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:34.773060 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_86b6d5a2-9069-4296-ae71-cc14fe79dac8/kube-rbac-proxy/0.log" Sep 11 09:45:34.812012 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:34.811987 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_86b6d5a2-9069-4296-ae71-cc14fe79dac8/kube-rbac-proxy-thanos/0.log" Sep 11 09:45:34.843259 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:34.843238 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_86b6d5a2-9069-4296-ae71-cc14fe79dac8/init-config-reloader/0.log" Sep 11 09:45:34.904131 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:34.904101 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-4xxgk_5cc1de09-5625-4d17-a546-1c273a1183ba/prometheus-operator/0.log" Sep 11 09:45:34.944765 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:34.944736 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-4xxgk_5cc1de09-5625-4d17-a546-1c273a1183ba/kube-rbac-proxy/0.log" Sep 11 09:45:35.059096 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:35.059013 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-655fd9c684-n6hz2_aea7ca13-21b0-4745-9493-cfa07e40e36b/telemeter-client/0.log" Sep 11 09:45:35.115247 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:35.115218 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-655fd9c684-n6hz2_aea7ca13-21b0-4745-9493-cfa07e40e36b/reload/0.log" Sep 11 09:45:35.168274 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:35.168248 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-655fd9c684-n6hz2_aea7ca13-21b0-4745-9493-cfa07e40e36b/kube-rbac-proxy/0.log" Sep 11 09:45:35.229032 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:35.229005 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6bc45d4b6-qfb5s_91974a2e-4506-4077-897b-3b0ec57820a1/thanos-query/0.log" Sep 11 09:45:35.260009 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:35.259985 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6bc45d4b6-qfb5s_91974a2e-4506-4077-897b-3b0ec57820a1/kube-rbac-proxy-web/0.log" Sep 11 09:45:35.291119 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:35.291088 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6bc45d4b6-qfb5s_91974a2e-4506-4077-897b-3b0ec57820a1/kube-rbac-proxy/0.log" Sep 11 09:45:35.320855 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:35.320777 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6bc45d4b6-qfb5s_91974a2e-4506-4077-897b-3b0ec57820a1/prom-label-proxy/0.log" Sep 11 09:45:35.354175 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:35.354150 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6bc45d4b6-qfb5s_91974a2e-4506-4077-897b-3b0ec57820a1/kube-rbac-proxy-rules/0.log" Sep 11 09:45:35.390347 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:35.390312 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6bc45d4b6-qfb5s_91974a2e-4506-4077-897b-3b0ec57820a1/kube-rbac-proxy-metrics/0.log" Sep 11 09:45:37.464448 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:37.464414 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-m44nw/perf-node-gather-daemonset-pml6q"] Sep 11 09:45:37.464851 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:37.464758 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dff6a025-7b07-4f35-a5d0-a300818e4a4a" containerName="copy" Sep 11 09:45:37.464851 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:37.464770 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="dff6a025-7b07-4f35-a5d0-a300818e4a4a" containerName="copy" Sep 11 09:45:37.464851 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:37.464782 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dff6a025-7b07-4f35-a5d0-a300818e4a4a" containerName="gather" Sep 11 09:45:37.464851 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:37.464787 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="dff6a025-7b07-4f35-a5d0-a300818e4a4a" containerName="gather" Sep 11 09:45:37.464851 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:37.464832 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="dff6a025-7b07-4f35-a5d0-a300818e4a4a" containerName="gather" Sep 11 09:45:37.464851 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:37.464845 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="dff6a025-7b07-4f35-a5d0-a300818e4a4a" containerName="copy" Sep 11 09:45:37.466642 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:37.466619 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-m44nw/perf-node-gather-daemonset-pml6q" Sep 11 09:45:37.469017 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:37.468993 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-m44nw\"/\"openshift-service-ca.crt\"" Sep 11 09:45:37.469124 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:37.468993 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-m44nw\"/\"kube-root-ca.crt\"" Sep 11 09:45:37.469741 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:37.469725 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-m44nw\"/\"default-dockercfg-cvsth\"" Sep 11 09:45:37.478291 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:37.478267 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-m44nw/perf-node-gather-daemonset-pml6q"] Sep 11 09:45:37.654783 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:37.654745 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d00bf94e-8cf0-4c03-a422-54a684fbfd7c-lib-modules\") pod \"perf-node-gather-daemonset-pml6q\" (UID: \"d00bf94e-8cf0-4c03-a422-54a684fbfd7c\") " pod="openshift-must-gather-m44nw/perf-node-gather-daemonset-pml6q" Sep 11 09:45:37.654988 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:37.654826 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d00bf94e-8cf0-4c03-a422-54a684fbfd7c-proc\") pod \"perf-node-gather-daemonset-pml6q\" (UID: \"d00bf94e-8cf0-4c03-a422-54a684fbfd7c\") " pod="openshift-must-gather-m44nw/perf-node-gather-daemonset-pml6q" Sep 11 09:45:37.654988 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:37.654867 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d00bf94e-8cf0-4c03-a422-54a684fbfd7c-podres\") pod \"perf-node-gather-daemonset-pml6q\" (UID: \"d00bf94e-8cf0-4c03-a422-54a684fbfd7c\") " pod="openshift-must-gather-m44nw/perf-node-gather-daemonset-pml6q" Sep 11 09:45:37.654988 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:37.654936 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d00bf94e-8cf0-4c03-a422-54a684fbfd7c-sys\") pod \"perf-node-gather-daemonset-pml6q\" (UID: \"d00bf94e-8cf0-4c03-a422-54a684fbfd7c\") " pod="openshift-must-gather-m44nw/perf-node-gather-daemonset-pml6q" Sep 11 09:45:37.655177 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:37.654987 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r299d\" (UniqueName: \"kubernetes.io/projected/d00bf94e-8cf0-4c03-a422-54a684fbfd7c-kube-api-access-r299d\") pod \"perf-node-gather-daemonset-pml6q\" (UID: \"d00bf94e-8cf0-4c03-a422-54a684fbfd7c\") " pod="openshift-must-gather-m44nw/perf-node-gather-daemonset-pml6q" Sep 11 09:45:37.755824 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:37.755783 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r299d\" (UniqueName: \"kubernetes.io/projected/d00bf94e-8cf0-4c03-a422-54a684fbfd7c-kube-api-access-r299d\") pod \"perf-node-gather-daemonset-pml6q\" (UID: \"d00bf94e-8cf0-4c03-a422-54a684fbfd7c\") " pod="openshift-must-gather-m44nw/perf-node-gather-daemonset-pml6q" Sep 11 09:45:37.756029 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:37.755847 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d00bf94e-8cf0-4c03-a422-54a684fbfd7c-lib-modules\") pod \"perf-node-gather-daemonset-pml6q\" (UID: \"d00bf94e-8cf0-4c03-a422-54a684fbfd7c\") " pod="openshift-must-gather-m44nw/perf-node-gather-daemonset-pml6q" Sep 11 09:45:37.756029 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:37.755895 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d00bf94e-8cf0-4c03-a422-54a684fbfd7c-proc\") pod \"perf-node-gather-daemonset-pml6q\" (UID: \"d00bf94e-8cf0-4c03-a422-54a684fbfd7c\") " pod="openshift-must-gather-m44nw/perf-node-gather-daemonset-pml6q" Sep 11 09:45:37.756029 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:37.755932 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d00bf94e-8cf0-4c03-a422-54a684fbfd7c-podres\") pod \"perf-node-gather-daemonset-pml6q\" (UID: \"d00bf94e-8cf0-4c03-a422-54a684fbfd7c\") " pod="openshift-must-gather-m44nw/perf-node-gather-daemonset-pml6q" Sep 11 09:45:37.756029 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:37.755969 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d00bf94e-8cf0-4c03-a422-54a684fbfd7c-sys\") pod \"perf-node-gather-daemonset-pml6q\" (UID: \"d00bf94e-8cf0-4c03-a422-54a684fbfd7c\") " pod="openshift-must-gather-m44nw/perf-node-gather-daemonset-pml6q" Sep 11 09:45:37.756029 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:37.755995 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d00bf94e-8cf0-4c03-a422-54a684fbfd7c-proc\") pod \"perf-node-gather-daemonset-pml6q\" (UID: \"d00bf94e-8cf0-4c03-a422-54a684fbfd7c\") " pod="openshift-must-gather-m44nw/perf-node-gather-daemonset-pml6q" Sep 11 09:45:37.756243 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:37.756031 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d00bf94e-8cf0-4c03-a422-54a684fbfd7c-lib-modules\") pod \"perf-node-gather-daemonset-pml6q\" (UID: \"d00bf94e-8cf0-4c03-a422-54a684fbfd7c\") " pod="openshift-must-gather-m44nw/perf-node-gather-daemonset-pml6q" Sep 11 09:45:37.756243 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:37.756062 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d00bf94e-8cf0-4c03-a422-54a684fbfd7c-sys\") pod \"perf-node-gather-daemonset-pml6q\" (UID: \"d00bf94e-8cf0-4c03-a422-54a684fbfd7c\") " pod="openshift-must-gather-m44nw/perf-node-gather-daemonset-pml6q" Sep 11 09:45:37.756243 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:37.756124 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d00bf94e-8cf0-4c03-a422-54a684fbfd7c-podres\") pod \"perf-node-gather-daemonset-pml6q\" (UID: \"d00bf94e-8cf0-4c03-a422-54a684fbfd7c\") " pod="openshift-must-gather-m44nw/perf-node-gather-daemonset-pml6q" Sep 11 09:45:37.767094 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:37.767056 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r299d\" (UniqueName: \"kubernetes.io/projected/d00bf94e-8cf0-4c03-a422-54a684fbfd7c-kube-api-access-r299d\") pod \"perf-node-gather-daemonset-pml6q\" (UID: \"d00bf94e-8cf0-4c03-a422-54a684fbfd7c\") " pod="openshift-must-gather-m44nw/perf-node-gather-daemonset-pml6q" Sep 11 09:45:37.776018 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:37.775992 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-m44nw/perf-node-gather-daemonset-pml6q" Sep 11 09:45:38.044889 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:38.044863 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-m44nw/perf-node-gather-daemonset-pml6q"] Sep 11 09:45:38.047462 ip-10-0-135-119 kubenswrapper[2575]: W0911 09:45:38.047437 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podd00bf94e_8cf0_4c03_a422_54a684fbfd7c.slice/crio-a9f9038906d569d27cbce4b29aad7e84811fb11580aa7273ece9a251f7d8fd6a WatchSource:0}: Error finding container a9f9038906d569d27cbce4b29aad7e84811fb11580aa7273ece9a251f7d8fd6a: Status 404 returned error can't find the container with id a9f9038906d569d27cbce4b29aad7e84811fb11580aa7273ece9a251f7d8fd6a Sep 11 09:45:38.717488 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:38.717447 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-vb5ck_54fff86e-d33b-4b2b-bedf-b00daa2f301c/dns/0.log" Sep 11 09:45:38.742120 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:38.742090 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-vb5ck_54fff86e-d33b-4b2b-bedf-b00daa2f301c/kube-rbac-proxy/0.log" Sep 11 09:45:38.795817 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:38.795789 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-g6t8z_5a29489a-506a-4070-b30d-7366366058dc/dns-node-resolver/0.log" Sep 11 09:45:38.892118 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:38.892088 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-m44nw/perf-node-gather-daemonset-pml6q" event={"ID":"d00bf94e-8cf0-4c03-a422-54a684fbfd7c","Type":"ContainerStarted","Data":"43dc1693c97b4fb5019c38ac1d917e8b079f3ae822a19dc506a262a259db61dd"} Sep 11 09:45:38.892308 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:38.892136 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-m44nw/perf-node-gather-daemonset-pml6q" event={"ID":"d00bf94e-8cf0-4c03-a422-54a684fbfd7c","Type":"ContainerStarted","Data":"a9f9038906d569d27cbce4b29aad7e84811fb11580aa7273ece9a251f7d8fd6a"} Sep 11 09:45:38.892308 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:38.892157 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-m44nw/perf-node-gather-daemonset-pml6q" Sep 11 09:45:38.915247 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:38.915193 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-m44nw/perf-node-gather-daemonset-pml6q" podStartSLOduration=1.915176816 podStartE2EDuration="1.915176816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 09:45:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 09:45:38.913778781 +0000 UTC m=+1955.254388988" watchObservedRunningTime="2026-09-11 09:45:38.915176816 +0000 UTC m=+1955.255787021" Sep 11 09:45:39.348261 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:39.348227 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-skn2c_08d48976-3c61-4273-88da-fa30ba4057bc/node-ca/0.log" Sep 11 09:45:40.488947 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:40.488916 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-bsj85_4e0aace6-bbd9-4cbb-8759-165ffbaa3b2a/serve-healthcheck-canary/0.log" Sep 11 09:45:41.165113 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:41.165086 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-znq28_31745858-acda-4939-97ef-b5d9f6ff710a/kube-rbac-proxy/0.log" Sep 11 09:45:41.198101 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:41.198069 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-znq28_31745858-acda-4939-97ef-b5d9f6ff710a/exporter/0.log" Sep 11 09:45:41.236380 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:41.236356 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-znq28_31745858-acda-4939-97ef-b5d9f6ff710a/extractor/0.log" Sep 11 09:45:44.905027 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:44.904993 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-m44nw/perf-node-gather-daemonset-pml6q" Sep 11 09:45:47.356914 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:47.356884 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8z7wv_34c95ff5-365b-414d-9968-2db918292b7f/kube-multus-additional-cni-plugins/0.log" Sep 11 09:45:47.384588 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:47.384517 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8z7wv_34c95ff5-365b-414d-9968-2db918292b7f/egress-router-binary-copy/0.log" Sep 11 09:45:47.407147 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:47.407118 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8z7wv_34c95ff5-365b-414d-9968-2db918292b7f/cni-plugins/0.log" Sep 11 09:45:47.429931 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:47.429879 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8z7wv_34c95ff5-365b-414d-9968-2db918292b7f/bond-cni-plugin/0.log" Sep 11 09:45:47.452962 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:47.452938 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8z7wv_34c95ff5-365b-414d-9968-2db918292b7f/routeoverride-cni/0.log" Sep 11 09:45:47.476412 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:47.476383 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8z7wv_34c95ff5-365b-414d-9968-2db918292b7f/whereabouts-cni-bincopy/0.log" Sep 11 09:45:47.501961 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:47.501937 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8z7wv_34c95ff5-365b-414d-9968-2db918292b7f/whereabouts-cni/0.log" Sep 11 09:45:47.864959 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:47.864927 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-dvpdv_27736c8f-eddf-4460-94fd-b211c07ae744/kube-multus/0.log" Sep 11 09:45:48.051493 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:48.051448 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-rlftv_abeb037b-2e12-4485-974d-35280f982f4a/network-metrics-daemon/0.log" Sep 11 09:45:48.072652 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:48.072617 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-rlftv_abeb037b-2e12-4485-974d-35280f982f4a/kube-rbac-proxy/0.log" Sep 11 09:45:49.507010 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:49.506978 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wgrlw_3c0627e0-f086-4fc4-b2d7-724b67c685d4/ovn-controller/0.log" Sep 11 09:45:49.530715 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:49.530689 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wgrlw_3c0627e0-f086-4fc4-b2d7-724b67c685d4/ovn-acl-logging/0.log" Sep 11 09:45:49.553560 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:49.553536 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wgrlw_3c0627e0-f086-4fc4-b2d7-724b67c685d4/kube-rbac-proxy-node/0.log" Sep 11 09:45:49.579581 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:49.579554 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wgrlw_3c0627e0-f086-4fc4-b2d7-724b67c685d4/kube-rbac-proxy-ovn-metrics/0.log" Sep 11 09:45:49.600723 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:49.600700 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wgrlw_3c0627e0-f086-4fc4-b2d7-724b67c685d4/northd/0.log" Sep 11 09:45:49.626568 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:49.626541 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wgrlw_3c0627e0-f086-4fc4-b2d7-724b67c685d4/nbdb/0.log" Sep 11 09:45:49.663503 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:49.663462 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wgrlw_3c0627e0-f086-4fc4-b2d7-724b67c685d4/sbdb/0.log" Sep 11 09:45:49.708452 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:49.708424 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wgrlw_3c0627e0-f086-4fc4-b2d7-724b67c685d4/ovnkube-controller/0.log" Sep 11 09:45:51.056644 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:51.056614 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-bb6x9_92eb1c40-b8d0-428c-955a-c005e197f70a/network-check-target-container/0.log" Sep 11 09:45:52.123192 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:52.123163 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-5xwxg_65ac2ff5-03f0-4143-871f-061152802913/iptables-alerter/0.log" Sep 11 09:45:53.045753 ip-10-0-135-119 kubenswrapper[2575]: I0911 09:45:53.045727 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-9zgvm_fcb16b38-81cb-41f7-81d2-730c64ee3a59/tuned/0.log"